I want to know how can I select a most recent CSV file from a specific folder using Node Red? I want dynamic selection of a CSV file from a specific folder. I basically, I know the folder from which the CSV file has to be selected. After selecting the file, I am sending the CSV data to MSSQL database. Currently, I am entering the file name and path in MSSQL node. But I want to select a latest file from a specific folder? I am unable to do it. How can I proceed?
What platofrm are you running on - Windows or Linux ?
Do you know the filename or just the directory and it will be the newest CSV file ?
I would use the exec node on linux and do a bash shell script to find and return the newest file with CSV
http://mywiki.wooledge.org/BashFAQ/003
This assumes that you will have permission to access that file/directory
If you are on Windows 10 you could use the same (if you want to use the WSL in WIndows 10) or you could do a powershell script like this
regards
Craig
I am working on a Windows 10 based system. I know the file name. But the folder will be updated with new flies constantly. File name can be 'Name_**' where the ** will be changing part in file name. I want node red to access the folder and access the most recent file. This is what I intend to do.Is it possible?
Yep - just pass the wildcard parameter into powershell
I suggest you get this working as a PS script first and then try to exec out to it (i have not tried it on Windows as i use Linux for NR only)
But i do know and work with PS on my local windows machine etc
Craig
here for example
Is the name under your control? If so then if you made the variable bit be a timestamp (so something like Name_200511_1051) then you could use node-red-contrib-fsops to fetch the files in the directory and sort the names to get the most recent.
To save having to continuously monitor the directory then use a Watch node on the directory to tell you when things change.
However you do it watch out for the problem of picking up a file while it is still being written to. The best way is to write it with a identifiable name (with a .tmp extension for example) so you know to ignore it, and then rename it to the correct name when it is fully written.
Another, cross-platform solution would be to attach the node.js standard fs library to a global var in settings.js then you can write a function that will read the folder. Better still, use the fs-extra npm package (which Node-RED already uses as does uibuilder).
I find the fs-ops nodes perfect for this kinda thing...
[{"id":"6456763b.e07d78","type":"fs-ops-dir","z":"6ef1cfb5.30ff2","name":"","path":"payload","pathType":"msg","filter":"*.csv","filterType":"str","dir":"files","dirType":"msg","x":380,"y":460,"wires":[["35811671.90ebea","572a14c4.62aa3c"]]},{"id":"35811671.90ebea","type":"fs-ops-stats","z":"6ef1cfb5.30ff2","name":"","path":"payload","pathType":"msg","filename":"files","filenameType":"msg","stats":"stats","sizeType":"msg","x":540,"y":460,"wires":[["20ef5f9.b8d63a","ee6515ea.f968c8"]]},{"id":"7dbdf6ef.a73868","type":"inject","z":"6ef1cfb5.30ff2","name":"","topic":"","payload":"c:\\temp","payloadType":"str","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":250,"y":460,"wires":[["6456763b.e07d78"]]},{"id":"dd9228cf.983768","type":"debug","z":"6ef1cfb5.30ff2","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"combined","targetType":"msg","x":940,"y":520,"wires":[]},{"id":"572a14c4.62aa3c","type":"debug","z":"6ef1cfb5.30ff2","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"files","targetType":"msg","x":520,"y":520,"wires":[]},{"id":"20ef5f9.b8d63a","type":"debug","z":"6ef1cfb5.30ff2","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"stats","targetType":"msg","x":720,"y":520,"wires":[]},{"id":"ee6515ea.f968c8","type":"function","z":"6ef1cfb5.30ff2","name":"Combine and sort","func":"var combined = [];\n\nfor (var i = 0; i < msg.files.length; i++) {\n var file = msg.files[i];\n var stat = msg.stats[i];\n \n combined.push({\n path: msg.payload,\n file,\n ...stat\n })\n}\n\ncombined.sort((a, b) => b.birthtimeMs - a.birthtimeMs);\n\nmsg.combined = combined;\nreturn msg;","outputs":1,"noerr":0,"x":730,"y":460,"wires":[["dd9228cf.983768","50034043.e2347"]]},{"id":"50034043.e2347","type":"change","z":"6ef1cfb5.30ff2","name":"","rules":[{"t":"set","p":"payload","pt":"msg","to":"combined[0]","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":940,"y":460,"wires":[["4f9068e.e12c498"]]},{"id":"4f9068e.e12c498","type":"debug","z":"6ef1cfb5.30ff2","name":"newest CSV","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","x":1110,"y":460,"wires":[]},{"id":"25428842.245c68","type":"comment","z":"6ef1cfb5.30ff2","name":"Directory gets list of CSVs. File Stats gets their details. combine and sort puts data together and sorts. Switch node takes 1st item and puts it in payload","info":"","x":700,"y":400,"wires":[]}]
See flows library for more detail - https://flows.nodered.org/flow/5f3bf35079ec128a8ddb11180da279ad
Actually, I think my own fs node would also do what is needed in this case 
It hasn't had a lot of TLC recently but still works fine. And if it doesn't do what is needed, raising an issue would surely prompt me to dig into the midnight oil to add something, I doubt it would be difficult.
Which node are you referring to? I didn't see anything called fs-extra?
Edit ...
What I really like about fs-ops is the generous use of typedinput to allow you to build up data during the passage of msg and it's handling of single or array of files. Makes the flow (like above) quite simple and elegant.
Was that to me?
Yeah. Sorry, thought I'd replied to your message (not the general thread).
Cheers.
This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.
