Hi all,
Thanks to @Colin for reporting an issue with this node that would crash Node-RED due to an out-of-memory error. That has now been fixed.
As a bonus, not only should the node operate quite a bit faster, you now also get an extra option for turning off the "cannot access file" warnings - most of them relate to files that the OS user running node-red doesn't have permissions to access and there can be an excessive number in some cases.
Excellent, thanks Julian
There is a bug introduced with this release. The ‘start’ option is being ignored. I opened a issue in github with a proposed solution.
Is it entirely ignored? It seems to be basically functioning for me.
It's happening when you pass in the start value in msg.payload.
Here is a simple example of two flows. This is for running on a Pi. The idea is to look for the file "settings.js" in the /home/pi/.node-red' folder.
CASE 1 has all the options in the
fs file listernode
while
CASE 2 passs all the options in msg.payload.
[{"id":"c4660d28.b7e31","type":"inject","z":"9d6e4527.18bf3","name":"CASE 1 - options all in fs-lister","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":190,"y":440,"wires":[["c7b8b65a.2a247"]]},{"id":"c7b8b65a.2a247","type":"fs-file-lister","z":"9d6e4527.18bf3","name":"","start":"/home/pi/.node-red","pattern":"settings.js","folders":"*","hidden":false,"lstype":"files","path":true,"single":true,"depth":"1","stat":false,"showWarnings":true,"x":500,"y":440,"wires":[["96e2367d.c84d98"]]},{"id":"96e2367d.c84d98","type":"debug","z":"9d6e4527.18bf3","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":650,"y":440,"wires":[]},{"id":"23cec9a0.0a26d6","type":"inject","z":"9d6e4527.18bf3","name":"CASE 2 - options PASSED INTO fs-lister","topic":"","payload":"{\"start\":\"/home/pi/songs\",\"depth\":1,\"pattern\":\"settings.js\"}","payloadType":"json","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":220,"y":500,"wires":[["5a0b0d1d.e63064"]]},{"id":"5a0b0d1d.e63064","type":"fs-file-lister","z":"9d6e4527.18bf3","name":"","start":"/","pattern":"*.*","folders":"*","hidden":false,"lstype":"files","path":true,"single":true,"depth":"0","stat":false,"showWarnings":true,"x":500,"y":500,"wires":[["705fce68.f8c41"]]},{"id":"705fce68.f8c41","type":"debug","z":"9d6e4527.18bf3","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":650,"y":500,"wires":[]}]
If you look at msg.config in the two debugs you will see they are identical. The problem is in the code, it calls readdirp passing node.start which never gets updated with what's passed in from msg.payload. However clonedMsg.config.start is properly updated if the start is passed in msg.payload
OK, I don't use that feature so that explains it.
I missed a configuration when I did the last update. Basically, this update separates out changes to the nodes configuration so that they are only done once (e.g. outside the node.on('input'... section.
That includes the overrides on the incoming message. Before, they were making changes to the node object which is the object shared by all instances. Now, they make changes to the clonedMsg object (which already has a copy of all of the relavent settings anyway) & changes to the node object only ever happen once for all instances.
I simply missed one of the references to node which should have been - and now is - a reference to clonedMsg.config.
v1.3.2 will be along shortly.
Published v1.3.2
I am using the file lister and it is great. In my setup I have the file lister configured to return file and folders as well.
I am having issues telling if the record in the array is a file or a folder.
I can see the msg.payload.stat.mode field, but I don't know how to tell from this if this is a file or a folder.
Can you guys help with that?
Looks like you can use payload.stat.mode (or payload[n].stat.mode it returning an array where n is the index in the array) and check it's value. If it is equal to '0x81a4' it is a file
and it it is equal to '0x41ed' it is a folder (at least that is what I get on my Mac)
I suspect it is a little bit more complicated that that. I have a few files where the mode also comes same as the folder. So I am confused. I assume this is somehow a list of bitbanged value, but can't find out what it is.
I would have expected the mode to be same as you see when you use ls -l in which case is should be of the form drwxrwxrwx so bit 9 should be the directory flag (octal 01000 or in hex 0x0200), but I don't know why it would ever be larger than 0x3ff.
Possibly @TotallyInformation can comment on this.
possibly useful thread (or not)
Maybe, I just need to use two file lister nodes but I thought there is a better option for that.
Actually, I changed my flow to use two separate file lister nodes, and did a simple "File Browser" in dashboard. So I can quickly check files on my dashboard instead of opening putty. What do you guys think?
[{"id":"4fa73dd9.83cca4","type":"comment","z":"74f191ff.db063","name":"File Browser","info":"","x":110,"y":2940,"wires":[]},{"id":"993d7272.843ae","type":"fs-file-lister","z":"74f191ff.db063","name":"Files","start":"/home/pi","pattern":"*.*","folders":"*","hidden":true,"lstype":"files","path":true,"single":true,"depth":0,"stat":true,"showWarnings":true,"x":510,"y":3040,"wires":[["dd88bd23.37cde","3e094f82.15d6e"]]},{"id":"d185a45.2327a58","type":"inject","z":"74f191ff.db063","name":"Refresh","topic":"refresh","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":110,"y":3000,"wires":[["3d4e5e51.bdf952"]]},{"id":"a715f7a4.a398a8","type":"ui_button","z":"74f191ff.db063","name":"","group":"160e81fb.f1c86e","order":1,"width":"2","height":"1","passthru":false,"label":"Refresh","tooltip":"","color":"","bgcolor":"","icon":"refresh","payload":"","payloadType":"str","topic":"refresh","x":100,"y":3040,"wires":[["3d4e5e51.bdf952"]]},{"id":"eb17c7ee.e61988","type":"ui_dropdown","z":"74f191ff.db063","name":"File Selector","label":"","tooltip":"","place":"Select a file","group":"160e81fb.f1c86e","order":8,"width":"5","height":"1","passthru":false,"options":[{"label":"","value":"","type":"str"}],"payload":"","topic":"","x":910,"y":3040,"wires":[["eeb79a9b.4421e8"]]},{"id":"dd88bd23.37cde","type":"function","z":"74f191ff.db063","name":"Format data","func":"msg.options = [];\nfor (var i=0; i<msg.payload.length; i++) {\n // This is a file\n obj = {};\n obj [msg.payload[i].name.replace(/^.*(\\\\|\\/|\\:)/, '')]=msg.payload[i].name;\n msg.options.push(obj);\n}\nmsg.payload={};\nreturn msg;","outputs":1,"noerr":0,"x":710,"y":3040,"wires":[["eb17c7ee.e61988"]]},{"id":"638728aa.cd0b08","type":"ui_template","z":"74f191ff.db063","group":"160e81fb.f1c86e","name":"","order":5,"width":"18","height":"6","format":"<div ng-bind-html=\"msg.payload\" height=\"400\" style=\"height: 400px;\"><br/>\n</div>\n\n","storeOutMessages":true,"fwdInMessages":true,"templateScope":"local","x":1100,"y":3100,"wires":[[]]},{"id":"9d27e846.b31db8","type":"template","z":"74f191ff.db063","name":"","field":"payload","fieldType":"msg","format":"handlebars","syntax":"mustache","template":"<table width=\"100%\">\n <tr><th>File Name</th><th>Size</th><th>Created</th><th>Changed</th></tr>\n {{#payload}}\n <tr>\n <td><a href=\"/download?filename={{name}}\" target=\"blank\">{{fname}}</a></td>\n <td>{{stat.size}}</td>\n <td>{{stat.created}}</td>\n <td>{{stat.changed}}</td>\n </tr>\n {{/payload}}\n</table>\n","output":"str","x":940,"y":3100,"wires":[["638728aa.cd0b08"]]},{"id":"3e094f82.15d6e","type":"function","z":"74f191ff.db063","name":"Convert timestamps","func":"for (var i=0; i<msg.payload.length; i++) {\n msg.payload[i].stat.created = msg.payload[i].stat.created.toISOString().slice(0, 19).replace('T', ' ');\n msg.payload[i].stat.changed = msg.payload[i].stat.changed.toISOString().slice(0, 19).replace('T', ' ');\n msg.payload[i].stat.accessed = msg.payload[i].stat.accessed.toISOString().slice(0, 19).replace('T', ' ');\n msg.payload[i].stat.statusChanged = msg.payload[i].stat.statusChanged.toISOString().slice(0, 19).replace('T', ' ');\n msg.payload[i].fname = msg.payload[i].name.replace(/^.*(\\\\|\\/|\\:)/, '');\n}\nreturn msg;","outputs":1,"noerr":0,"x":740,"y":3100,"wires":[["9d27e846.b31db8"]]},{"id":"eeb79a9b.4421e8","type":"function","z":"74f191ff.db063","name":"Save selection","func":"// Save the file name selected from the dropdown in the flow context\nflow.set(\"fileselected\", msg.payload);\nreturn msg;","outputs":1,"noerr":0,"x":1140,"y":3040,"wires":[[]]},{"id":"7b7e0d58.05e244","type":"ui_button","z":"74f191ff.db063","name":"","group":"160e81fb.f1c86e","order":9,"width":"2","height":"1","passthru":false,"label":"Delete","tooltip":"","color":"","bgcolor":"","icon":"delete","payload":"","payloadType":"str","topic":"","x":130,"y":3200,"wires":[["8da8147a.999af8"]]},{"id":"7d490dd1.8458b4","type":"function","z":"74f191ff.db063","name":"Get filename","func":"// Get the filename from the flow context\nlet filename = flow.get(\"fileselected\");\n\n// check, if the filename is undefined that means it does not exist yet, nothing is selected yet\n// return: do not output anything\nif (filename===undefined) {\n return;\n}\n\n// return the filename to the file-in node to delete\nmsg.filename = filename;\n\nif (msg.filename.replace(/^.*(\\\\|\\/|\\:)/, '')[0]!==\".\") {\n // Only do this if this is a file, we don't delete folders\n // and delete the context/selection as we are deleting the file as well\n flow.set(\"fileselected\");\n return msg;\n}","outputs":1,"noerr":0,"x":330,"y":3260,"wires":[["372ee262.fc94de"]]},{"id":"372ee262.fc94de","type":"file","z":"74f191ff.db063","name":"Delete file","filename":"","appendNewline":true,"createDir":false,"overwriteFile":"delete","encoding":"none","x":540,"y":3260,"wires":[["993d7272.843ae","5219875b.c070d8"]]},{"id":"aa8c482.93734b8","type":"http in","z":"74f191ff.db063","name":"","url":"/download","method":"get","upload":false,"swaggerDoc":"","x":140,"y":3400,"wires":[["d969ba04.e24028"]]},{"id":"d19cc7d8.646328","type":"http response","z":"74f191ff.db063","name":"","statusCode":"","headers":{},"x":930,"y":3400,"wires":[]},{"id":"d969ba04.e24028","type":"function","z":"74f191ff.db063","name":"Get the file name","func":"msg.filename = msg.req.query.filename;\nmsg.contentdisposition = \"attachment; filename=\\\"\" + msg.req.query.filename.replace(/^.*(\\\\|\\/|\\:)/, '') + \"\\\"\";\nreturn msg;\n","outputs":1,"noerr":0,"x":390,"y":3400,"wires":[["e92381c3.c4cd2"]],"outputLabels":["Folder selected","File selected"]},{"id":"e92381c3.c4cd2","type":"file in","z":"74f191ff.db063","name":"","filename":"","format":"","chunk":false,"sendError":false,"encoding":"none","x":580,"y":3400,"wires":[["99ff4953.d0d5c8"]]},{"id":"99ff4953.d0d5c8","type":"change","z":"74f191ff.db063","name":"Set Headers","rules":[{"t":"set","p":"headers","pt":"msg","to":"{}","tot":"json"},{"t":"set","p":"headers.content-type","pt":"msg","to":"text/csv","tot":"str"},{"t":"set","p":"headers.Content-Disposition","pt":"msg","to":"contentdisposition","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":750,"y":3400,"wires":[["d19cc7d8.646328"]]},{"id":"2519dd0d.54d6b2","type":"ui_button","z":"74f191ff.db063","name":"","group":"160e81fb.f1c86e","order":10,"width":"2","height":"1","passthru":false,"label":"Graph","tooltip":"","color":"","bgcolor":"","icon":"show_chart","payload":"","payloadType":"str","topic":"","x":90,"y":3520,"wires":[["30f41c01.358a54"]]},{"id":"30f41c01.358a54","type":"function","z":"74f191ff.db063","name":"Get filename","func":"// Get the filename from the flow context\nlet filename = flow.get(\"fileselected\");\n\n// check, if the filename is undefined that means it does not exist yet, nothing is selected yet\n// return: do not output anything\nif (filename===undefined) {\n return;\n}\n\n// return the filename to the file-in node to delete\nmsg.filename = filename;\n\nif (msg.filename.replace(/^.*(\\\\|\\/|\\:)/, '')[0]!==\".\") {\n // Only do this if this is a file, we don't delete folders\n return msg;\n}","outputs":1,"noerr":0,"x":260,"y":3520,"wires":[["4f4072b6.0c320c"]]},{"id":"4f4072b6.0c320c","type":"file in","z":"74f191ff.db063","name":"","filename":"","format":"utf8","chunk":false,"sendError":false,"encoding":"none","x":440,"y":3520,"wires":[["4d254665.d508e8"]]},{"id":"4d254665.d508e8","type":"csv","z":"74f191ff.db063","name":"","sep":",","hdrin":true,"hdrout":"","multi":"mult","ret":"\\n","temp":"","skip":"0","strings":true,"x":590,"y":3520,"wires":[["7dcec769.580718"]]},{"id":"84810f0e.43f4e","type":"debug","z":"74f191ff.db063","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","x":1010,"y":3480,"wires":[]},{"id":"7dcec769.580718","type":"function","z":"74f191ff.db063","name":"Format data for chart","func":"var chart = [{\n \"series\":[],\n \"data\":[],\n \"labels\":[msg.filename]\n}];\n\n\n/*\nvar pressure = [];\nvar out2 = [];\n\nfor (var i=0; i<msg.payload.length; i++) {\n pressure.push({\"x\":msg.payload[i].timestamp, \"y\":msg.payload[i].pressure});\n out2.push({\"x\":msg.payload[i].timestamp, \"y\":msg.payload[i].out2*200});\n\n}\nchart[0].data.push(pressure);\nchart[0].data.push(out2);\n*/\n\nlet columns = 0;\n\nfor(var series in msg.payload[0]) {\n if(series!==\"timestamp\") {\n chart[0].series.push(series);\n chart[0].data.push([]);\n columns++;\n }\n}\n\n\nfor (var j=0; j<msg.payload.length; j++) {\n for(var i=0;i<columns;i++) {\n chart[0].data[i].push({\"x\":msg.payload[j].timestamp, \"y\":msg.payload[j][chart[0].series[i]]});\n } \n}\n\n\n\n\nmsg.payload = chart;\n\nreturn msg;","outputs":1,"noerr":0,"x":800,"y":3520,"wires":[["5022cb53.8adc44","84810f0e.43f4e"]]},{"id":"5022cb53.8adc44","type":"ui_chart","z":"74f191ff.db063","name":"","group":"160e81fb.f1c86e","order":11,"width":"18","height":"6","label":"","chartType":"line","legend":"false","xformat":"HH:mm:ss","interpolate":"linear","nodata":"","dot":false,"ymin":"","ymax":"","removeOlder":1,"removeOlderPoints":"","removeOlderUnit":"3600","cutout":0,"useOneColor":false,"colors":["#1f77b4","#aec7e8","#ff7f0e","#2ca02c","#98df8a","#d62728","#ff9896","#9467bd","#c5b0d5"],"useOldStyle":false,"outputs":1,"x":1000,"y":3520,"wires":[[]]},{"id":"554f0095.ac1b2","type":"ui_toast","z":"74f191ff.db063","position":"dialog","displayTime":"3","highlight":"","sendall":false,"outputs":1,"ok":"Yes","cancel":"No","raw":false,"topic":"","name":"Confirmation","x":510,"y":3200,"wires":[["24df84fe.ebf45c"]]},{"id":"8da8147a.999af8","type":"change","z":"74f191ff.db063","name":"Set message","rules":[{"t":"set","p":"topic","pt":"msg","to":"Delete confirmation","tot":"str"},{"t":"set","p":"payload","pt":"msg","to":"Are you sure you want to delete this file?","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":310,"y":3200,"wires":[["554f0095.ac1b2"]]},{"id":"24df84fe.ebf45c","type":"switch","z":"74f191ff.db063","name":"","property":"payload","propertyType":"msg","rules":[{"t":"eq","v":"Yes","vt":"str"}],"checkall":"true","repair":false,"outputs":1,"x":710,"y":3200,"wires":[["7d490dd1.8458b4"]]},{"id":"3d4e5e51.bdf952","type":"function","z":"74f191ff.db063","name":"Folder handling","func":"let folder = context.get(\"folder\");\nif (folder===undefined) {\n folder=\"/home/pi\"\n context.set(\"folder\", folder);\n}\nif (msg.topic===\"up\") {\n var the_arr = folder.split('/');\n the_arr.pop();\n folder=the_arr.join('/'); \n context.set(\"folder\", folder);\n}\nif (msg.topic===\"change\") {\n folder=msg.payload; \n context.set(\"folder\", folder);\n}\nmsg.payload = {\"start\":folder};\nreturn msg;","outputs":1,"noerr":0,"x":300,"y":3100,"wires":[["993d7272.843ae","399ac341.7d43bc","5219875b.c070d8"]]},{"id":"399ac341.7d43bc","type":"ui_text","z":"74f191ff.db063","group":"160e81fb.f1c86e","order":4,"width":"11","height":"1","name":"","label":"Folder:","format":"{{msg.payload.start}}","layout":"row-left","x":500,"y":2980,"wires":[]},{"id":"b40ea1d8.c700a","type":"ui_button","z":"74f191ff.db063","name":"","group":"160e81fb.f1c86e","order":3,"width":"2","height":"1","passthru":false,"label":"Up","tooltip":"","color":"","bgcolor":"","icon":"arrow_upwards","payload":"","payloadType":"str","topic":"up","x":110,"y":3120,"wires":[["3d4e5e51.bdf952"]]},{"id":"80940e39.5035b","type":"ui_button","z":"74f191ff.db063","name":"","group":"160e81fb.f1c86e","order":7,"width":"2","height":"1","passthru":false,"label":"Open","tooltip":"","color":"","bgcolor":"","icon":"folder_open","payload":"","payloadType":"str","topic":"","x":130,"y":3320,"wires":[["8c1dfaac.979588"]]},{"id":"8c1dfaac.979588","type":"function","z":"74f191ff.db063","name":"Change folder","func":"// Get the filename from the flow context\nlet folderselected = flow.get(\"folderselected\");\n\n// check, if the filename is undefined that means it does not exist yet, nothing is selected yet\n// return: do not output anything\nif (folderselected===undefined) {\n return;\n}\n\nmsg.topic = \"change\";\nmsg.payload = folderselected;\n\nreturn msg;","outputs":1,"noerr":0,"x":340,"y":3320,"wires":[["3d4e5e51.bdf952"]]},{"id":"58414ec4.c716e","type":"ui_button","z":"74f191ff.db063","name":"","group":"160e81fb.f1c86e","order":2,"width":"2","height":"1","passthru":false,"label":"Reset","tooltip":"","color":"","bgcolor":"","icon":"autorenew","payload":"/home/pi","payloadType":"str","topic":"change","x":110,"y":3080,"wires":[["3d4e5e51.bdf952"]]},{"id":"5219875b.c070d8","type":"fs-file-lister","z":"74f191ff.db063","name":"Folders","start":"/home/pi","pattern":"*.*","folders":"*","hidden":true,"lstype":"directories","path":true,"single":true,"depth":0,"stat":true,"showWarnings":true,"x":520,"y":3140,"wires":[["ebc234be.d53fe8"]]},{"id":"37485c24.212054","type":"ui_dropdown","z":"74f191ff.db063","name":"Folder Selector","label":"","tooltip":"","place":"Select a folder","group":"160e81fb.f1c86e","order":6,"width":"5","height":"1","passthru":false,"options":[{"label":"","value":"","type":"str"}],"payload":"","topic":"","x":920,"y":3140,"wires":[["eeccda5.bea9b28"]]},{"id":"ebc234be.d53fe8","type":"function","z":"74f191ff.db063","name":"Format data","func":"msg.options = [];\nfor (var i=0; i<msg.payload.length; i++) {\n // This is a foler\n obj = {};\n obj [\"[\"+msg.payload[i].name.replace(/^.*(\\\\|\\/|\\:)/, '')+\"]\"]=msg.payload[i].name;\n msg.options.push(obj);\n}\nmsg.payload={};\nreturn msg;","outputs":1,"noerr":0,"x":710,"y":3140,"wires":[["37485c24.212054"]]},{"id":"eeccda5.bea9b28","type":"function","z":"74f191ff.db063","name":"Save selection","func":"// Save the file name selected from the dropdown in the flow context\nflow.set(\"folderselected\", msg.payload);\nreturn msg;","outputs":1,"noerr":0,"x":1140,"y":3140,"wires":[[]]},{"id":"160e81fb.f1c86e","type":"ui_group","z":"","name":"File Browser","tab":"b63d1f91.68095","order":1,"disp":true,"width":"18","collapse":false},{"id":"b63d1f91.68095","type":"ui_tab","z":"","name":"Files","icon":"dashboard","disabled":false,"hidden":false}]
Two things I would do
(1) put the base floder name in a flow variable and initilise it when the flow is deployed. That way you only have to change it in one place in case you want to move the flow to a different platform or a different Linux user other than 'pi'
(2) add an option to hide/show hidden files
(1) is done, but it looks like the hidden file option cannot be controlled by the input message. At least it is not mentioned in the documentation.
Hmmm, you should be able to pass msg.payload {"hidden":true} or false to get it to work but I see it is not working. Let me take a look (I played with this node a while back.)
Let me know if I need to add that. I'm afraid that don't hasn't had a lot of love so is doubtless due an update anyway.
Yes, that tells you what the values all mean and you can derive that it is a folder by looking at the appropriate bit.
Here is a technical descripttion: https://linux.die.net/man/2/stat
The node.js documentation is distinctly unhelpful - File system | Node.js v12.22.12 Documentation
You are looking for the file type: https://linuxconfig.org/identifying-file-types-in-linux
Perhaps I should include the outputs from the various isXxxx() functions?
If anyone wants to test. I've pushed v1.3.3 to GitHub Master branch. Fixes the override of hidden files issue.
Copyright OpenJS Foundation and Node-RED contributors. All rights reserved. The OpenJS Foundation has registered trademarks and uses trademarks. For a list of trademarks of the OpenJS Foundation, please see our Trademark Policy and Trademark List. Trademarks and logos not indicated on the list of OpenJS Foundation trademarks are trademarks™ or registered® trademarks of their respective holders. Use of them does not imply any affiliation with or endorsement by them.
The OpenJS Foundation | Terms of Use | Privacy Policy | OpenJS Foundation Bylaws | Trademark Policy | Trademark List | Cookie Policy