How to loop thru *.json files in a directory

General
1

Hi,
i have a directory containing ~500 *.json files, that must be processed. How can i create a loop, that reads one file after another, processes it (this is not the problem) and stores it with another filename.extension ?

The files were created with a bash in linux of the form 'for F in *.file; process ${F}; done'
Is there a way to create a processing procedure,with nodered, that can be called from a command line?

Edit: process in bash added.

2

One option is using this node:

it can list files in a folder, then you can read and edit them as you like. Afterwards, you can save them with a new filename

example flow

[{"id":"5fc40d6d5d98a4ab","type":"fs-copy-move","z":"145ff57cd2c491cc","oper":"mv","srcPath":"fs.source","srcPathType":"msg","srcFilename":"fs.file_source","srcFilenameType":"msg","dstPath":"fs.destination","dstPathType":"msg","dstFilename":"fs.file_destiation","dstFilenameType":"msg","overwrite":true,"name":"","x":1120,"y":860,"wires":[["deafd3825d135bf9"]]},{"id":"ba21ae9e59608571","type":"fs-list","z":"145ff57cd2c491cc","name":"","path":"path","pathType":"msg","pattern":"*","patternType":"str","filter":"files","recursive":false,"follow":true,"property":"list","propertyType":"msg","x":320,"y":860,"wires":[["bef468905228d1d1"]]},{"id":"94d8bb01a990b555","type":"inject","z":"145ff57cd2c491cc","name":"path","props":[{"p":"path","v":"/home/USER/folder","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","x":150,"y":860,"wires":[["ba21ae9e59608571"]]},{"id":"bef468905228d1d1","type":"split","z":"145ff57cd2c491cc","name":"","splt":"\\n","spltType":"str","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"","property":"list","x":490,"y":860,"wires":[["d52f9bf4dbd26ab8"]]},{"id":"d52f9bf4dbd26ab8","type":"function","z":"145ff57cd2c491cc","name":"set paths","func":"const fname = msg.list\n\n// https://stackoverflow.com/questions/73500403/get-file-name-and-file-type-using-split-function-in-javascript\nconst lastDot = fname.lastIndexOf('.'); // exactly what it says on the tin\nconst name = fname.slice(0, lastDot); // characters from the start to the last dot\nconst extension = fname.slice(lastDot + 1); // characters after the last dot\n\nconst new_fname = `${name}_new.${extension}`\n\nlet fs = {\n    source : msg.path,\n    file_source: fname,\n    destination : `${msg.path}/archive1`,\n    file_destiation: `${new_fname}`,\n}\nmsg.fs = fs\nreturn msg;","outputs":1,"timeout":0,"noerr":0,"initialize":"","finalize":"","libs":[],"x":940,"y":860,"wires":[["5fc40d6d5d98a4ab"]]},{"id":"deafd3825d135bf9","type":"debug","z":"145ff57cd2c491cc","name":"debug 57","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":1280,"y":860,"wires":[]},{"id":"114937dff4a9b575","type":"comment","z":"145ff57cd2c491cc","name":"change path","info":"","x":130,"y":800,"wires":[]},{"id":"433046eb654af1ea","type":"comment","z":"145ff57cd2c491cc","name":"change fs options here","info":"","x":320,"y":800,"wires":[]},{"id":"40c17883856ff080","type":"comment","z":"145ff57cd2c491cc","name":"add nodes here","info":"","x":680,"y":800,"wires":[]}]
3

Thank you, i'll try that in the afternoon.

4

Alternatively, since you can now use npm modules in function nodes, you could use Node.js's native modules to read the file list, split into separate messages (split node) and then process each file.

5

I can see the advantages in being able to trigger a flow from the Linux command line, passing a filename or other parameters.

for file in abc202505*
do
triggerfilehandlerflow $file
done

It's probably possible, or if not, it should be!

6

You can do that using a command line that writes to MQTT which node-red would pick up.

7

That's an option yes.
You could send an email too, or place a file in a directory that Node-red watches.

I'm imagining a more direct communication between Bash and Node-red, analogous to how you can include python, sed, awk etc in a pipeline.
I suppose that to use it like that, Node-red would have to be able to act as a filter - read from stdin, write to stdout.
Maybe there's something like that in an API?

8

When you run python, awk etc in a command window you run that s/w as part of the bash process, or as a spawned process. Assuming that Node red is already running as another process you need some sort of inter-process comms, such as an http api, or other methods already suggested.

9

Boring update day so... just for future reference :sweat_smile:

grafik
grafik1042×219 14.2 KB

[{"id":"8651173a2beba2a6","type":"function","z":"d205ec506260c61e","name":"read file","func":"fs.readFile(msg.files.path, 'utf8', (err, data) => {\n  if (err) {\n    //node.warn(err);\n    node.status({fill:\"red\",shape:\"dot\",text:\"Error reading file: \" + msg.files.path});\n    return\n  } \n  msg.payload = data;\n  node.send(msg);\n  node.status({fill:\"green\",shape:\"dot\",text:\"Done\"});\n  node.done();\n\n});\nreturn\n\n","outputs":1,"timeout":0,"noerr":0,"initialize":"","finalize":"","libs":[{"var":"fs","module":"fs"}],"x":860,"y":640,"wires":[["0329db9f676a6193"]]},{"id":"4875781a96d4bca9","type":"function","z":"d205ec506260c61e","name":"read dir","func":"fs.readdir(msg.path,{withFileTypes: true}, function (err, files) {\n    //handling error\n    if (err) {\n        //node.warn('Unable to scan directory: ' + err);\n        node.status({fill:\"red\",shape:\"ring\",text:'Unable to scan directory: ' + err});\n        return;\n    } \n    msg.files = files.filter((dirent) => dirent.isFile());\n    node.send(msg);\n    node.status({fill:\"green\",shape:\"dot\",text:\"Done\"});\n    node.done();\n});\nreturn\n\n\n\n\n","outputs":1,"timeout":0,"noerr":0,"initialize":"","finalize":"","libs":[{"var":"fs","module":"fs"},{"var":"path","module":"path"}],"x":400,"y":620,"wires":[["513a08f4be7f06eb","dfdc981a700d8042"]]},{"id":"b8a6979723d2ddc3","type":"inject","z":"d205ec506260c61e","name":"path","props":[{"p":"path","v":"/home/USER/test/","vt":"str"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","x":210,"y":620,"wires":[["4875781a96d4bca9"]]},{"id":"ebe1a7751aeedbd9","type":"split","z":"d205ec506260c61e","name":"","splt":"\\n","spltType":"str","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"","property":"files","x":690,"y":640,"wires":[["8651173a2beba2a6"]]},{"id":"ac9bf89eaeb65c23","type":"function","z":"d205ec506260c61e","name":"mv","func":"\nfor (let index = 0; index < msg.files.length; index++) {\n\n  const name = msg.files[index].name\n  const parentPath = msg.files[index].parentPath\n  const sourceFile = msg.files[index].path\n  const targetDir = path.join(parentPath, \"archive\");\n  const targetFile = path.join(targetDir, name);\n\n  fs.mkdir(targetDir, { recursive: true }, (err) => {\n    if (err) node.warn(err);\n    return;\n  });\n\n\n  fs.rename(sourceFile, targetFile, (err) => {\n    if (err) {\n      node.status({fill:\"red\",shape:\"dot\",text:\"Error moving files: \" + err});\n    } else {\n      node.status({fill:\"green\",shape:\"dot\",text:\"Done\"});\n    }  \n  });\n\n  node.done\n  \n}\nreturn","outputs":1,"timeout":0,"noerr":0,"initialize":"","finalize":"","libs":[{"var":"fs","module":"fs"},{"var":"path","module":"path"}],"x":690,"y":580,"wires":[[]]},{"id":"0329db9f676a6193","type":"debug","z":"d205ec506260c61e","name":"debug 8","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":1020,"y":640,"wires":[]},{"id":"dfdc981a700d8042","type":"debug","z":"d205ec506260c61e","name":"debug 9","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":620,"y":720,"wires":[]},{"id":"af8012b919ce775c","type":"comment","z":"d205ec506260c61e","name":"set path here!","info":"","x":210,"y":580,"wires":[]},{"id":"513a08f4be7f06eb","type":"junction","z":"d205ec506260c61e","x":540,"y":620,"wires":[["ebe1a7751aeedbd9"]]}]
10

I already have this. I use both MQTT output and, for some tasks, a webhook provided by Node-RED.

MQTT example:

#! /usr/bin/env bash

# Redirect stdout to syslog -
#   show with: `sudo journalctl -t nrmain-backup`
#   or `sudo cat /var/log/syslog | grep nrmain-backup`
exec 1> >(logger -t nrmain-backup -p local0.info)
# redirect stderr to syslog
exec 2> >(logger -t nrmain-backup -p local0.err)

# ... do interesting stuff ...

# Validate return code
# 0 = no error,
# 24 is fine, happens when files are being touched during sync (logs etc)
# all other codes are fatal -- see man (1) rsync
# You can output the result to MQTT or to a Node-RED http-in endpoint
if ! [ $? = 24 -o $? = 0 ] ; then
        echo "Fatal: Node-RED daily backup finished with errors!"
    #curl --insecure -I 'https://localhost:1880/nrnotify?type=backup&schedule=daily&result=fail'
    mosquitto_pub -r -t services/$NR_SERVICE_NAME/backup/daily/fail -m $ENDDATE
else
    echo "Finished Node-RED daily backup, no errors."
    #curl --insecure -I 'https://localhost:1880/nrnotify?type=backup&schedule=daily&result=success'
    mosquitto_pub -r -t services/$NR_SERVICE_NAME/backup/daily/success -m $ENDDATE
fi

# Sync disks to make sure data is written to disk
sync

#EOF

Node-RED webhook example:

#! /usr/bin/env bash
# Fast scan the local network for live devices and record
# to /tmp/nmap.xml which can be used in Node-RED
#

# Run the scan
nmap -sn --oX /tmp/nmap.xml --privileged -R --system-dns --webxml 192.168.1.0/24
# Make sure ownership & ACLs on the output are secure
chown root:home /tmp/nmap.xml
chmod --silent 640 /tmp/nmap.xml
# Trigger the Node-RED update
curl -I 'http://localhost:1880/localnetscan'

For the webhook example, I've left Node-RED to read in the XML file which can be fairly large. I have other bash scripts where I pass useful output via the curl query parameters, as always, there are lots of options.

11

Thanks a lot, both flows seem to work. But let me add a question about timing: the split function sends out a bunch of messages. What, if the processing takes longer than the gap between the messages is? Will i lose messages? As i said i have ~500 files to process. And these are json files that must be separated and two different output files are assembled.

12

Tested with 50 files. Works as expected. Many thanks!