How to append JSON to an array (to be stored in a text file)

[raspberry]

Hey folks,

I'm not a javascript wizard by any means so this has me thinking too hard. I want to keep a text file of on/off events that happen over time in an array of JSON strings, which can be loaded into Node-Red with a read file and displayed using the node-red-node-ui-table.

To make that happen, I know the text file has to look something like this:

[{"Sensor":"A","Event":"Online","Time":"1631746534420"},
{"Sensor":"B","Event":"Online","Time":"1631746661503"},
{"Sensor":"A","Event":"Offline","Time":"1631746661503"},
{"Sensor":"C","Event":"Online","Time":"1631746662088"}]

When a new event happens, I can generate one JSON just fine:
[{"Sensor":"C","Event":"Online","Time":"1631746662088"}]

But if I append this to my file, it of course messes up the bracket notation. How can I maintain a list with this structure and periodically add new events to it? I'm flexible to any strategy.

A simple example of what I'm trying to do:

[{"id":"190c8195.72b3be","type":"tab","label":"Flow 2","disabled":false,"info":""},{"id":"9cf2fd85.5495b","type":"ui_button","z":"190c8195.72b3be","name":"Load Data","group":"f5f08c2f.dc4f6","order":4,"width":"3","height":"1","passthru":false,"label":"Restore","tooltip":"","color":"","bgcolor":"","icon":"","payload":"","payloadType":"str","topic":"","topicType":"str","x":210,"y":300,"wires":[["305f7540.f2906a"]]},{"id":"cbcb713.7856f9","type":"file","z":"190c8195.72b3be","name":"","filename":"{File Path}","appendNewline":true,"createDir":true,"overwriteFile":"false","encoding":"none","x":930,"y":200,"wires":[[]]},{"id":"305f7540.f2906a","type":"file in","z":"190c8195.72b3be","name":"","filename":"{File Path}","format":"utf8","x":390,"y":300,"wires":[["e28d58a4.2dd898"]]},{"id":"80da4778.f25c58","type":"json","z":"190c8195.72b3be","name":"","property":"payload","action":"","pretty":false,"x":770,"y":200,"wires":[["cbcb713.7856f9"]]},{"id":"e28d58a4.2dd898","type":"json","z":"190c8195.72b3be","name":"","property":"payload","action":"","pretty":false,"x":830,"y":300,"wires":[["79f9266b.4d8148"]]},{"id":"67aac911.bfc1a8","type":"inject","z":"190c8195.72b3be","name":"Simulate Sensor Online","props":[{"p":"payload"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"1","payloadType":"num","x":220,"y":200,"wires":[["b879df26.f75e7"]]},{"id":"4c84ef9d.f506f","type":"template","z":"190c8195.72b3be","name":"","field":"payload","fieldType":"msg","format":"handlebars","syntax":"mustache","template":"[{\"Sensor\": \"{{sensor}}\",\"Event\": \"{{status}}\",\"Time\": \"{{eventtime}}\"}]","output":"json","x":620,"y":200,"wires":[["80da4778.f25c58"]]},{"id":"b879df26.f75e7","type":"change","z":"190c8195.72b3be","name":"","rules":[{"t":"set","p":"sensor","pt":"msg","to":"A","tot":"str"},{"t":"set","p":"status","pt":"msg","to":"Online","tot":"str"},{"t":"set","p":"eventtime","pt":"msg","to":"","tot":"date"}],"action":"","property":"","from":"","to":"","reg":false,"x":440,"y":200,"wires":[["4c84ef9d.f506f"]]},{"id":"79f9266b.4d8148","type":"ui_table","z":"190c8195.72b3be","group":"87ae6969.0fdaa8","name":"","order":4,"width":0,"height":0,"columns":[],"outputs":0,"cts":false,"x":970,"y":300,"wires":[]},{"id":"f5f08c2f.dc4f6","type":"ui_group","name":"Group 1","tab":"83eea11f.48dd2","order":1,"disp":true,"width":"6","collapse":false},{"id":"87ae6969.0fdaa8","type":"ui_group","name":"Group 2","tab":"83eea11f.48dd2","order":2,"disp":true,"width":6},{"id":"83eea11f.48dd2","type":"ui_tab","name":"Home","icon":"dashboard","disabled":false,"hidden":false}]

[dceejay]

Hi, if logging to a file then don't try to save it as an array. Just save each "event" json to a line in the file. Then read it back in (one msg per line) and push to an array at that point.

[raspberry]

Thanks @dceejay, that sounds like the simplest approach. I'm still getting a sense of how powerful the JS side of node-red can be. At the moment, array is loaded into Node-Red, then a function node pushes the new event object to the array, then the array is re-written back into the file (via overwrite), but I'm not comfortable with overwriting the file each time a new event happens. I had a feeling it was unnecessary.

[dceejay]

Yes - that will get inefficient really quickly if the array grows. Much better just to append as you go (and let the file node add the new line) - then rebuild the array as and when you read it all back in...

[raspberry]

@dceejay Now that I have this working, a broader question has come up. How does Node-Red deal with (if at all) a race condition for a file read/write? For instance, if I have 5 sensors in 5 flows, and they are going to write to my "Events" file on disk whenever they go offline, I can see an issue if they all go offline together, as now 5 file-write nodes are trying to append to the same file at the same time. Does this require some kind of buffer? Separate files?

[dceejay]

node.js itself is single threaded so in reality they are never exactly at the same time - they will always take turns... so there should be no problem.

[Colin]

@dceejay

they will always take turns... so there should be no problem.

Is not file writing asynchronous? If there are multiple file nodes all writing to the same file will not the asynchronous nature of them cause overlapping?

@raspberry whether there is a possible race condition or not, in that situation I would feed all the requests to one file node so they will be handled serially. It just seems the right way to do it.