My Data logging project

Just figured I'd share this because I'm constantly trying to improve and figure out better and more compact ways of doing things in node-red.

This project is reading data from a struct in a PLC, and storing it in a pre-existing or not-yet created file in our server location.

My background is largely PLC, which is probably reflected in the way my flow is structured. Anyway, let me know what you think and anything that could possibly be improved upon.

[{"id":"be33617967535245","type":"tab","label":"Data Logging Flow","disabled":false,"info":"","env":[]},{"id":"e1f77484f1398983","type":"group","z":"be33617967535245","name":"subscribe to tags on OPC UA server ","style":{"stroke":"#000000","fill":"#dbcbe7","label":true,"color":"#000000"},"nodes":["2cfed632a35b57f7","38e10aa7efcc32f9","f3740202f1097a2f","f7cf5a20ab1e64d9","2423aadab5c442fe","5284b6fa15e3d699","fda1cf78ac0b5e38","8f210ae7a921ded9","c29f4a583d848ee9"],"x":14,"y":19,"w":1112,"h":162},{"id":"758d9330ecd17308","type":"group","z":"be33617967535245","name":"Read Contents of Directory Location, search for a file name, and indicate if found or not","style":{"fill":"#ffefbf","label":true,"color":"#000000"},"nodes":["79f5feacf2296487","b59039df1cf89645","cc13f3f3bcbd6a60","ae4f0de5617b4b18","cf295c2979011f65"],"x":274,"y":199,"w":932,"h":142},{"id":"83e38814f7fd6408","type":"group","z":"be33617967535245","name":"Monitor for file read/write errors and write to XM if one occurs.....This bit must be reset on the PLC side","style":{"stroke":"#000000","fill":"#ffbfbf","label":true,"color":"#000000"},"nodes":["070b9a99b2a05dce","26082379e1547ade","f9791657e6e9a56e","cb3c1d3086075d5c"],"x":74,"y":631.5,"w":652,"h":97},{"id":"6e5c0f1f3caf5084","type":"group","z":"be33617967535245","name":"If data Log Start is true, then we read the tags of interest, and check if the file already exists or not, and then either create a new file or add lines to existing file","style":{"stroke":"#000000","fill":"#e3f3d3","label":true,"color":"#000000"},"nodes":["991ecb25622d7584","fb037f17dbba58ac","290b5004492db9cc","a0a2cd7de6e0f761","3b389c5506e8272b","8aeef9f76e222843","2564af11c42ffdc7","c44cc646337b17a2","563fecd3efce5e23","4ac1b747f007160b","4956bc67b973012f","6301fe49310681cf","b331b303a6aaeccb","1899dcfa3763406c","8e54f7c1bf1c4020","e568a31fe31fafc4","167a5c7a2dba4865"],"x":74,"y":351.5,"w":1152,"h":269.5},{"id":"8e54f7c1bf1c4020","type":"junction","z":"be33617967535245","g":"6e5c0f1f3caf5084","x":340,"y":440,"wires":[["563fecd3efce5e23"]]},{"id":"e568a31fe31fafc4","type":"junction","z":"be33617967535245","g":"6e5c0f1f3caf5084","x":1020,"y":420,"wires":[["8e54f7c1bf1c4020"]]},{"id":"167a5c7a2dba4865","type":"junction","z":"be33617967535245","g":"6e5c0f1f3caf5084","x":340,"y":400,"wires":[["290b5004492db9cc"]]},{"id":"da9bae50f546abc9","type":"junction","z":"be33617967535245","x":60,"y":380,"wires":[["fb037f17dbba58ac"]]},{"id":"c7ea93bb6809cd5c","type":"inject","z":"be33617967535245","name":"recurring 2 sec","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"2","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":120,"y":220,"wires":[["79f5feacf2296487","da9bae50f546abc9"]]},{"id":"991ecb25622d7584","type":"debug","z":"be33617967535245","g":"6e5c0f1f3caf5084","name":"debug 8","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":1120,"y":400,"wires":[]},{"id":"fb037f17dbba58ac","type":"switch","z":"be33617967535245","g":"6e5c0f1f3caf5084","name":"is DataLogStart true?","property":"DataLogStart","propertyType":"flow","rules":[{"t":"true"}],"checkall":"true","repair":false,"outputs":1,"x":200,"y":400,"wires":[["167a5c7a2dba4865"]]},{"id":"2cfed632a35b57f7","type":"OpcUa-Client","z":"be33617967535245","g":"e1f77484f1398983","endpoint":"f36d8d26eafdcc32","action":"subscribe","deadbandtype":"a","deadbandvalue":1,"time":"500","timeUnit":"ms","certificate":"n","localfile":"","localkeyfile":"","securitymode":"None","securitypolicy":"None","useTransport":false,"maxChunkCount":1,"maxMessageSize":8192,"receiveBufferSize":8192,"sendBufferSize":8192,"name":"Subscribe to OPC UA","x":520,"y":80,"wires":[["f3740202f1097a2f","5284b6fa15e3d699","8f210ae7a921ded9"],[],[]]},{"id":"38e10aa7efcc32f9","type":"OpcUa-Item","z":"be33617967535245","g":"e1f77484f1398983","item":"ns=2;s=Application.HandshakeGVL.DataLogStart","datatype":"Boolean","value":"","name":"DataLogStart","x":300,"y":60,"wires":[["2cfed632a35b57f7"]]},{"id":"f3740202f1097a2f","type":"debug","z":"be33617967535245","g":"e1f77484f1398983","name":"debug 9","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":740,"y":60,"wires":[]},{"id":"f7cf5a20ab1e64d9","type":"inject","z":"be33617967535245","g":"e1f77484f1398983","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":true,"onceDelay":"1","topic":"","payload":"multiple","payloadType":"str","x":120,"y":60,"wires":[["38e10aa7efcc32f9","fda1cf78ac0b5e38"]]},{"id":"2423aadab5c442fe","type":"function","z":"be33617967535245","g":"e1f77484f1398983","name":"Create flow variable for DataLogStart","func":"flow.set('DataLogStart', msg.payload);\nreturn msg;","outputs":1,"timeout":0,"noerr":0,"initialize":"","finalize":"","libs":[],"x":950,"y":100,"wires":[[]]},{"id":"290b5004492db9cc","type":"OpcUa-Item","z":"be33617967535245","g":"6e5c0f1f3caf5084","item":"ns=2;s=Application.DataLoggingGVL.DataLog","datatype":"Extension Object","value":"","name":"Indicate the tags we want to read from the XM","x":540,"y":400,"wires":[["a0a2cd7de6e0f761"]]},{"id":"a0a2cd7de6e0f761","type":"OpcUa-Client","z":"be33617967535245","g":"6e5c0f1f3caf5084","endpoint":"f36d8d26eafdcc32","action":"read","deadbandtype":"a","deadbandvalue":1,"time":10,"timeUnit":"s","certificate":"n","localfile":"","localkeyfile":"","securitymode":"None","securitypolicy":"None","useTransport":false,"maxChunkCount":1,"maxMessageSize":8192,"receiveBufferSize":8192,"sendBufferSize":8192,"name":"Read the tags from the XM","x":860,"y":400,"wires":[["991ecb25622d7584","e568a31fe31fafc4"],[],[]]},{"id":"5284b6fa15e3d699","type":"switch","z":"be33617967535245","g":"e1f77484f1398983","name":"","property":"topic","propertyType":"msg","rules":[{"t":"eq","v":"ns=2;s=Application.HandshakeGVL.DataLogStart","vt":"str"}],"checkall":"true","repair":false,"outputs":1,"x":730,"y":100,"wires":[["2423aadab5c442fe"]]},{"id":"cc13f3f3bcbd6a60","type":"function","z":"be33617967535245","g":"758d9330ecd17308","name":"Search for a specific file in the array, and indicate if its found or not in the flow variable \"found\"","func":"var newMsg = {};\nvar fileName = flow.get('fileName');\nfor(var i = 0; i < msg.payload.length; i++) {\n    \n    if (msg.payload[i] == fileName) {\n        newMsg.payload = 'found file';\n        flow.set('found', true);\n        i = msg.payload.length;\n    }\n    else {\n        newMsg.payload = 'file NOT found';\n        flow.set('found', false);\n    }\n}\n\nreturn newMsg;\n","outputs":1,"timeout":0,"noerr":0,"initialize":"","finalize":"","libs":[],"x":650,"y":300,"wires":[["ae4f0de5617b4b18"]]},{"id":"ae4f0de5617b4b18","type":"debug","z":"be33617967535245","g":"758d9330ecd17308","name":"debug 10","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":1100,"y":300,"wires":[]},{"id":"79f5feacf2296487","type":"fs-file-lister","z":"be33617967535245","g":"758d9330ecd17308","name":"","start":"H:/","pattern":"*.*","folders":"*","hidden":true,"lstype":"files","path":false,"single":false,"depth":0,"stat":false,"showWarnings":true,"x":360,"y":240,"wires":[["b59039df1cf89645"]]},{"id":"b59039df1cf89645","type":"join","z":"be33617967535245","g":"758d9330ecd17308","name":"Join files found in directory into an array","mode":"custom","build":"array","property":"payload","propertyType":"msg","key":"topic","joiner":"\\n","joinerType":"str","accumulate":false,"timeout":".2","count":"","reduceRight":false,"reduceExp":"","reduceInit":"","reduceInitType":"num","reduceFixup":"","x":660,"y":240,"wires":[["cc13f3f3bcbd6a60","cf295c2979011f65"]]},{"id":"3b389c5506e8272b","type":"file","z":"be33617967535245","g":"6e5c0f1f3caf5084","name":"Add Line to existing file","filename":"filename","filenameType":"msg","appendNewline":true,"createDir":false,"overwriteFile":"false","encoding":"none","x":810,"y":520,"wires":[["b331b303a6aaeccb"]]},{"id":"8aeef9f76e222843","type":"csv","z":"be33617967535245","g":"6e5c0f1f3caf5084","name":"its NOT","spec":"rfc","sep":",","hdrin":true,"hdrout":"all","multi":"one","ret":"\\n","temp":"dateFromVR,timeFromVR,assemblyNumber,workOrderNumber,serial,flm1Feedback,pT1Feedback,,tT1Feedback,flM2Feedback,pT2Feedback,tT2Feedback,ISO_4_DISPLAY,ISO_6_DISPLAY,ISO_14_DISPLAY,ISO_21_DISPLAY,RPT1H_scaled,RPT2H_scaled,RPT1L_scaled,RPT2L_scaled,ambientSound,rightSideSound,leftSideSound,frontSideSound,backSideSound,drive1HiResCurrentReal,drive2HiResCurrentReal,drive4HiResCurrentReal,drive5HiResCurrentReal,drive6HiResCurrentReal,drive7HiResCurrentReal,drive8HiResCurrentReal,drive9HiResCurrentReal,drive10HiResCurrentReal,drive11HiResCurrentReal,drive12HiResCurrentReal,drive13HiResCurrentReal,drive1_actual_freq,drive2_actual_freq,drive4_actual_freq,drive5_actual_freq,drive6_actual_freq,drive7_actual_freq,drive8_actual_freq,drive9_actual_freq,drive10_actual_freq,drive11_actual_freq,drive12_actual_freq,drive13_actual_freq","skip":"0","strings":true,"include_empty_strings":"","include_null_values":"","x":620,"y":580,"wires":[["2564af11c42ffdc7"]]},{"id":"2564af11c42ffdc7","type":"file","z":"be33617967535245","g":"6e5c0f1f3caf5084","name":"New File","filename":"filename","filenameType":"msg","appendNewline":false,"createDir":false,"overwriteFile":"true","encoding":"none","x":780,"y":580,"wires":[["1899dcfa3763406c"]]},{"id":"c44cc646337b17a2","type":"csv","z":"be33617967535245","g":"6e5c0f1f3caf5084","name":"It Is","spec":"rfc","sep":",","hdrin":false,"hdrout":"none","multi":"one","ret":"\\n","temp":"dateFromVR,timeFromVR,assemblyNumber,workOrderNumber,serial,flm1Feedback,pT1Feedback,,tT1Feedback,flM2Feedback,pT2Feedback,tT2Feedback,ISO_4_DISPLAY,ISO_6_DISPLAY,ISO_14_DISPLAY,ISO_21_DISPLAY,RPT1H_scaled,RPT2H_scaled,RPT1L_scaled,RPT2L_scaled,ambientSound,rightSideSound,leftSideSound,frontSideSound,backSideSound,drive1HiResCurrentReal,drive2HiResCurrentReal,drive4HiResCurrentReal,drive5HiResCurrentReal,drive6HiResCurrentReal,drive7HiResCurrentReal,drive8HiResCurrentReal,drive9HiResCurrentReal,drive10HiResCurrentReal,drive11HiResCurrentReal,drive12HiResCurrentReal,drive13HiResCurrentReal,drive1_actual_freq,drive2_actual_freq,drive4_actual_freq,drive5_actual_freq,drive6_actual_freq,drive7_actual_freq,drive8_actual_freq,drive9_actual_freq,drive10_actual_freq,drive11_actual_freq,drive12_actual_freq,drive13_actual_freq","skip":"0","strings":true,"include_empty_strings":false,"include_null_values":false,"x":610,"y":520,"wires":[["3b389c5506e8272b"]]},{"id":"563fecd3efce5e23","type":"function","z":"be33617967535245","g":"6e5c0f1f3caf5084","name":"assign filename to the msg , and format decimal values to 2 decimal places","func":"//assign the file name\nmsg.filename = \"H:/\" + msg.payload.serial + \".csv\";\n//transducers 1\nmsg.payload.flm1Feedback = msg.payload.flm1Feedback.toFixed(2);\nmsg.payload.pT1Feedback = msg.payload.pT1Feedback.toFixed(2);\nmsg.payload.tT1Feedback = msg.payload.tT1Feedback.toFixed(2);\n//transducers 2\nmsg.payload.flM2Feedback = msg.payload.flM2Feedback.toFixed(2);\nmsg.payload.pT2Feedback = msg.payload.pT2Feedback.toFixed(2);\nmsg.payload.tT2Feedback = msg.payload.tT2Feedback.toFixed(2);\n//iso values\nmsg.payload.ISO_4_DISPLAY = msg.payload.ISO_4_DISPLAY.toFixed(2);\nmsg.payload.ISO_6_DISPLAY = msg.payload.ISO_6_DISPLAY.toFixed(2);\nmsg.payload.ISO_14_DISPLAY = msg.payload.ISO_14_DISPLAY.toFixed(2);\nmsg.payload.ISO_21_DISPLAY = msg.payload.ISO_21_DISPLAY.toFixed(2);\n//remote transducers\nmsg.payload.RPT1H_scaled = msg.payload.RPT1H_scaled.toFixed(2);\nmsg.payload.RPT2H_scaled = msg.payload.RPT2H_scaled.toFixed(2);\nmsg.payload.RPT1L_scaled = msg.payload.RPT1L_scaled.toFixed(2);\nmsg.payload.RPT2L_scaled = msg.payload.RPT2L_scaled.toFixed(2);\n//drive currents\nmsg.payload.drive1HiResCurrentReal = msg.payload.drive1HiResCurrentReal.toFixed(2);\nmsg.payload.drive2HiResCurrentReal = msg.payload.drive2HiResCurrentReal.toFixed(2);\nmsg.payload.drive4HiResCurrentReal = msg.payload.drive4HiResCurrentReal.toFixed(2);\nmsg.payload.drive5HiResCurrentReal = msg.payload.drive5HiResCurrentReal.toFixed(2);\nmsg.payload.drive6HiResCurrentReal = msg.payload.drive6HiResCurrentReal.toFixed(2);\nmsg.payload.drive7HiResCurrentReal = msg.payload.drive7HiResCurrentReal.toFixed(2);\nmsg.payload.drive8HiResCurrentReal = msg.payload.drive8HiResCurrentReal.toFixed(2);\nmsg.payload.drive9HiResCurrentReal = msg.payload.drive9HiResCurrentReal.toFixed(2);\nmsg.payload.drive10HiResCurrentReal = msg.payload.drive10HiResCurrentReal.toFixed(2);\nmsg.payload.drive11HiResCurrentReal = msg.payload.drive11HiResCurrentReal.toFixed(2);\nmsg.payload.drive12HiResCurrentReal = msg.payload.drive12HiResCurrentReal.toFixed(2);\nmsg.payload.drive13HiResCurrentReal = msg.payload.drive13HiResCurrentReal.toFixed(2);\n\nreturn msg;","outputs":1,"timeout":0,"noerr":0,"initialize":"","finalize":"","libs":[],"x":670,"y":460,"wires":[["4ac1b747f007160b","6301fe49310681cf"]]},{"id":"4ac1b747f007160b","type":"switch","z":"be33617967535245","g":"6e5c0f1f3caf5084","name":"Is the log file found in the directory already?","property":"found","propertyType":"flow","rules":[{"t":"true"},{"t":"false"}],"checkall":"true","repair":false,"outputs":2,"x":310,"y":540,"wires":[["c44cc646337b17a2"],["8aeef9f76e222843"]]},{"id":"4956bc67b973012f","type":"inject","z":"be33617967535245","g":"6e5c0f1f3caf5084","name":"Manual start","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":190,"y":440,"wires":[["167a5c7a2dba4865"]]},{"id":"6301fe49310681cf","type":"debug","z":"be33617967535245","g":"6e5c0f1f3caf5084","name":"debug 12","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":1120,"y":460,"wires":[]},{"id":"b331b303a6aaeccb","type":"debug","z":"be33617967535245","g":"6e5c0f1f3caf5084","name":"debug 13","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":1120,"y":520,"wires":[]},{"id":"1899dcfa3763406c","type":"debug","z":"be33617967535245","g":"6e5c0f1f3caf5084","name":"debug 14","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":1120,"y":580,"wires":[]},{"id":"cf295c2979011f65","type":"debug","z":"be33617967535245","g":"758d9330ecd17308","name":"debug 16","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":1100,"y":240,"wires":[]},{"id":"fda1cf78ac0b5e38","type":"OpcUa-Item","z":"be33617967535245","g":"e1f77484f1398983","item":"ns=2;s=Application.DataLoggingGVL.DataLog.Serial","datatype":"String","value":"","name":"Serial","x":270,"y":100,"wires":[["2cfed632a35b57f7"]]},{"id":"8f210ae7a921ded9","type":"switch","z":"be33617967535245","g":"e1f77484f1398983","name":"","property":"topic","propertyType":"msg","rules":[{"t":"eq","v":"ns=2;s=Application.DataLoggingGVL.DataLog.Serial","vt":"str"}],"checkall":"true","repair":false,"outputs":1,"x":730,"y":140,"wires":[["c29f4a583d848ee9"]]},{"id":"c29f4a583d848ee9","type":"function","z":"be33617967535245","g":"e1f77484f1398983","name":"Create flow variable for fileName","func":"flow.set('fileName', msg.payload + \".csv\");\nreturn msg;","outputs":1,"timeout":0,"noerr":0,"initialize":"","finalize":"","libs":[],"x":940,"y":140,"wires":[[]]},{"id":"070b9a99b2a05dce","type":"catch","z":"be33617967535245","g":"83e38814f7fd6408","name":"","scope":["79f5feacf2296487","3b389c5506e8272b","2564af11c42ffdc7"],"uncaught":false,"x":150,"y":680,"wires":[["26082379e1547ade"]]},{"id":"26082379e1547ade","type":"OpcUa-Item","z":"be33617967535245","g":"83e38814f7fd6408","item":"ns=2;s=Application.HandshakeGVL.DataLogError","datatype":"Boolean","value":"true","name":"","x":290,"y":680,"wires":[["f9791657e6e9a56e"]]},{"id":"f9791657e6e9a56e","type":"OpcUa-Client","z":"be33617967535245","g":"83e38814f7fd6408","endpoint":"f36d8d26eafdcc32","action":"write","deadbandtype":"a","deadbandvalue":1,"time":10,"timeUnit":"s","certificate":"n","localfile":"","localkeyfile":"","securitymode":"None","securitypolicy":"None","useTransport":false,"maxChunkCount":1,"maxMessageSize":8192,"receiveBufferSize":8192,"sendBufferSize":8192,"name":"","x":460,"y":680,"wires":[["cb3c1d3086075d5c"],[],[]]},{"id":"cb3c1d3086075d5c","type":"debug","z":"be33617967535245","g":"83e38814f7fd6408","name":"debug 17","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":620,"y":680,"wires":[]},{"id":"f36d8d26eafdcc32","type":"OpcUa-Endpoint","endpoint":"Opc.tcp://10.33.90.4:4840","secpol":"None","secmode":"None","none":true,"login":false,"usercert":false,"usercertificate":"","userprivatekey":""}]```

The Write-file node, in append mode, will create a new file if it does not already exist.
So you can perhaps dispense with your switch and duplicated write-file nodes?

Indeed, can't your Opcua nodes which receive the data output directly to the write-file, via a change node or function to set up filename etc?

1 Like

Added detail:

I deployed this for production use right before the holidays. My colleague has been doing the PLC portion and I the data logging portion. There was no time limit implemented for the data logging functionality, and someone started it and left it running for 26hrs, so I checked the most recent file when I arrived back to work and we had one file which had 46,### entries.

Unintended test, but actually pretty good to know it will work.

1 Like

Thanks for the clarification about the write-file node. I am still using the switch because for one case I want the headers to be included, and the other not. I will test and verify, but this definitely will clean it up a little.

As far as using the change node vs function I'm not clear on what you're asking. I used the single function to assign the filename and format all of the float values that I receive so I'm not having 10 decimal places.

Hmm, you are certainly going to need either log rotation or some kind of clear-down on that :smile:

Added explanation about the flow's application:

We build assemblies and test them after initial construction so that we can provide customers with a sheet showing the setup when they receive it. The idea with the log is that in #1 we have records, and #2 if it ever ended up returning to us perhaps for refurbishment, we could test it again, and we would have records for that assembly basically for its lifetime. This is why I am taking the file name at the beginning, and then I search our server location to see if the name exists already.

If it does, append to that file. If it doesn't create a new file. There are something like 48 values that I log from the data struct, but hard to imagine all of them ever being used.

My colleague added some conditions which stop the logging function so ideally this won't occur again... (that said nature finds a way)

1 Like

You really would be better with some form of database rather than using files to store the data.

1 Like

I understand the logic of what you propose and agree in general, but not for this application.