Order serial data for csv

[xsilvergs]

This is my first project and age goes against me but I'm trying to learn node-red and use it in this project.

This project is to collect data from Victron equipment and other brands in the future.

There are 2 USB serial devices independently outputting data every second. The data needs to be saved to possibly a csv file every minute (there may be better alternatives to csv).

The serial data needs to be put in order i.e. usb1 followed by usb2 and saved, a minute later the data needs to be saved on the next row in the same order.

I have read, tried and watched YouTube for the answer but not managed to solve my problem so please help.

How do I order the data from the usb's (MPPT - BMV) and save it to csv?

[{"id":"43fa7db7.c87434","type":"tab","label":"ChargerLogger","disabled":false,"info":""},{"id":"d086478.7242db8","type":"function","z":"43fa7db7.c87434","name":"ExtractDataBMV712","func":"var data = msg.payload.split(\"\\t\");\nswitch(data[0]) { \n        case    'V':\n            MainBatVoltage = parseFloat(data[1]/1000);\n            break;\n        case    'VS':\n            AuxBatVoltage = parseFloat(data[1]/1000);\n            break;\n        case    'I':\n            BatteryCurrent = parseFloat(data[1]/1000);\n            break;\n        case    'P':\n            InstantaneousPower = parseInt(data[1]);\n            break;\n        case    'CE':\n            ConsumedAmpHours = parseFloat(data[1]/1000);\n            break;\n        case    'SOC':\n            SOC = parseFloat(data[1]/10);\n            break;\n        case    'TTG':\n            TimeToGo = parseFloat(data[1]);\n            break;\n        case    'Alarm':\n            AlarmCon = String(data[1]);\n            break;\n        case    'AR':\n            AlarmReason = String(data[1]);\n            break;\n            \n                }\nreturn  [\n    {\n        payload: {\n                MainBatVoltage: MainBatVoltage,\n                AuxBatVoltage: AuxBatVoltage,\n                BatteryCurrent: BatteryCurrent,\n                InstantaneousPower: InstantaneousPower,\n                ConsumedAmpHours: ConsumedAmpHours,\n                SOC: SOC,\n                TimeToGo: TimeToGo,\n                AlarmCon: AlarmCon,\n                AlarmReason: AlarmReason\n                    }\n    }\n]\n","outputs":1,"noerr":0,"x":200,"y":620,"wires":[["dacd5295.cfc12","f16ad309.7178c","1ede5097.59799f"]]},{"id":"948b01ce.c3bc6","type":"ui_text","z":"43fa7db7.c87434","group":"3fcc4874.77b7f8","order":3,"width":0,"height":0,"name":"Leisure Battery SOC (BMV)","label":"SOC","format":"{{msg.payload.SOC | number:0}}","layout":"col-center","x":840,"y":120,"wires":[]},{"id":"92e69817.387bf8","type":"ui_text","z":"43fa7db7.c87434","group":"3fcc4874.77b7f8","order":1,"width":0,"height":0,"name":"Leisure Battery Volts (BMV)","label":"Volts","format":"{{msg.payload.MainBatVoltage | number:1}}","layout":"col-center","x":840,"y":40,"wires":[]},{"id":"c3bb298b.e9ab58","type":"ui_text","z":"43fa7db7.c87434","group":"3fcc4874.77b7f8","order":2,"width":0,"height":0,"name":"Leisure Battery Amps (BMV)","label":"Amps","format":"{{msg.payload.BatteryCurrent | number:1}}","layout":"col-center","x":840,"y":80,"wires":[]},{"id":"25c4feea.0d0ad2","type":"ui_text","z":"43fa7db7.c87434","group":"5f89a742.c39168","order":1,"width":0,"height":0,"name":"Starter Battery Volts (BMV)","label":"Volts","format":"{{msg.payload.AuxBatVoltage | number:1}}","layout":"col-center","x":840,"y":240,"wires":[]},{"id":"f0fb7a2d.18c178","type":"ui_text","z":"43fa7db7.c87434","group":"3fcc4874.77b7f8","order":3,"width":0,"height":0,"name":"Leisure Battery TTG (BMV)","label":"TTG","format":"{{msg.payload.TimeToGo}}","layout":"col-center","x":840,"y":160,"wires":[]},{"id":"3ed3337c.597fdc","type":"ui_chart","z":"43fa7db7.c87434","name":"Leisure Battery V (Web)","group":"ba16cc09.e9994","order":1,"width":0,"height":0,"label":"Volts","chartType":"line","legend":"false","xformat":"HH:mm","interpolate":"linear","nodata":"Some text","dot":false,"ymin":"10","ymax":"16","removeOlder":"6","removeOlderPoints":"","removeOlderUnit":"3600","cutout":0,"useOneColor":false,"colors":["#1f77b4","#aec7e8","#ff7f0e","#2ca02c","#98df8a","#d62728","#ff9896","#9467bd","#c5b0d5"],"useOldStyle":false,"outputs":1,"x":830,"y":380,"wires":[[]]},{"id":"3aaf9cef.f6a0f4","type":"function","z":"43fa7db7.c87434","name":"Leisure Battery V extract","func":"msg.payload = msg.payload.MainBatVoltage;\nreturn msg;\n","outputs":1,"noerr":0,"x":830,"y":340,"wires":[["3ed3337c.597fdc"]]},{"id":"709276ea.6d2418","type":"function","z":"43fa7db7.c87434","name":"Leisure Battery A extract","func":"msg.payload = msg.payload.BatteryCurrent;\nreturn msg;","outputs":1,"noerr":0,"x":830,"y":420,"wires":[["a31433d2.b0e57"]]},{"id":"a31433d2.b0e57","type":"ui_chart","z":"43fa7db7.c87434","name":"Leisure Battery Amps (Web)","group":"ba16cc09.e9994","order":2,"width":0,"height":0,"label":"Amps","chartType":"line","legend":"false","xformat":"HH:mm","interpolate":"linear","nodata":"","dot":false,"ymin":"-50","ymax":"50","removeOlder":"6","removeOlderPoints":"","removeOlderUnit":"3600","cutout":0,"useOneColor":false,"colors":["#1f77b4","#aec7e8","#ff7f0e","#2ca02c","#98df8a","#d62728","#ff9896","#9467bd","#c5b0d5"],"useOldStyle":false,"outputs":1,"x":840,"y":460,"wires":[[]]},{"id":"40d23927.6759a8","type":"ui_chart","z":"43fa7db7.c87434","name":"Leisure Battery SOC (Web)","group":"ba16cc09.e9994","order":3,"width":0,"height":0,"label":"SOC","chartType":"line","legend":"false","xformat":"HH:mm","interpolate":"linear","nodata":"","dot":false,"ymin":"0","ymax":"110","removeOlder":"6","removeOlderPoints":"","removeOlderUnit":"3600","cutout":0,"useOneColor":false,"colors":["#1f77b4","#aec7e8","#ff7f0e","#2ca02c","#98df8a","#d62728","#ff9896","#9467bd","#c5b0d5"],"useOldStyle":false,"outputs":1,"x":840,"y":540,"wires":[[]]},{"id":"9c4ad4ff.7d1518","type":"function","z":"43fa7db7.c87434","name":"Leisure Battery SOC extract","func":"msg.payload = msg.payload.SOC;\nreturn msg;","outputs":1,"noerr":0,"x":840,"y":500,"wires":[["40d23927.6759a8"]]},{"id":"11544f9c.8844","type":"function","z":"43fa7db7.c87434","name":"Starter Battery V extract","func":"msg.payload = msg.payload.AuxBatVoltage;\nreturn msg;","outputs":1,"noerr":0,"x":830,"y":580,"wires":[["9cd68c23.f3cd"]]},{"id":"9cd68c23.f3cd","type":"ui_chart","z":"43fa7db7.c87434","name":"Starter Battery Volts (Web)","group":"5ca09b48.127554","order":1,"width":0,"height":0,"label":"Volts","chartType":"line","legend":"false","xformat":"HH:mm","interpolate":"linear","nodata":"","dot":false,"ymin":"10","ymax":"16","removeOlder":"6","removeOlderPoints":"","removeOlderUnit":"3600","cutout":0,"useOneColor":false,"colors":["#1f77b4","#aec7e8","#ff7f0e","#2ca02c","#98df8a","#d62728","#ff9896","#9467bd","#c5b0d5"],"useOldStyle":false,"outputs":1,"x":840,"y":620,"wires":[[]]},{"id":"ddea5ea0.f3cd9","type":"function","z":"43fa7db7.c87434","name":"ExtractDataMPPT","func":"var data = msg.payload.split(\"\\t\");\nswitch(data[0]) { \n        case    'CS':\n            StateOfOperation = parseFloat(data[1]);\n            break;\n        case    'VPV':\n            PanelVoltage = parseFloat(data[1]/1000);\n            break;\n        case    'PPV':\n            PanelPower = parseFloat(data[1]);\n            break;\n        case    'I':\n            BatteryAmps = parseInt(data[1]/1000);\n            break;\n        /*case    'Relay':\n            RelayState = parseFloat(data[1]);\n            break;*/\n        case    'H20':\n            YieldToday = parseFloat(data[1]/100);\n            break;\n        case    'H21':\n            MaxPtoday = parseInt(data[1]);\n            break;\n        case    'H22':\n            YieldYesterday = parseFloat(data[1]/100);\n            break;\n        case    'H23':\n            MaxPyesterday = parseFloat(data[1]);\n            break;\n        case    'ERR':\n            ErrorCode = parseFloat(data[1]);\n            break;\n                }\nreturn  [\n    {\n        payload: {\n                StateOfOperation: StateOfOperation,\n                PanelVoltage: PanelVoltage,\n                PanelPower: PanelPower,\n                BatteryAmps: BatteryAmps,\n                //RelayState: RelayState,\n                YieldToday: YieldToday,\n                MaxPtoday: MaxPtoday,\n                YieldYesterday: YieldYesterday,\n                MaxPyesterday: MaxPyesterday,\n                ErrorCode: ErrorCode\n                    }\n    }\n]","outputs":1,"noerr":0,"x":210,"y":820,"wires":[["9a8f8c13.79f46","121fcb53.a578c5","89bef54.30ea808"]]},{"id":"ef00795c.8a1668","type":"function","z":"43fa7db7.c87434","name":"State of operation","func":"if(msg.payload.StateOfOperation === 0)\n    {msg.payload = \"OFF\"}\nelse if(msg.payload.StateOfOperation === 1)\n    {msg.payload = \"FAULT\"}\nelse if(msg.payload.StateOfOperation === 3)\n    {msg.payload = \"Bulk\"}\nelse if(msg.payload.StateOfOperation === 4)\n    {msg.payload = \"Absorption\"}\nelse if(msg.payload.StateOfOperation === 5)\n    {msg.payload = \"Float\"}\nelse\n    {msg.payload = \"\"}    //do nothing\nreturn msg;","outputs":1,"noerr":0,"x":810,"y":880,"wires":[["f4a08d8c.a0c5e","3f10599b.02e1c6"]]},{"id":"f4a08d8c.a0c5e","type":"ui_text","z":"43fa7db7.c87434","group":"7595a526.d16c9c","order":1,"width":0,"height":0,"name":"State of Operation (Web)","label":"State","format":"{{msg.payload}}","layout":"row-spread","x":830,"y":920,"wires":[]},{"id":"cec0dfaa.f18c4","type":"ui_text","z":"43fa7db7.c87434","group":"7595a526.d16c9c","order":2,"width":0,"height":0,"name":"Panel Voltage (Web)","label":"Panel Voltage","format":"{{msg.payload.PanelVoltage | number:1}}","layout":"row-spread","x":820,"y":1000,"wires":[]},{"id":"3eb1c80a.645c88","type":"ui_text","z":"43fa7db7.c87434","group":"7595a526.d16c9c","order":5,"width":0,"height":0,"name":"Panel Power W (Web)","label":"Panel Power W","format":"{{msg.payload.PanelPower}}","layout":"row-spread","x":820,"y":1080,"wires":[]},{"id":"e844d230.83367","type":"ui_text","z":"43fa7db7.c87434","group":"7595a526.d16c9c","order":6,"width":0,"height":0,"name":"Yield Today kWh (Web)","label":"Yield Today kWh","format":"{{msg.payload.YieldToday}}","layout":"row-spread","x":830,"y":1220,"wires":[]},{"id":"21458ff0.40b79","type":"ui_text","z":"43fa7db7.c87434","group":"7595a526.d16c9c","order":7,"width":0,"height":0,"name":"Max Power Today W (Web)","label":"Max Power Today W","format":"{{msg.payload.MaxPtoday}}","layout":"row-spread","x":840,"y":1260,"wires":[]},{"id":"fdf49e69.ed59d","type":"ui_text","z":"43fa7db7.c87434","group":"7595a526.d16c9c","order":8,"width":0,"height":0,"name":"Yield Yesterday (Web)","label":"Yield Yesterday","format":"{{msg.payload.YieldYesterday}}","layout":"row-spread","x":820,"y":1300,"wires":[]},{"id":"37914dd7.08d312","type":"ui_text","z":"43fa7db7.c87434","group":"7595a526.d16c9c","order":9,"width":0,"height":0,"name":"Max Power Yesterday (Web)","label":"Max Power Yesterday","format":"{{msg.payload.MaxPyesterday}}","layout":"row-spread","x":840,"y":1340,"wires":[]},{"id":"c15eaa16.b4da78","type":"ui_text","z":"43fa7db7.c87434","group":"7595a526.d16c9c","order":10,"width":0,"height":0,"name":"MPPT Error Code (Web)","label":"Error Code","format":"{{msg.payload}}","layout":"row-spread","x":1070,"y":1540,"wires":[]},{"id":"b97d829f.aabe9","type":"function","z":"43fa7db7.c87434","name":"MPPT Error Code","func":"if(msg.payload.ErrorCode === 0)\n    {msg.payload = \"NONE\"}\nelse if(msg.payload.ErrorCode === 2)\n    {msg.payload = \"Battery voltage too high\"}\nelse if(msg.payload.ErrorCode === 17)\n    {msg.payload = \"Charger temperature too high\"}\nelse if(msg.payload.ErrorCode === 18)\n    {msg.payload = \"Charger over current\"}\nelse if(msg.payload.StateOfOperation === 19)\n    {msg.payload = \"Charger current reversed\"}\nelse if(msg.payload.ErrorCode === 20)\n    {msg.payload = \"Bulk time limit exceeded\"}\nelse if(msg.payload.ErrorCode === 21)\n    {msg.payload = \"Current sensor issue (sensor bias/sensor broken)\"}\nelse if(msg.payload.ErrorCode === 26)\n    {msg.payload = \"Terminals overheated\"}\nelse if(msg.payload.ErrorCode === 33)\n    {msg.payload = \"Input voltage too high (solar panel)\"}\nelse if(msg.payload.ErrorCode === 34)\n    {msg.payload = \"Input current too high (solar panel)\"}\nelse if(msg.payload.ErrorCode === 38)\n    {msg.payload = \"Input shutdown (due to excessive battery voltage)\"}\nelse if(msg.payload.ErrorCode === 116)\n    {msg.payload = \"Factory calibration data lost\"}\nelse if(msg.payload.ErrorCode === 117)\n    {msg.payload = \"Invalid/incompatible firmware\"}\nelse if(msg.payload.ErrorCode === 119)\n    {msg.payload = \"User settings invalid\"}\nelse\n    {msg.payload = \"\"}    //do nothing\nreturn msg;","outputs":1,"noerr":0,"x":810,"y":1500,"wires":[["c15eaa16.b4da78","9115109.e3a33f"]]},{"id":"9d45cbcb.307928","type":"ui_text","z":"43fa7db7.c87434","group":"5f89a742.c39168","order":2,"width":0,"height":0,"name":"BMV Alarm (BMV)","label":"BMV Alarm","format":"{{msg.payload}}","layout":"row-spread","x":810,"y":280,"wires":[]},{"id":"93ff3a1e.c0abe8","type":"function","z":"43fa7db7.c87434","name":"Alarm Reason","func":"if(msg.payload.AlarmReason >= 1)\n    {msg.payload = \"See manual \" + AlarmReason}\nelse\n    {msg.payload = \"No Alarm\"}\n/*if(msg.payload.AlarmReason === 1)\n    {msg.payload = \"Low Voltage\"}\nelse if(msg.payload.AlarmReason === 2)\n    {msg.payload = \"High Voltage\"}\nelse if(msg.payload.AlarmReason === 4)\n    {msg.payload = \"Low SOC\"}\nelse if(msg.payload.AlarmReason === 8)\n    {msg.payload = \"Low Starter Voltage\"}\nelse if(msg.payload.AlarmReason === 16)\n    {msg.payload = \"High Starter Voltage\"}\nelse if(msg.payload.AlarmReason === 32)\n    {msg.payload = \"Low Temperature\"}\nelse if(msg.payload.AlarmReason === 64)\n    {msg.payload = \"High Temperature\"}\nelse if(msg.payload.AlarmReason === 128)\n    {msg.payload = \"Mid Voltage\"}\nelse\n    {msg.payload = \"\"}    //do nothing\n    */\nreturn msg;","outputs":1,"noerr":0,"x":440,"y":240,"wires":[["9d45cbcb.307928"]]},{"id":"61cfe391.8216bc","type":"function","z":"43fa7db7.c87434","name":"TTG infinite","func":"if(msg.payload.TimeToGo === -1)\n    {msg.payload.TimeToGo = \"Infinite\"}\nelse if(msg.payload.TimeToGo === \"---\")\n    {msg.payload.TimeToGo = \"---\"}\nreturn msg;","outputs":1,"noerr":0,"x":510,"y":160,"wires":[["f0fb7a2d.18c178"]]},{"id":"3f10599b.02e1c6","type":"ui_text","z":"43fa7db7.c87434","group":"b3c06fb5.d8985","order":1,"width":0,"height":0,"name":"State of Operation (BMV)","label":"State","format":"{{msg.payload}}","layout":"row-spread","x":870,"y":960,"wires":[]},{"id":"59b8eb7.829c214","type":"ui_text","z":"43fa7db7.c87434","group":"b3c06fb5.d8985","order":5,"width":0,"height":0,"name":"Panel Power W (BMV)","label":"Panel Watts","format":"{{msg.payload.PanelPower}}","layout":"row-spread","x":860,"y":1120,"wires":[]},{"id":"88fcc365.65155","type":"ui_text","z":"43fa7db7.c87434","group":"b3c06fb5.d8985","order":2,"width":0,"height":0,"name":"Panel Voltage (BMV)","label":"Panel Voltage","format":"{{msg.payload.PanelVoltage | number:1}}","layout":"row-spread","x":860,"y":1040,"wires":[]},{"id":"9115109.e3a33f","type":"ui_text","z":"43fa7db7.c87434","group":"b3c06fb5.d8985","order":5,"width":0,"height":0,"name":"MPPT Error Code (BMV)","label":"Error Code","format":"{{msg.payload}}","layout":"row-spread","x":1070,"y":1500,"wires":[]},{"id":"6b090dd6.5eb584","type":"serial in","z":"43fa7db7.c87434","name":"","serial":"b90a2efa.d1c1e","x":150,"y":680,"wires":[["d086478.7242db8"]]},{"id":"a3b2bb83.b1f9d8","type":"serial in","z":"43fa7db7.c87434","name":"","serial":"ee60460c.fb5fc8","x":150,"y":760,"wires":[["ddea5ea0.f3cd9"]]},{"id":"9086c4b0.e9a658","type":"ui_text","z":"43fa7db7.c87434","group":"b3c06fb5.d8985","order":3,"width":0,"height":0,"name":"Panel Amps (BMV)","label":"Panel Amps","format":"{{msg.payload | number:1}}","layout":"row-spread","x":310,"y":1240,"wires":[]},{"id":"f4b05d8f.13c5a","type":"function","z":"43fa7db7.c87434","name":"Panel A = W / V","func":"var panelAmps = msg.payload.PanelPower / msg.payload.PanelVoltage;\n{msg.payload = panelAmps}\nreturn msg;","outputs":1,"noerr":0,"x":290,"y":1180,"wires":[["9086c4b0.e9a658","d9d08b7b.f27038"]]},{"id":"d9d08b7b.f27038","type":"ui_text","z":"43fa7db7.c87434","group":"7595a526.d16c9c","order":3,"width":0,"height":0,"name":"Panel Amps (Web)","label":"Panel Amps","format":"{{msg.payload | number:1}}","layout":"row-spread","x":240,"y":1360,"wires":[]},{"id":"49c8d6ca.066498","type":"comment","z":"43fa7db7.c87434","name":"Data from BMV-712","info":"","x":150,"y":120,"wires":[]},{"id":"dacd5295.cfc12","type":"debug","z":"43fa7db7.c87434","d":true,"name":"BMV Data","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","x":340,"y":480,"wires":[]},{"id":"9a8f8c13.79f46","type":"debug","z":"43fa7db7.c87434","name":"MPPT Data","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","x":340,"y":680,"wires":[]},{"id":"f16ad309.7178c","type":"delay","z":"43fa7db7.c87434","name":"","pauseType":"rate","timeout":"5","timeoutUnits":"seconds","rate":"1","nbRateUnits":"5","rateUnits":"second","randomFirst":"1","randomLast":"5","randomUnits":"seconds","drop":false,"x":280,"y":400,"wires":[["11544f9c.8844","9c4ad4ff.7d1518","709276ea.6d2418","3aaf9cef.f6a0f4","25c4feea.0d0ad2","948b01ce.c3bc6","c3bb298b.e9ab58","92e69817.387bf8","93ff3a1e.c0abe8","61cfe391.8216bc"]]},{"id":"121fcb53.a578c5","type":"delay","z":"43fa7db7.c87434","name":"","pauseType":"queue","timeout":"5","timeoutUnits":"seconds","rate":"1","nbRateUnits":"5","rateUnits":"second","randomFirst":"1","randomLast":"5","randomUnits":"seconds","drop":true,"x":250,"y":900,"wires":[["ef00795c.8a1668","cec0dfaa.f18c4","88fcc365.65155","3eb1c80a.645c88","59b8eb7.829c214","e844d230.83367","21458ff0.40b79","fdf49e69.ed59d","37914dd7.08d312","f4b05d8f.13c5a","b97d829f.aabe9","90f939.e70746c8","54fd14a5.62926c","21bc5b5f.733e24","caf3b593.8ba728"]]},{"id":"90f939.e70746c8","type":"debug","z":"43fa7db7.c87434","d":true,"name":"","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","x":480,"y":840,"wires":[]},{"id":"54fd14a5.62926c","type":"ui_text","z":"43fa7db7.c87434","group":"b3c06fb5.d8985","order":4,"width":0,"height":0,"name":"MPPT O/P Amps to LB (BMV)","label":"Battery Amps","format":"{{msg.payload.BatteryAmps | number:1}}","layout":"row-spread","x":850,"y":1420,"wires":[]},{"id":"21bc5b5f.733e24","type":"ui_text","z":"43fa7db7.c87434","group":"7595a526.d16c9c","order":4,"width":0,"height":0,"name":"MPPT O/P Amps to LB (Web)","label":"Battery Amps","format":"{{msg.payload.BatteryAmps | number:1}}","layout":"row-spread","x":850,"y":1460,"wires":[]},{"id":"6ddd1b32.0d6dc4","type":"ui_chart","z":"43fa7db7.c87434","name":"MPPT State (Web)","group":"5ca09b48.127554","order":3,"width":0,"height":0,"label":"MPPT State","chartType":"line","legend":"false","xformat":"HH:mm:ss","interpolate":"linear","nodata":"","dot":false,"ymin":"0","ymax":"10","removeOlder":"6","removeOlderPoints":"","removeOlderUnit":"3600","cutout":0,"useOneColor":false,"colors":["#1f77b4","#aec7e8","#ff7f0e","#2ca02c","#98df8a","#d62728","#ff9896","#9467bd","#c5b0d5"],"useOldStyle":false,"outputs":1,"x":470,"y":1480,"wires":[[]]},{"id":"caf3b593.8ba728","type":"function","z":"43fa7db7.c87434","name":"MPPT State","func":"if(msg.payload.StateOfOperation === 0) //OFF\n    {msg.payload = 2.5} \nelse if(msg.payload.StateOfOperation === 1) //Fault\n    {msg.payload = 0}\nelse if(msg.payload.StateOfOperation === 3) //Bulk\n    {msg.payload = 10}\nelse if(msg.payload.StateOfOperation === 4) //Absorbtion\n    {msg.payload = 7.5}\nelse if(msg.payload.StateOfOperation === 5) //Float\n    {msg.payload = 5}\nelse\n    {}    //do nothing\nreturn msg;","outputs":1,"noerr":0,"x":270,"y":1480,"wires":[["6ddd1b32.0d6dc4","5bca7bb8.375e44"]]},{"id":"5bca7bb8.375e44","type":"debug","z":"43fa7db7.c87434","d":true,"name":"What","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","x":1430,"y":1820,"wires":[]},{"id":"fe8db9aa.436458","type":"file","z":"43fa7db7.c87434","d":true,"name":"","filename":"/home/pi/ChargerLogger4.txt","appendNewline":true,"createDir":true,"overwriteFile":"false","encoding":"none","x":1920,"y":460,"wires":[[]]},{"id":"2b8fc937.839586","type":"debug","z":"43fa7db7.c87434","name":"CSV Data","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","x":1800,"y":720,"wires":[]},{"id":"fad8f844.52e1c8","type":"join","z":"43fa7db7.c87434","name":"","mode":"custom","build":"object","property":"payload","propertyType":"msg","key":"topic","joiner":"\\n","joinerType":"str","accumulate":false,"timeout":"","count":"2","reduceRight":false,"reduceExp":"","reduceInit":"","reduceInitType":"","reduceFixup":"","x":1550,"y":720,"wires":[["e81c1d0.752aae"]]},{"id":"89bef54.30ea808","type":"change","z":"43fa7db7.c87434","name":"Topic 2 MPPT","rules":[{"t":"set","p":"topic","pt":"msg","to":"MPPT","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":1400,"y":760,"wires":[["fad8f844.52e1c8"]]},{"id":"1ede5097.59799f","type":"change","z":"43fa7db7.c87434","name":"Topic 2 BMV","rules":[{"t":"set","p":"topic","pt":"msg","to":"BMV","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":1390,"y":680,"wires":[["fad8f844.52e1c8"]]},{"id":"acc501b6.edaa1","type":"delay","z":"43fa7db7.c87434","name":"","pauseType":"rate","timeout":"5","timeoutUnits":"seconds","rate":"1","nbRateUnits":"20","rateUnits":"second","randomFirst":"1","randomLast":"5","randomUnits":"seconds","drop":false,"x":1180,"y":600,"wires":[[]]},{"id":"f101fdb8.005a6","type":"delay","z":"43fa7db7.c87434","name":"","pauseType":"rate","timeout":"5","timeoutUnits":"seconds","rate":"1","nbRateUnits":"20","rateUnits":"second","randomFirst":"1","randomLast":"5","randomUnits":"seconds","drop":false,"x":1240,"y":420,"wires":[[]]},{"id":"e81c1d0.752aae","type":"debug","z":"43fa7db7.c87434","name":"After delay","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","x":1610,"y":520,"wires":[]},{"id":"eed0a3af.084ee","type":"csv","z":"43fa7db7.c87434","name":"","sep":",","hdrin":"","hdrout":"","multi":"mult","ret":"\\n","temp":"","skip":"0","strings":true,"x":1710,"y":600,"wires":[["fe8db9aa.436458"]]},{"id":"3fcc4874.77b7f8","type":"ui_group","z":"","name":"Leisure Battery","tab":"74303f63.b1427","order":1,"disp":true,"width":"3","collapse":false},{"id":"5f89a742.c39168","type":"ui_group","z":"","name":"Starter Battery","tab":"74303f63.b1427","order":2,"disp":true,"width":"3","collapse":false},{"id":"ba16cc09.e9994","type":"ui_group","z":"","name":"Leisure Battery","tab":"3eec4b14.be2bf4","order":2,"disp":true,"width":6,"collapse":false},{"id":"5ca09b48.127554","type":"ui_group","z":"","name":"Starter Battery","tab":"3eec4b14.be2bf4","order":2,"disp":true,"width":"6","collapse":false},{"id":"7595a526.d16c9c","type":"ui_group","z":"","name":"MPPT Charger","tab":"3eec4b14.be2bf4","order":2,"disp":true,"width":"3","collapse":false},{"id":"b3c06fb5.d8985","type":"ui_group","z":"","name":"MPPT Charger","tab":"74303f63.b1427","order":5,"disp":true,"width":"3","collapse":false},{"id":"b90a2efa.d1c1e","type":"serial-port","z":"","serialport":"/dev/ttyUSB1","serialbaud":"19200","databits":"8","parity":"none","stopbits":"1","newline":"\\n","bin":"false","out":"char","addchar":false,"responsetimeout":"10000"},{"id":"ee60460c.fb5fc8","type":"serial-port","z":"","serialport":"/dev/ttyUSB0","serialbaud":"19200","databits":"8","parity":"none","stopbits":"1","newline":"\\n","bin":"false","out":"char","addchar":false,"responsetimeout":"10000"},{"id":"74303f63.b1427","type":"ui_tab","z":"","name":"BMV-700","icon":"MPPT","order":1,"disabled":false,"hidden":false},{"id":"3eec4b14.be2bf4","type":"ui_tab","z":"","name":"Web Portal","icon":"dashboard","order":2,"disabled":false,"hidden":false}]

[karlbo]

most likely, you would be better off storing in a sql, like mysql og sql lite
If you could make a sample of your data in array or string, it would be easier to render assistance.
this you can edit to fit/simulate your serial input

var myArray =[{ "where": "field1", "modtype": "software", "modby": "peter", "rigname": "drilling23"},
 { "where": "field2", "modtype": "software/hardware", "modby": "julius", "rigname": "drilling34"},
 { "where": "field3", "modtype": "hardware", "modby": "karl", "rigname": "drilling44"},
 { "where": "field4", "modtype": "none", "modby": "muhammed", "rigname": "drilling54"},
 { "where": "field5", "modtype": "welding", "modby": "alan", "rigname": "drilling64"}];
msg.payload = myArray;

return msg;

[karlbo]

also indicate from what point you are collecting the data you want to store

[xsilvergs]

karlbo

Thanks for reply. I have tried so many alternatives so here is my reasoning for the connections you highlight.

Connection 2 is the output from a delay timer. Without this delay the dashboard freezes.

Connection 1 is used as I thought it may be bad practice when it comes to saving data to fist pass it through a 5 second delay and then a 55 second delay (total 60 seconds) before saving.

What would you suggest as the best practice?

[xsilvergs]

I have tried sqlite but ran into the same difficulties and so stuck with csv as it was easier if not best practice.

But no matter to what I saved the data I couldn't get my head around getting the data in the same order to write to storage.

Thanks for replying

[karlbo]

show me what this gives in your debug window.
what you need first is to structure your data, then store it.

[xsilvergs]

[zenofmud]

@xsilvergs

There are 2 USB serial devices independently outputting data every second. The data needs to be saved to possibly a csv file every minute (there may be better alternatives to csv).

So you have a msg/second and you run it thru a 5 second delay, so after the 5 seconds you have one msg go thru but 4 are in the queue. after 10 seconds another msg is released and you have 8 in the queue. So after N seconds you have (N/5)*4 msgs in the queue.
After one minute (60 seconds) you have (60/5)*4 = 48
after 2 minutes 96 msgs in the queue
after 5 minutes 240 msgs in the queue
after 30 minutes 1440 msgs in the queue
each hour will add 2880 msgs to the queue

How will you handle the growing backlog and do you have enought memory to handle the growing queue. What will happen after 5 days?

[karlbo]

[{"id":"5c3bd3b0.f2134c","type":"file","z":"d7667819.1e1d18","name":"","filename":"/home/pi/test.txt","appendNewline":true,"createDir":true,"overwriteFile":"false","x":1132,"y":542,"wires":[[]]},{"id":"f2413aed.124728","type":"csv","z":"d7667819.1e1d18","name":"","sep":",","hdrin":"","hdrout":true,"multi":"one","ret":"\\n","temp":"BMV, MPPT","skip":"0","x":1076.5,"y":621,"wires":[["5c3bd3b0.f2134c"]]}]

works like a charm on my rpi

[xsilvergs]

You seem to be the man with all the answers, please tell me instead of making me look a fool.

[xsilvergs]

Thank you for this, I will try it and learn.

There are so many nodes and trying to understand what they all do and how to use them isn't easy.

I will try and report back.

Thanks again

[xsilvergs]

I have tried your example, quite easy really. My results look like this:
BMV,MPPT
"{""MainBatVoltage"":13.863,""AuxBatVoltage"":13.852,""BatteryCurrent"":0,""InstantaneousPower"":0,""ConsumedAmpHours"":0,""SOC"":100,""TimeToGo"":-1,""AlarmCon"":""OFF\r\n"",""AlarmReason"":""0\r\n""}","{""StateOfOperation"":5,""PanelVoltage"":20.18,""PanelPower"":28,""BatteryAmps"":2,""YieldToday"":0.08,""MaxPtoday"":55,""YieldYesterday"":0.08,""MaxPyesterday"":62,""ErrorCode"":0}"

I will now try to remove all un-required data so it looks good in csv.
Many thanks

[karlbo]

for better data integrity, try to learn how to use sql of some sort

[xsilvergs]

I did use mysql many years ago on a web server and felt quite proficient at it, using node-red to write the data will be interesting.

As pointed out in another reply perhaps I need to improve the way the data is delayed to stop it being buffered.

It's all smiles here at the moment.

[Colin]

Can I ask what you are going to do with the data later? What you want to use it for will affect how to store it.

[xsilvergs]

Hi.
Hopefully I'll be able review the data for up to 3 months. It will log the electrical system in our motorhome, AC in, solar, battery charging/discharging.

I would only need it in graphs and a few calculations. Reviewing would be done in something different to node-red I guess.

I've never used node-red so it's a bit of a learning curve, I will make mistakes but hopefully others will help.

Have you any experience/advice with storing date?

[Colin]

If you want to view historical data on graphs, zooming in and out and so on then I don't think there is anything to beat grafana. For the database I would use influxdb which is optimised for time series data and is very easy to write to in node-red via the influx nodes.
If you were to feed the string you showed earlier through a JSON node to convert it into a javacript object then you could feed directly to the influx node.
I haven't been following the thread in detail but I imagine you wouldn't need to use the join strategy that you have built, you can just feed the two (or more) sets of data to influx and it will sort it out.

[xsilvergs]

Colin,
Influxdb and Grafana installed with too much difficulty.

I'm having trouble getting Grafana to display a graph of my data.

Does this output from terminal look correct?

SELECT * FROM Victron LIMIT 10
name: Victron
time value

---

1581885655485045309 {"StateOfOperation":0,"PanelVoltage":0.01,"PanelPower":0,"BatteryAmps":0,"YieldToday":0,"MaxPtoday":0,"YieldYesterday":0.1,"MaxPyesterday":56,"ErrorCode":0}
1581885655497090964 {"StateOfOperation":0,"PanelVoltage":0.01,"PanelPower":0,"BatteryAmps":0,"YieldToday":0,"MaxPtoday":0,"YieldYesterday":0.1,"MaxPyesterday":56,"ErrorCode":0}

If the data above looks correct my problem may be in Grafana.

Thanks

[Colin]

No, I suspect you gave the node a string instead of an object so it has saved it as one string value instead of all the separate fields. Use a debug node to show what you are sending it. I did suggest that you needed to feed the string you posted (assuming it really was a string) through a JSON node. If it was already and object then feeding it through the JSON node will change it to a string which is not what you want.

[xsilvergs]

Colin
I have now changed the json node,see image.

From Terminal:

SELECT * FROM Victron LIMIT 10
name: Victron
time AlarmCon AlarmReason AuxBatVoltage BatteryCurrent ConsumedAmpHours InstantaneousPower MainBatVoltage SOC TimeToGo

---

1581889199983476325 OFF
0
13.5 0 0 0 13.5 100 -1
1581889199991739411 OFF
0
13.5 0 0 0 13.5 100 -1

Is that looking better?