I'm using a data logger to monitor current which creates very large log files. I can't find a tool to display the results in graphical form like you would see from an o-scope. I am trying to use node-red and the chart node to do this. I can only get the chart to work by sending each measurement in its own message. This process is incredibly slow. My question is can I send all the data at once as an array and will it render faster?
Could you share your flow and/or the info you would like to have extracted (debug)
I seem to have this very weird superpower. No matter how long I struggle with something the minute I ask for help I find the answer myself. In this case it's to use the UI template instead of the chart node. I'm still coding but it looks very promising.
Thank you for the quick response.
Just and FYI.
The data is floating point. I need fine detail, so I take a measurement every 2uS. The logger stores the data in multiple files, each resulting in an array of 300,000 measurements. My goal is to display an entire file at a time then I can scroll through the files to view/search the results for issues.
It looks like I may not be able to show the whole file at once so I may break it into a few charts that I stack. What would really be awesome is if I could scroll and zoom on all the data from all the files.
I'm open to any suggestions but I only have a small window of time to work on this.
This page (linked from the chart help text) shows how to do that.
However, the idea of putting 300,000 points on a chart, which is presumably just a few hundred, or maybe 1000 pixels wide, is ludicrous. You will end up with hundreds of dots for each pixel in the x axis. Also those 300,000 points would need to be sent to the browser, which will never be quick and will quite possibly overload it. You will need to pre-process the data before sending to the chart.
Or possibly feed the data into Influxdb and then use Grafana for the display, which will give you flexibility to zoom and scroll through the data for charting.
1 Like
I have viewed this page many times and don't see my mistake. My flow is below.
I put two small data files in a zip for you. Just change the extension to .zip. Put them in a dir which has a simple path and put the path in the dashboard. The button causes the dir to be read so my flow knows what files are available. The slider picks the file to chart. I'm only trying to plot the 3rd row of data from the file
data_1.log (3.4 KB)
[{"id":"62a0e5a5.fc59bc","type":"tab","label":"Flow 1","disabled":false,"info":""},{"id":"7b52c915.5f78b","type":"fs-ops-dir","z":"62a0e5a5.fc59bc","name":"","path":"path","pathType":"flow","filter":"data*","filterType":"str","dir":"files","dirType":"msg","x":560,"y":520,"wires":[["ce99c68c.614d68"]]},{"id":"cc5f13da.98d0e","type":"inject","z":"62a0e5a5.fc59bc","name":"","props":[{"p":"path","v":"C:\\Users\\Dave\\Dropbox\\My PC (DAVID-SPLITZ)\\Desktop\\RSI Fireless\\Base Design\\Current Logging\\test2","vt":"str"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","x":170,"y":520,"wires":[["eb96378d.35f4c"]]},{"id":"ce99c68c.614d68","type":"function","z":"62a0e5a5.fc59bc","name":"","func":"\nmin =0\nmax = (msg.files.length)\ndata = 0\nmsg.ui_control={\"max\":max}\n//msg = {payload:data,ui_control:{min:min,max:max},topic:\"Log\"};\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":780,"y":520,"wires":[["8d938864.0d36a"]]},{"id":"eb96378d.35f4c","type":"ui_button","z":"62a0e5a5.fc59bc","name":"","group":"fd0d5a26.3b961","order":1,"width":0,"height":0,"passthru":true,"label":"button","tooltip":"","color":"","bgcolor":"","icon":"","payload":"","payloadType":"str","topic":"topic","topicType":"msg","x":350,"y":520,"wires":[["7b52c915.5f78b","32a15b7f.41edf4"]]},{"id":"e693caf.189fcb8","type":"ui_text","z":"62a0e5a5.fc59bc","group":"fd0d5a26.3b961","order":4,"width":0,"height":0,"name":"","label":"Location","format":"{{msg.payload}}","layout":"col-center","x":1270,"y":520,"wires":[]},{"id":"5fe654c.aae212c","type":"file in","z":"62a0e5a5.fc59bc","name":"","filename":"","format":"utf8","chunk":false,"sendError":false,"encoding":"ascii","x":330,"y":680,"wires":[["faa19d2b.24a328"]]},{"id":"8c34ce17.6f916","type":"function","z":"62a0e5a5.fc59bc","name":"Pick File","func":"//\"C:/Users/Dave/CurrentLogs/data_0.dat\"\nmsg.filename = flow.get(\"path\")\nmsg.filename += \"/data_\"\nmsg.filename += msg.payload.toString(10)\nmsg.filename += \".dat\"\n\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":160,"y":680,"wires":[["5fe654c.aae212c"]]},{"id":"77fbc4c1.f63e04","type":"ui_text_input","z":"62a0e5a5.fc59bc","name":"","label":"Log Path (no spaces allowed)","tooltip":"","group":"fd0d5a26.3b961","order":2,"width":0,"height":0,"passthru":false,"mode":"text","delay":300,"topic":"topic","topicType":"msg","x":260,"y":300,"wires":[["4d92b29d.48e0ac"]]},{"id":"4d92b29d.48e0ac","type":"function","z":"62a0e5a5.fc59bc","name":"Store path","func":"path = msg.payload\nflow.set(\"path\",path)\nmsg.payload = path\nreturn msg","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":550,"y":300,"wires":[[]]},{"id":"8d938864.0d36a","type":"ui_slider","z":"62a0e5a5.fc59bc","name":"","label":"slider","tooltip":"","group":"fd0d5a26.3b961","order":3,"width":0,"height":0,"passthru":false,"outs":"end","topic":"topic","topicType":"msg","min":0,"max":10,"step":1,"x":1070,"y":520,"wires":[["e693caf.189fcb8","8c34ce17.6f916"]]},{"id":"faa19d2b.24a328","type":"string","z":"62a0e5a5.fc59bc","name":"","methods":[{"name":"delLeftMost","params":[{"type":"str","value":"y0"}]},{"name":"replaceAll","params":[{"type":"str","value":"\\r\\n"},{"type":"str","value":"\t"}]},{"name":"parseCSV","params":[{"type":"str","value":"\t"},{"type":"str","value":""},{"type":"str","value":""},{"type":"str","value":"\\r\\n"}]}],"prop":"payload","propout":"payload","object":"msg","objectout":"msg","x":470,"y":680,"wires":[["6228d367.94d5dc"]]},{"id":"32a15b7f.41edf4","type":"function","z":"62a0e5a5.fc59bc","name":"","func":"msg.payload=[]\nflow.set('array',[])\nflow.set('array2',[])\nflow.set('count',0)\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":520,"y":580,"wires":[["91cdf91d.baff38"]]},{"id":"91cdf91d.baff38","type":"ui_chart","z":"62a0e5a5.fc59bc","name":"","group":"fd0d5a26.3b961","order":5,"width":0,"height":0,"label":"chart","chartType":"line","legend":"false","xformat":"ms","interpolate":"linear","nodata":"","dot":false,"ymin":"","ymax":"","removeOlder":1,"removeOlderPoints":"","removeOlderUnit":"3600","cutout":0,"useOneColor":false,"useUTC":false,"colors":["#1f77b4","#aec7e8","#ff7f0e","#2ca02c","#98df8a","#d62728","#ff9896","#9467bd","#c5b0d5"],"outputs":1,"useDifferentColor":false,"x":990,"y":680,"wires":[[]]},{"id":"6228d367.94d5dc","type":"function","z":"62a0e5a5.fc59bc","name":"","func":"datain = msg.payload[0]\ntemp=msg.payload\n\n\nsize = datain.length/3\nvar array =[]\nvar array2 =[]\nvar i=0\n\n\nfor (x=3;x < size;x+=3)//only plot 3rd row of data\n{\n array[i]=Number(datain[x])\n array2[i++] = i\n}\nnode.warn(array)\nnode.warn(array2)\n\nmsg.payload=[{\"series\":[\"\"],\"data\":[array],\"labels\":array2}]\nnode.warn(msg.payload)\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":640,"y":680,"wires":[["8668de65.acb9b","91cdf91d.baff38"]]},{"id":"8668de65.acb9b","type":"debug","z":"62a0e5a5.fc59bc","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":820,"y":740,"wires":[]},{"id":"fd0d5a26.3b961","type":"ui_group","name":"Info","tab":"9d002959.43f038","order":2,"disp":true,"width":30,"collapse":false},{"id":"9d002959.43f038","type":"ui_tab","name":"Home","icon":"dashboard","disabled":false,"hidden":false}]```Show us what you see in a debug node going to the chart.
[Edit} I mean the debug node should be connected to show what is going into the chart.
Example of sending your log file to chart
[{"id":"59058a6b.dd780c","type":"inject","z":"bf9e1e33.030598","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":140,"y":1140,"wires":[["45fb6f7.46b211"]]},{"id":"45fb6f7.46b211","type":"template","z":"bf9e1e33.030598","name":"","field":"payload","fieldType":"msg","format":"handlebars","syntax":"mustache","template":"2/17/2022\n4:14:49 PM\n\n\n\nTime\tv0\ty0\n1.000000\t0.024761\t0.000248\n1.000010\t0.023183\t0.000232\n1.000020\t0.022867\t0.000229\n1.000030\t0.023814\t0.000238\n1.000040\t0.022236\t0.000222\n1.000050\t0.023814\t0.000238\n1.000060\t0.021920\t0.000219\n1.000070\t0.024130\t0.000241\n1.000080\t0.024446\t0.000244\n1.000090\t0.021605\t0.000216\n1.000100\t0.021920\t0.000219\n1.000110\t0.020974\t0.000210\n1.000120\t0.018764\t0.000188\n1.000130\t0.019395\t0.000194\n1.000140\t0.023499\t0.000235\n1.000150\t0.023183\t0.000232\n1.000160\t0.023499\t0.000235\n1.000170\t0.023499\t0.000235\n1.000180\t0.024761\t0.000248\n1.000190\t0.024761\t0.000248\n1.000200\t0.021605\t0.000216\n1.000210\t0.020342\t0.000203\n1.000220\t0.022552\t0.000226\n1.000230\t0.019711\t0.000197\n1.000240\t0.023814\t0.000238\n1.000250\t0.021920\t0.000219\n1.000260\t0.021605\t0.000216\n1.000270\t0.019080\t0.000191\n1.000280\t0.021920\t0.000219\n1.000290\t0.022867\t0.000229\n1.000300\t0.022552\t0.000226\n1.000310\t0.022552\t0.000226\n1.000320\t0.022552\t0.000226\n1.000330\t0.024130\t0.000241\n1.000340\t0.022552\t0.000226\n1.000350\t0.023814\t0.000238\n1.000360\t0.021920\t0.000219\n1.000370\t0.022867\t0.000229\n1.000380\t0.021920\t0.000219\n1.000390\t0.022867\t0.000229\n1.000400\t0.023499\t0.000235\n1.000410\t0.020658\t0.000207\n1.000420\t0.024446\t0.000244\n1.000430\t0.022236\t0.000222\n","output":"str","x":320,"y":1140,"wires":[["b498ed7d.7acf8"]]},{"id":"b498ed7d.7acf8","type":"csv","z":"bf9e1e33.030598","name":"","sep":"\\t","hdrin":true,"hdrout":"none","multi":"mult","ret":"\\n","temp":"","skip":"5","strings":true,"include_empty_strings":"","include_null_values":"","x":450,"y":1100,"wires":[["6228d367.94d5dc"]]},{"id":"6228d367.94d5dc","type":"function","z":"bf9e1e33.030598","name":"","func":"var array1 = [];\nlet array2 = [];\n\nfor (const element of msg.payload){\n array1.push({x:element.Time*100000,y:element.v0})\n array2.push({x:element.Time*100000,y:element.y0})\n}\n\nmsg.payload=[{\"series\":[\"v0\",\"y0\"],\"data\":[array1,array2],\"labels\":[\"v0\",\"y0\"]}]\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","x":630,"y":1080,"wires":[["8668de65.acb9b","91cdf91d.baff38"]]},{"id":"8668de65.acb9b","type":"debug","z":"bf9e1e33.030598","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":730,"y":1180,"wires":[]},{"id":"91cdf91d.baff38","type":"ui_chart","z":"bf9e1e33.030598","name":"","group":"2d4fe667.28f8ba","order":5,"width":0,"height":0,"label":"chart","chartType":"line","legend":"false","xformat":"ms","interpolate":"linear","nodata":"","dot":false,"ymin":"","ymax":"","removeOlder":1,"removeOlderPoints":"","removeOlderUnit":"3600","cutout":0,"useOneColor":false,"useUTC":false,"colors":["#1f77b4","#aec7e8","#ff7f0e","#2ca02c","#98df8a","#d62728","#ff9896","#9467bd","#c5b0d5"],"outputs":1,"useDifferentColor":false,"className":"","x":900,"y":1120,"wires":[[]]},{"id":"2d4fe667.28f8ba","type":"ui_group","name":"demo","tab":"1caa8458.b17814","order":1,"disp":true,"width":"12","collapse":false},{"id":"1caa8458.b17814","type":"ui_tab","name":"Demo","icon":"dashboard","order":1,"disabled":false,"hidden":false}]
i simulated your file read node with a template
I edited the data files to shorten them for upload and it worked. I then captured a new set of logs and again it fails. Here is the debug from the failure.
I found I can configure the tool to give me smaller files and it seems to be working now. I think I can make it play like a movie now.
Thanks for your help. Have a great weekend.
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.
