Node-red-contrib-influxdb overloads CPU with queries

[MartinJensen37]

Hello.

I have an issue regarding CPU usage while using the node-red-contrib-influxdb.

I have a lot of data from different sensors getting sent to an InfluxDB database. I then have to use multiple computers to send queries/requests for this data. However the Raspberry Pi 4B I'm running this on will start using more and more CPU power until the influxdb process is killed.

It seems like the node-red-contrib-influxdb doesn't close the connection to the RPI4 every time a query is sent. Is there a way to close this connection once the data has been sent?

I'm using influxdb1.8.

Thanks in advance!

[Colin]

@MartinJensen37

It seems like the node-red-contrib-influxdb doesn't close the connection to the RPI4 every time a query is sent. Is there a way to close this connection once the data has been sent?

What makes you think that is the problem?

Have you looked in the influx logs to see what it is doing?

[MartinJensen37]

Hello Colin.

The issue is that I have around 2000 TCP connections open at the same time currently this takes up 80-90% of my raspberry computing power.

It runs fine in the start, but the longer it runs the more connections are established until InfluxDB shuts down. I don't know whether my flow is bad(see below) or there is an oversight in the node-red-contrib-influxdb so that it doesn't shut down the connections it creates.

I'm sending quite a bit of data and requesting to read quite a bit of data, so maybe it simply can't close the TCP connections as fast as it is creating them.

[{"id":"8463a30a.98469","type":"tab","label":"Flow 2","disabled":false,"info":""},{"id":"38e32758.e955b8","type":"influxdb in","z":"8463a30a.98469","influxdb":"5ecb4dc3.6713a4","name":"Influxdb","query":"SELECT * FROM \"ESP03/light\" WHERE time > now()-10m\n","rawOutput":false,"precision":"ms","retentionPolicy":"","org":"my-org","x":380,"y":280,"wires":[["6e5ed824.af83b8"]]},{"id":"ce7ab224.553f2","type":"inject","z":"8463a30a.98469","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"0.5","crontab":"","once":true,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":110,"y":420,"wires":[["38e32758.e955b8","69676660.b82608","bf6b3cb5.5806f","5585e9f4.251548","c85ffadc.91a3c8","54734824.a36aa8","3c51c6a3.e6f1fa","a25f8c36.ddcd3"]]},{"id":"f7ee3802.f7bdc8","type":"debug","z":"8463a30a.98469","name":"","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":830,"y":320,"wires":[]},{"id":"8bbe2f64.c9b53","type":"ui_chart","z":"8463a30a.98469","name":"","group":"4f0696b7.3e3b78","order":2,"width":"20","height":"6","label":"Light Intensity over time","chartType":"line","legend":"false","xformat":"HH:mm:ss","interpolate":"linear","nodata":"","dot":false,"ymin":"","ymax":"","removeOlder":"2","removeOlderPoints":"","removeOlderUnit":"3600","cutout":0,"useOneColor":false,"useUTC":false,"colors":["#e6ff29","#aec7e8","#ff7f0e","#2ca02c","#98df8a","#d62728","#ff9896","#9467bd","#c5b0d5"],"outputs":1,"useDifferentColor":false,"x":850,"y":280,"wires":[[]]},{"id":"6e5ed824.af83b8","type":"function","z":"8463a30a.98469","name":"compute series data","func":"var series = [\"light\"];\nvar data = [];\nvar labels = [\"brightness\"];\n\nfor(var i = 0, len = msg.payload.length; i < len; i++){\n  data.push({x: msg.payload[i].time, y: msg.payload[i].light});\n}\n\nmsg.payload = [{\"series\": series, \"data\": [data], \"labels\": labels}];\n\nreturn msg","outputs":1,"noerr":0,"initialize":"","finalize":"","x":580,"y":280,"wires":[["8bbe2f64.c9b53","f7ee3802.f7bdc8"]]},{"id":"bf6b3cb5.5806f","type":"influxdb in","z":"8463a30a.98469","influxdb":"5ecb4dc3.6713a4","name":"Influxdb","query":"SELECT * FROM \"ESP04/temperature\" WHERE time > now()-1h\n","rawOutput":false,"precision":"ms","retentionPolicy":"","org":"my-org","x":380,"y":420,"wires":[["95c79a1c.5561a8"]]},{"id":"a1cd2ae2.4f8058","type":"debug","z":"8463a30a.98469","name":"","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":810,"y":480,"wires":[]},{"id":"39ff684f.78e358","type":"ui_chart","z":"8463a30a.98469","name":"","group":"1490c23e.22a8ce","order":2,"width":"20","height":"6","label":"Temperature over time","chartType":"line","legend":"true","xformat":"HH:mm:ss","interpolate":"linear","nodata":"","dot":false,"ymin":"10","ymax":"40","removeOlder":"2","removeOlderPoints":"","removeOlderUnit":"3600","cutout":0,"useOneColor":false,"useUTC":false,"colors":["#b31e1e","#aec7e8","#ff7f0e","#2ca02c","#98df8a","#d62728","#ff9896","#9467bd","#c5b0d5"],"outputs":1,"useDifferentColor":false,"x":840,"y":420,"wires":[[]]},{"id":"95c79a1c.5561a8","type":"function","z":"8463a30a.98469","name":"compute series data","func":"var series = [\"temperature\"];\nvar data = [];\nvar labels = [\"Temp in Celsius\"];\n\nfor(var i = 0, len = msg.payload.length; i < len; i++){\n  data.push({x: msg.payload[i].time, y: msg.payload[i].temperature});\n}\n\nmsg.payload = [{\"series\": series, \"data\": [data], \"labels\": labels}];\n\nreturn msg","outputs":1,"noerr":0,"initialize":"","finalize":"","x":580,"y":420,"wires":[["39ff684f.78e358","a1cd2ae2.4f8058"]]},{"id":"5585e9f4.251548","type":"influxdb in","z":"8463a30a.98469","influxdb":"5ecb4dc3.6713a4","name":"Influxdb","query":"SELECT * FROM \"ESP04/humidity\" WHERE time > now()-1h\n","rawOutput":false,"precision":"ms","retentionPolicy":"","org":"my-org","x":380,"y":540,"wires":[["c761588b.1da828"]]},{"id":"1c0549be.bea276","type":"debug","z":"8463a30a.98469","name":"","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":810,"y":600,"wires":[]},{"id":"9e16d4de.8926d8","type":"ui_chart","z":"8463a30a.98469","name":"","group":"dfa426fa.425fe8","order":2,"width":"20","height":"6","label":"Humidity over time","chartType":"line","legend":"true","xformat":"HH:mm:ss","interpolate":"linear","nodata":"","dot":false,"ymin":"","ymax":"","removeOlder":"2","removeOlderPoints":"10000","removeOlderUnit":"3600","cutout":0,"useOneColor":false,"useUTC":false,"colors":["#1f77b4","#aec7e8","#ff7f0e","#2ca02c","#98df8a","#d62728","#ff9896","#9467bd","#c5b0d5"],"outputs":1,"useDifferentColor":false,"x":830,"y":540,"wires":[[]]},{"id":"c761588b.1da828","type":"function","z":"8463a30a.98469","name":"compute series data","func":"var series = [\"humidity\"];\nvar data = [];\nvar labels = [\"STPLC_01\"];\n\nfor(var i = 0, len = msg.payload.length; i < len; i++){\n  data.push({x: msg.payload[i].time, y: msg.payload[i].humidity});\n}\n\nmsg.payload = [{\"series\": series, \"data\": [data], \"labels\": labels}];\n\nreturn msg","outputs":1,"noerr":0,"initialize":"","finalize":"","x":580,"y":540,"wires":[["1c0549be.bea276","9e16d4de.8926d8"]]},{"id":"69676660.b82608","type":"influxdb in","z":"8463a30a.98469","influxdb":"5ecb4dc3.6713a4","name":"Influxdb","query":"SELECT * FROM \"ESP01/light\" WHERE time > now()-10s\n","rawOutput":false,"precision":"ms","retentionPolicy":"","org":"my-org","x":380,"y":200,"wires":[["de5ad39e.4d33"]]},{"id":"78287bac.e3fc34","type":"ui_gauge","z":"8463a30a.98469","name":"light_intensity","group":"2f2c6a68.1b0326","order":1,"width":0,"height":0,"gtype":"gage","title":"Light Intensity","label":"","format":"{{value}}","min":0,"max":"255","colors":["#00b500","#e6e600","#ca3838"],"seg1":"100","seg2":"200","x":740,"y":200,"wires":[]},{"id":"de5ad39e.4d33","type":"function","z":"8463a30a.98469","name":"","func":"msg.payload = [msg.payload[0].light];\n\nreturn msg","outputs":1,"noerr":0,"initialize":"","finalize":"","x":540,"y":200,"wires":[["78287bac.e3fc34"]]},{"id":"c85ffadc.91a3c8","type":"influxdb in","z":"8463a30a.98469","influxdb":"5ecb4dc3.6713a4","name":"Influxdb","query":"SELECT * FROM \"ESP07/detection\" WHERE time > now()-10m\n","rawOutput":false,"precision":"ms","retentionPolicy":"","org":"my-org","x":380,"y":680,"wires":[["de6fdcb7.9268f"]]},{"id":"59da06a0.646fc8","type":"ui_led","z":"8463a30a.98469","order":0,"group":"5d4d0fbb.ae46a","width":0,"height":0,"label":"Object moved on","labelPlacement":"left","labelAlignment":"left","colorForValue":[{"color":"#008000","value":"0","valueType":"str"},{"color":"#ff0000","value":"1","valueType":"str"},{"color":"#ff0000","value":"2","valueType":"str"},{"color":"#ff0000","value":"3","valueType":"str"}],"allowColorForValueInMessage":false,"shape":"circle","showGlow":true,"name":"","x":970,"y":660,"wires":[]},{"id":"de6fdcb7.9268f","type":"function","z":"8463a30a.98469","name":"","func":"msg.payload = msg.payload[msg.payload.length-1].detection;\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","x":540,"y":680,"wires":[["59da06a0.646fc8","302b7465.628f1c","769a3f32.47275","7b28567b.b75478"]]},{"id":"302b7465.628f1c","type":"ui_led","z":"8463a30a.98469","order":0,"group":"5d4d0fbb.ae46a","width":0,"height":0,"label":"Object detected","labelPlacement":"left","labelAlignment":"left","colorForValue":[{"color":"#ff0000","value":"0","valueType":"str"},{"color":"#008000","value":"1","valueType":"str"},{"color":"#ff0000","value":"2","valueType":"str"},{"color":"#ff0000","value":"3","valueType":"str"}],"allowColorForValueInMessage":false,"shape":"circle","showGlow":true,"name":"","x":970,"y":700,"wires":[]},{"id":"769a3f32.47275","type":"ui_led","z":"8463a30a.98469","order":0,"group":"5d4d0fbb.ae46a","width":0,"height":0,"label":"Object still here","labelPlacement":"left","labelAlignment":"left","colorForValue":[{"color":"#ff0000","value":"0","valueType":"str"},{"color":"#ff0000","value":"1","valueType":"str"},{"color":"#008000","value":"2","valueType":"str"},{"color":"#ff0000","value":"3","valueType":"str"}],"allowColorForValueInMessage":false,"shape":"circle","showGlow":true,"name":"","x":970,"y":740,"wires":[]},{"id":"7b28567b.b75478","type":"ui_led","z":"8463a30a.98469","order":0,"group":"5d4d0fbb.ae46a","width":0,"height":0,"label":"No object","labelPlacement":"left","labelAlignment":"left","colorForValue":[{"color":"#ff0000","value":"2","valueType":"str"},{"color":"#008000","value":"3","valueType":"str"},{"color":"#ff0000","value":"1","valueType":"str"},{"color":"#ff0000","value":"0","valueType":"str"}],"allowColorForValueInMessage":false,"shape":"circle","showGlow":true,"name":"","x":970,"y":780,"wires":[]},{"id":"516f18ba.697f48","type":"ui_text","z":"8463a30a.98469","group":"c77859de.54fe38","order":0,"width":0,"height":0,"name":"","label":"X-axis","format":"{{msg.payload['x']}}","layout":"row-left","x":970,"y":820,"wires":[]},{"id":"5b55cce2.c9d854","type":"ui_text","z":"8463a30a.98469","group":"c77859de.54fe38","order":0,"width":0,"height":0,"name":"","label":"Y-axis","format":"{{msg.payload['y']}}","layout":"row-left","x":970,"y":860,"wires":[]},{"id":"79835ce7.f99804","type":"ui_text","z":"8463a30a.98469","group":"c77859de.54fe38","order":0,"width":0,"height":0,"name":"","label":"Z-axis","format":"{{msg.payload['z']}}","layout":"row-left","x":970,"y":900,"wires":[]},{"id":"b28deb83.01b7c8","type":"ui_chart","z":"8463a30a.98469","name":"","group":"ff3b6aa4.06fe98","order":0,"width":0,"height":0,"label":"IMU","chartType":"line","legend":"false","xformat":"HH:mm:ss","interpolate":"linear","nodata":"","dot":false,"ymin":"","ymax":"","removeOlder":1,"removeOlderPoints":"","removeOlderUnit":"3600","cutout":0,"useOneColor":false,"useUTC":false,"colors":["#1f77b4","#aec7e8","#ff7f0e","#2ca02c","#98df8a","#d62728","#ff9896","#9467bd","#c5b0d5"],"outputs":1,"useDifferentColor":false,"x":970,"y":940,"wires":[[]]},{"id":"54734824.a36aa8","type":"influxdb in","z":"8463a30a.98469","influxdb":"5ecb4dc3.6713a4","name":"Influxdb","query":"SELECT * FROM \"ESP01/acceleration\" WHERE time > now()-3s\n","rawOutput":false,"precision":"ms","retentionPolicy":"","org":"my-org","x":380,"y":800,"wires":[["91cb3e9c.b5bc8"]]},{"id":"91cb3e9c.b5bc8","type":"function","z":"8463a30a.98469","name":"","func":"msg.payload = msg.payload[msg.payload.length-1]\n\nreturn msg","outputs":1,"noerr":0,"initialize":"","finalize":"","x":540,"y":800,"wires":[["516f18ba.697f48","5b55cce2.c9d854","79835ce7.f99804"]]},{"id":"3c51c6a3.e6f1fa","type":"influxdb in","z":"8463a30a.98469","influxdb":"5ecb4dc3.6713a4","name":"Influxdb","query":"SELECT * FROM \"ESP01/acceleration\" WHERE time > now()-1h\n","rawOutput":false,"precision":"ms","retentionPolicy":"","org":"my-org","x":380,"y":860,"wires":[["d651b5df.dcc068"]]},{"id":"d651b5df.dcc068","type":"function","z":"8463a30a.98469","name":"","func":"var series = [\"IMU\"];\nvar msg1 = [];\nvar msg2 = [];\nvar msg3 = [];\nvar labels = [\"X-axis\", \"Y-axis\", \"Z-axis\"];\n\nfor(var i = 0, len = msg.payload.length; i < len; i++){\n  msg1.push({x: msg.payload[i].time, y: msg.payload[i].x});\n  msg2.push({x: msg.payload[i].time, y: msg.payload[i].y});\n  msg3.push({x: msg.payload[i].time, y: msg.payload[i].z});\n}\n\nmsg.payload = [{\"series\": series, \"data\": [msg1, msg2, msg3], \"labels\": labels}];\n\nreturn msg","outputs":1,"noerr":0,"initialize":"","finalize":"","x":540,"y":860,"wires":[["b28deb83.01b7c8"]]},{"id":"a25f8c36.ddcd3","type":"influxdb in","z":"8463a30a.98469","influxdb":"5ecb4dc3.6713a4","name":"Influxdb","query":"SELECT * FROM \"ESP08/sound\" WHERE time > now()-1h\n","rawOutput":false,"precision":"ms","retentionPolicy":"","org":"my-org","x":380,"y":360,"wires":[["53d2415.5dc4ec"]]},{"id":"53d2415.5dc4ec","type":"function","z":"8463a30a.98469","name":"compute series data","func":"var series = [\"light\"];\nvar data = [];\nvar labels = [\"brightness\"];\n\nfor(var i = 0, len = msg.payload.length; i < len; i++){\n  data.push({x: msg.payload[i].time, y: msg.payload[i].sound});\n}\n\nmsg.payload = [{\"series\": series, \"data\": [data], \"labels\": labels}];\n\nreturn msg","outputs":1,"noerr":0,"initialize":"","finalize":"","x":580,"y":360,"wires":[["1618b786.2961e8"]]},{"id":"1618b786.2961e8","type":"ui_chart","z":"8463a30a.98469","name":"","group":"a7f6f0cf.271e6","order":2,"width":"21","height":"7","label":"Sound Intensity over time","chartType":"line","legend":"false","xformat":"HH:mm:ss","interpolate":"linear","nodata":"","dot":false,"ymin":"","ymax":"","removeOlder":"2","removeOlderPoints":"","removeOlderUnit":"3600","cutout":0,"useOneColor":false,"useUTC":false,"colors":["#e6ff29","#aec7e8","#ff7f0e","#2ca02c","#98df8a","#d62728","#ff9896","#9467bd","#c5b0d5"],"outputs":1,"useDifferentColor":false,"x":850,"y":360,"wires":[[]]},{"id":"5ecb4dc3.6713a4","type":"influxdb","hostname":"172.20.66.5","port":"8086","protocol":"http","database":"FestoSensors","name":"smartProdLab","usetls":false,"tls":"d50d0c9f.31e858","influxdbVersion":"1.x","url":"http://localhost:8086","rejectUnauthorized":true},{"id":"4f0696b7.3e3b78","type":"ui_group","name":"Light","tab":"4ff4ca1.8bd2f34","order":6,"disp":true,"width":"20","collapse":false},{"id":"1490c23e.22a8ce","type":"ui_group","name":"Temperature","tab":"4ff4ca1.8bd2f34","order":5,"disp":true,"width":"20","collapse":false},{"id":"dfa426fa.425fe8","type":"ui_group","name":"Humidity","tab":"4ff4ca1.8bd2f34","order":7,"disp":true,"width":"20","collapse":false},{"id":"2f2c6a68.1b0326","type":"ui_group","name":"Light(Live)","tab":"4ff4ca1.8bd2f34","order":1,"disp":true,"width":"6","collapse":false},{"id":"5d4d0fbb.ae46a","type":"ui_group","name":"Linetracker","tab":"4ff4ca1.8bd2f34","order":3,"disp":true,"width":"6","collapse":false},{"id":"c77859de.54fe38","type":"ui_group","name":"IMU(live)","tab":"4ff4ca1.8bd2f34","order":2,"disp":true,"width":"6","collapse":false},{"id":"ff3b6aa4.06fe98","type":"ui_group","name":"IMU","tab":"4ff4ca1.8bd2f34","order":9,"disp":true,"width":"20","collapse":false},{"id":"a7f6f0cf.271e6","type":"ui_group","name":"festoSound","tab":"4ff4ca1.8bd2f34","order":10,"disp":true,"width":"21","collapse":false},{"id":"d50d0c9f.31e858","type":"tls-config","name":"","cert":"","key":"","ca":"","certname":"","keyname":"","caname":"","servername":"","verifyservercert":false},{"id":"4ff4ca1.8bd2f34","type":"ui_tab","name":"Festo Dashboard","icon":"FestoSensors","disabled":false,"hidden":false}]

[Colin]

Which version of influx are you using? Are you using the latest version of the influx node?

Please stop node red then start it in a terminal and post the full startup log here.

[Edit] Also check using top whether anything is hogging the processor (influx or node-red maybe) and using iotop whether anything is held up trying to access the disc or card.

Are you using a USB Disc for the influx data?

Further edit Where is the browser running? I see you are completely rebuilding the charts every 0.5 seconds. How many data points are there along the time axis on those. How often are the values written to the db? I am wondering whether the pi is just completely bogged down and cannot do all the work you are asking it to do. It keeps asking influx for more data, but there is not sufficient processor power to handle it. Alternatively (or also) if you are using an SD card for the database perhaps the card cannot keep up with the amount of data you are transferring. So again you keep starting new influx operations before it has completed the previous ones so more and more influx requests keep piling up.

You can prove that by slowing the whole system down (initially maybe just change the inject node to every 5 seconds instead of 0.5, which will at least slow the reads down).