How to diagnose crashing possibly caused by memory leak

General
1

Hi everyone, I have a remote node red installation that is being used to schedule events using the cronplus node and sends and receives commands over OSC.

I installed and run the Pi according to the instructions on the node red site and the service was installed using 'sudo systemctl enable nodered.service' The Pi is set to reboot at midnight each day.

The Pi will run some days no problem and will crash on other days and I have no idea why. When I have tried to use terminal to login either there is no response or the terminal is really, really slow, 5-10mins from accepting connection to getting to a command line and then taking 3-4mins to act on each command.
I have run node-red-log and cannot see any obvious errors.
When running 'top' the memory usage of node creeps up to around 88.4% where it then stays.
I was wondering if you could help with any pointers to fix the problem.
Thanks for any help.
Steve

2

Can you show us your flaws?

Is there a chance you have a loop causing the system to runaway?

What is OSC?

3

Hi Steve the flow is below.
OSC is Open Sound Control, it is a protocol for sending messages across UDP.
Thanks for any help.
Steve

[{"id":"585b49b.1d26cb8","type":"tab","label":"Schedule","disabled":false,"info":""},{"id":"b05d2abc.c27498","type":"udp out","z":"585b49b.1d26cb8","name":"","addr":"192.168.100.90","iface":"","port":"12321","ipv":"udp4","outport":"","base64":false,"multicast":"false","x":2040,"y":220,"wires":[]},{"id":"d12470a.45b239","type":"inject","z":"585b49b.1d26cb8","name":"","props":[{"p":"payload"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"Schedule 1","payloadType":"str","x":2380,"y":1140,"wires":[["f52a058e.d3f818"]]},{"id":"949c3273.77805","type":"change","z":"585b49b.1d26cb8","name":"","rules":[{"t":"set","p":"payload","pt":"msg","to":"\"clip ID \"&payload","tot":"jsonata"}],"action":"","property":"","from":"","to":"","reg":false,"x":2500,"y":1340,"wires":[[]]},{"id":"79ccc87d.9e6028","type":"inject","z":"585b49b.1d26cb8","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"stop-all","payload":"","payloadType":"str","x":100,"y":500,"wires":[["75f2d1d1.e5f92"]]},{"id":"ab52baff.c49f98","type":"inject","z":"585b49b.1d26cb8","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"start-all","payload":"","payloadType":"str","x":100,"y":540,"wires":[["75f2d1d1.e5f92"]]},{"id":"a1cdf62b.0414f8","type":"inject","z":"585b49b.1d26cb8","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"2","crontab":"","once":true,"onceDelay":0.1,"topic":"status-all","payload":"","payloadType":"str","x":110,"y":580,"wires":[["75f2d1d1.e5f92"]]},{"id":"9677f8db.0c7078","type":"udp in","z":"585b49b.1d26cb8","name":"","iface":"","port":"12345","ipv":"udp4","multicast":"false","group":"","datatype":"buffer","x":140,"y":120,"wires":[["e086c0c.12e234"]]},{"id":"9110933a.3d89c","type":"change","z":"585b49b.1d26cb8","name":"","rules":[{"t":"set","p":"topic","pt":"msg","to":"payload","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":660,"y":120,"wires":[["75f2d1d1.e5f92"]]},{"id":"e06dd35d.6a1d9","type":"switch","z":"585b49b.1d26cb8","name":"","property":"topic","propertyType":"msg","rules":[{"t":"eq","v":"/schedule","vt":"str"}],"checkall":"true","repair":false,"outputs":1,"x":470,"y":120,"wires":[["9110933a.3d89c"]]},{"id":"6e165e2.d2776a","type":"inject","z":"585b49b.1d26cb8","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"list-all","payload":"","payloadType":"str","x":90,"y":620,"wires":[["75f2d1d1.e5f92"]]},{"id":"c307a5c7.a10448","type":"change","z":"585b49b.1d26cb8","name":"","rules":[{"t":"set","p":"payload","pt":"msg","to":"payload.result[0].status.isRunning","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":980,"y":460,"wires":[["54e76aa5.c03ec4"]]},{"id":"54e76aa5.c03ec4","type":"switch","z":"585b49b.1d26cb8","name":"","property":"payload","propertyType":"msg","rules":[{"t":"true"},{"t":"false"}],"checkall":"true","repair":false,"outputs":2,"x":1170,"y":460,"wires":[["f7351c91.1bdd4","8159c5ad.4fc3f8"],["fbd6925d.7ef38","7a915c0.6bef1a4"]]},{"id":"f7351c91.1bdd4","type":"change","z":"585b49b.1d26cb8","name":"schedule on green","rules":[{"t":"set","p":"payload","pt":"msg","to":"[0,255,0]","tot":"json"}],"action":"","property":"","from":"","to":"","reg":false,"x":1410,"y":360,"wires":[["23bad0f9.7193c"]]},{"id":"fbd6925d.7ef38","type":"change","z":"585b49b.1d26cb8","name":"schedule off red","rules":[{"t":"set","p":"payload","pt":"msg","to":"[255,0,0]","tot":"json"}],"action":"","property":"","from":"","to":"","reg":false,"x":1400,"y":600,"wires":[["4eae10ff.f0e62"]]},{"id":"23bad0f9.7193c","type":"osc","z":"585b49b.1d26cb8","name":"Schedule on button control","path":"/style/bgcolor/1/3","metadata":false,"x":1740,"y":520,"wires":[["b05d2abc.c27498"]]},{"id":"f52a058e.d3f818","type":"osc","z":"585b49b.1d26cb8","name":"","path":"/style/text/1/1","metadata":false,"x":2530,"y":1140,"wires":[["b05d2abc.c27498"]]},{"id":"e086c0c.12e234","type":"osc","z":"585b49b.1d26cb8","name":"","path":"","metadata":false,"x":310,"y":120,"wires":[["e06dd35d.6a1d9"]]},{"id":"8414c055.fb11a","type":"osc","z":"585b49b.1d26cb8","name":"","path":"","metadata":false,"x":990,"y":220,"wires":[["b05d2abc.c27498"]]},{"id":"75f2d1d1.e5f92","type":"cronplus","z":"585b49b.1d26cb8","name":"","outputField":"topic","timeZone":"","persistDynamic":false,"commandResponseMsgOutput":"output2","outputs":2,"options":[{"name":"WDSlide 1","topic":"Slide 1","payloadType":"str","payload":"/press/bank/1/2","expressionType":"cron","expression":"0 45 10 ? * MON,TUE,WED,THU,FRI,SAT *","location":"","offset":"0","solarType":"all","solarEvents":"sunrise,sunset"},{"name":"WD_Video 1","topic":"Video 1","payloadType":"str","payload":"/press/bank/1/10","expressionType":"cron","expression":"0 05 11 ? * MON,TUE,WED,THU,FRI,SAT *","location":"","offset":"0","solarType":"all","solarEvents":"sunrise,sunset"},{"name":"WD_Video 2","topic":"Video 2","payloadType":"str","payload":"/press/bank/1/10","expressionType":"cron","expression":"0 22 13 ? * MON,TUE,WED,THU,FRI,SAT *","location":"","offset":"0","solarType":"all","solarEvents":"sunrise,sunset"},{"name":"WD_Video 3","topic":"Video 3","payloadType":"str","payload":"/press/bank/1/10","expressionType":"cron","expression":"0 39 15 ? * MON,TUE,WED,THU,FRI,SAT *","location":"","offset":"0","solarType":"all","solarEvents":"sunrise,sunset"},{"name":"WD_Video 4","topic":"Video 4","payloadType":"str","payload":"/press/bank/1/10","expressionType":"cron","expression":"0 56 17 ? * MON,TUE,WED,THU,FRI,SAT *","location":"","offset":"0","solarType":"all","solarEvents":"sunrise,sunset"},{"name":"Sunday Early Slide","topic":"Sunday Early Slide","payloadType":"str","payload":"/press/bank/1/2","expressionType":"cron","expression":"0 45 9 ? * SUN *","location":"","offset":"0","solarType":"all","solarEvents":"sunrise,sunset"},{"name":"SundayVideo1","topic":"SundayVideo1","payloadType":"str","payload":"/press/bank/1/10","expressionType":"cron","expression":"0 05 10 ? * SUN *","location":"","offset":"0","solarType":"all","solarEvents":"sunrise,sunset"},{"name":"SundayVideo2","topic":"SundayVideo2","payloadType":"str","payload":"/press/bank/1/10","expressionType":"cron","expression":"0 22 12 ? * SUN *","location":"","offset":"0","solarType":"all","solarEvents":"sunrise,sunset"},{"name":"SundayVideo3","topic":"SundayVideo2","payloadType":"str","payload":"/press/bank/1/10","expressionType":"cron","expression":"0 39 14 ? * SUN *","location":"","offset":"0","solarType":"all","solarEvents":"sunrise,sunset"},{"name":"SundayVideo4","topic":"SundayVideo4","payloadType":"str","payload":"/press/bank/1/10","expressionType":"cron","expression":"0 56 16 ? * SUN *","location":"","offset":"0","solarType":"all","solarEvents":"sunrise,sunset"},{"name":"projectorOnMon-Sat","topic":"projector on weekday","payloadType":"str","payload":"/press/bank/99/10","expressionType":"cron","expression":"0 30 10 ? * MON,TUE,WED,THU,FRI,SAT *","location":"","offset":"0","solarType":"all","solarEvents":"sunrise,sunset"},{"name":"projectorOffMon-Sat","topic":"projector off weekday","payloadType":"str","payload":"/press/bank/99/11","expressionType":"cron","expression":"0 15 20 ? * MON,TUE,WED,THU,FRI,SAT *","location":"","offset":"0","solarType":"all","solarEvents":"sunrise,sunset"},{"name":"projector on sunday","topic":"projector on sunday","payloadType":"str","payload":"/press/bank/99/10","expressionType":"cron","expression":"0 30 9 ? * SUN *","location":"","offset":"0","solarType":"all","solarEvents":"sunrise,sunset"},{"name":"projector off sunday","topic":"projector off sunday","payloadType":"str","payload":"/press/bank/99/11","expressionType":"cron","expression":"0 15 19 ? * SUN *","location":"","offset":"0","solarType":"all","solarEvents":"sunrise,sunset"}],"x":440,"y":460,"wires":[["8414c055.fb11a"],["c307a5c7.a10448"]]},{"id":"c8ec1533.38f478","type":"change","z":"585b49b.1d26cb8","name":"set colour off","rules":[{"t":"set","p":"payload","pt":"msg","to":"[255,0,0]","tot":"json"}],"action":"","property":"","from":"","to":"","reg":false,"x":2510,"y":1440,"wires":[["10c718ad.cdea37"]]},{"id":"75525179.13b33","type":"switch","z":"585b49b.1d26cb8","name":"","property":"payload","propertyType":"msg","rules":[{"t":"eq","v":"/press/bank/1/10","vt":"str"},{"t":"neq","v":"/press/bank/1/10","vt":"str"}],"checkall":"true","repair":false,"outputs":2,"x":2350,"y":1460,"wires":[["c8ec1533.38f478"],["2c134e40.1213f2"]]},{"id":"10c718ad.cdea37","type":"osc","z":"585b49b.1d26cb8","name":"Hold Button Colour","path":"/style/bgcolor/1/10","metadata":false,"x":2730,"y":1460,"wires":[[]]},{"id":"2c134e40.1213f2","type":"change","z":"585b49b.1d26cb8","name":"set colour off","rules":[{"t":"set","p":"payload","pt":"msg","to":"[0,0,0]","tot":"json"}],"action":"","property":"","from":"","to":"","reg":false,"x":2510,"y":1480,"wires":[["10c718ad.cdea37"]]},{"id":"96bb15e6.1f3198","type":"osc","z":"585b49b.1d26cb8","name":"Play Button Colour","path":"/style/bgcolor/1/11","metadata":false,"x":2730,"y":1540,"wires":[[]]},{"id":"9b71df8e.4e4f6","type":"change","z":"585b49b.1d26cb8","name":"set colour off","rules":[{"t":"set","p":"payload","pt":"msg","to":"[255,0,0]","tot":"json"}],"action":"","property":"","from":"","to":"","reg":false,"x":2510,"y":1520,"wires":[["96bb15e6.1f3198"]]},{"id":"a0ef9fff.a4d9a","type":"switch","z":"585b49b.1d26cb8","name":"","property":"payload","propertyType":"msg","rules":[{"t":"eq","v":"/press/bank/1/11","vt":"str"},{"t":"neq","v":"/press/bank/1/11","vt":"str"}],"checkall":"true","repair":false,"outputs":2,"x":2350,"y":1540,"wires":[["9b71df8e.4e4f6"],["b8348f07.9d709"]]},{"id":"b8348f07.9d709","type":"change","z":"585b49b.1d26cb8","name":"set colour off","rules":[{"t":"set","p":"payload","pt":"msg","to":"[0,0,0]","tot":"json"}],"action":"","property":"","from":"","to":"","reg":false,"x":2510,"y":1560,"wires":[["96bb15e6.1f3198"]]},{"id":"18f4c3ba.6f3e8c","type":"inject","z":"585b49b.1d26cb8","name":"rescan","props":[{"p":"topic","vt":"str"}],"repeat":"60","crontab":"","once":false,"onceDelay":0.1,"topic":"/press/bank/99/3","x":1540,"y":1060,"wires":[["6484fbf9.d03e44"]]},{"id":"a83ab847.f29f28","type":"inject","z":"585b49b.1d26cb8","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"5","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":150,"y":820,"wires":[["5d39ab30.bf9d04"]]},{"id":"7bd20e.3b98edf4","type":"change","z":"585b49b.1d26cb8","name":"","rules":[{"t":"set","p":"payload","pt":"msg","to":"mytime","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":540,"y":820,"wires":[["f95243a4.bdbe"]]},{"id":"f95243a4.bdbe","type":"osc","z":"585b49b.1d26cb8","name":"Set time and stop","path":"/style/text/1/1","metadata":false,"x":870,"y":820,"wires":[["b05d2abc.c27498"]]},{"id":"bf26ba8b.8d23e8","type":"function","z":"585b49b.1d26cb8","name":"add time and stop","func":"var time = msg.payload\nmsg.payload = \"Stop \" + time\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":2690,"y":1280,"wires":[[]]},{"id":"7ffc745d.d882ec","type":"status","z":"585b49b.1d26cb8","d":true,"name":"","scope":["75f2d1d1.e5f92"],"x":2260,"y":1080,"wires":[["660fdaff.34d404"]]},{"id":"9683dade.8687c8","type":"debug","z":"585b49b.1d26cb8","d":true,"name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":2710,"y":1080,"wires":[]},{"id":"660fdaff.34d404","type":"change","z":"585b49b.1d26cb8","d":true,"name":"","rules":[{"t":"set","p":"payload","pt":"msg","to":"status.text","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":2440,"y":1080,"wires":[["9683dade.8687c8"]]},{"id":"de6b463f.fa0f38","type":"debug","z":"585b49b.1d26cb8","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"mytime","targetType":"msg","statusVal":"","statusType":"auto","x":510,"y":920,"wires":[]},{"id":"5d39ab30.bf9d04","type":"simpletime","z":"585b49b.1d26cb8","name":"","mydate":false,"myymd":false,"myyear":false,"mymonth":false,"mymonthn":false,"mydom":false,"mydoy":false,"myday":false,"myhourpm":false,"myhour":false,"mytime":true,"mytimes":false,"myminute":false,"myminutes":false,"mysecond":false,"mymillis":false,"myepoch":false,"myrawdate":false,"mypm":false,"x":370,"y":820,"wires":[["7bd20e.3b98edf4","de6b463f.fa0f38"]]},{"id":"6484fbf9.d03e44","type":"osc","z":"585b49b.1d26cb8","name":"","path":"","metadata":false,"x":1810,"y":1060,"wires":[["b05d2abc.c27498"]]},{"id":"4eae10ff.f0e62","type":"osc","z":"585b49b.1d26cb8","name":"schedule off button control","path":"/style/bgcolor/1/11","metadata":false,"x":1740,"y":560,"wires":[["b05d2abc.c27498"]]},{"id":"8159c5ad.4fc3f8","type":"change","z":"585b49b.1d26cb8","name":"schedule off black","rules":[{"t":"set","p":"payload","pt":"msg","to":"[0,0,0]","tot":"json"}],"action":"","property":"","from":"","to":"","reg":false,"x":1410,"y":400,"wires":[["4eae10ff.f0e62"]]},{"id":"7a915c0.6bef1a4","type":"change","z":"585b49b.1d26cb8","name":"schedule on black","rules":[{"t":"set","p":"payload","pt":"msg","to":"[0,0,0]","tot":"json"}],"action":"","property":"","from":"","to":"","reg":false,"x":1410,"y":540,"wires":[["23bad0f9.7193c"]]},{"id":"e74bd38c.dccdb","type":"comment","z":"585b49b.1d26cb8","name":"OSC control on/off schedule","info":"","x":200,"y":60,"wires":[]},{"id":"98fce9c7.dc4b98","type":"inject","z":"585b49b.1d26cb8","d":true,"name":"test button press","props":[{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"/press/bank/99/10","x":1560,"y":1120,"wires":[["6484fbf9.d03e44"]]}]
4
  1. what version of OS are you running on the pi?
  2. if it is raspbian, is it the desktop or lite version?
  3. what version of NR and node.js are you running (see startup log)
5

My guess (i dont know about this OSP) is this that you are creating an endless loop, causing the system to exhaust resources.

image
image1591Ă—682 114 KB

The example on the flows site clearly separates operations

image
image895Ă—311 27.8 KB

6

Do you really need to run cron every 5 seconds?

7

The OS is Raspbian Buster Lite and NR version is v1.3.2.
Thanks
Steve

8

The "Status All" feeds back to a control panel on another Pi to show whether the schedule is running or not.
S

9

Do you mean with the UDP nodes as they are set to different ports so should not be talking to each other.
Thanks
Steve

10

Yes, i realise that, however, I am assuming sending a value to UDP:12321 causes a result to be fed back to UDP:12345?

As i said earlier, i dont know whats happening outside of the flow or the OSD parts.

Also, I observed from that 2nd demo flow, it only runs upon reception of an MQTT message (an external "command" - where as your flow seems to be operating every x secs.

11

Unless someone with both good knowledge of OSD and node-red joins in the chat, the only thing i can suggest is to add logging to see what is happening.

Try adding flogger nodes throughout your flow to see it if doing / not doing what you think it is.

12

Thanks for you help BTW I really appreciate it.
Basically the OSC input (top left) turns the scheduler on and off.
The OSC output on the bottom right send out commands out when the schedule triggers and status updates every 2 seconds.
Is there a way of seeing what resources are being used by nodes in node red?
Thanks
Steve

13

I have run some tests and when I disable the 'cron-plus' node I do not see the memory running out of control on the Pi. I may have to find a different way of scheduling things.
Thanks
Steve

14

All that suggests is you have prevented the loop or are not executing as often.

Cron-plus has been downloaded hundreds of thousands of times.

You need to understand what is happening and I'm pretty sure it is due to your arrangement of nodes.

15

Thanks I will keep looking as I like Cron-plus it is very useful.
S

16

You where right I don't think it is 'cron-plus' causing the issue directly. Everytime status-all on the cron-plus node the memory usage increase. I now simply call status-all when there is an incoming message that needs this feeding back. If the control panel computer has an issue it will not automatically get updated but I can live with that.
Thanks so much for you help
Steve

1 Like
17

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.