Restarting in middle of night and multiple failures to deploy

Node-Red is running my home automation. I have had no major issues until recently updating to v3.0.2. I'm not sure if my issue is related to this update or not.

In the middle of the night node-red seems to restart and activates things in the house.
I also now get several "Deploy failed: no response from server" messages when deploying. If I attempt deploy 3-4 times in a row, it will go through. I'm not sure if both issues are related.

Checking apport.log shows me the following.
ERROR: apport (pid 244021) Fri Aug 26 02:43:34 2022: called for pid 174718, signal 6, core limit 0, dump mode 1
ERROR: apport (pid 244021) Fri Aug 26 02:43:34 2022: executable: /usr/local/bin/node (command line "node-red")
ERROR: apport (pid 244021) Fri Aug 26 02:43:34 2022: executable does not belong to a package, ignoring

All help is appreciated.
Chris

you need to inspect the node-red logs (syslog) - that should provide some clues.

The "no response" may be due to several factors - e.g. your flows might be causing high CPU, node-red may be crashing more than you realise (hence the no-response) or indeed something else on the server is hogging CPU

The crashing may be related to a bad node or out of memory?

What OS and hardware is this running on?
How did you install node-red (the official script? on docker? as a HA plugin etc etc)?

Again,

I believe I installed with the official script. Where are the node-red logs?

I am running on Ubuntu 20.04.5 LTS.

In which case...

node-red-log

as documented here...

Note : if you are running as a service on the Raspberry Pi using our install script, you can use the node-red-log command to view the service log.

and here...

Running as a service

The install script for the Pi also sets it up to run as a service. This means it can run in the background and be enabled to automatically start on boot.

The following commands are provided to work with the service:

  • node-red-start - this starts the Node-RED service and displays its log output. Pressing Ctrl-C or closing the window does not stop the service; it keeps running in the background
  • node-red-stop - this stops the Node-RED service
  • node-red-restart - this stops and restarts the Node-RED service
  • node-red-log - this displays the log output of the service

Or you can look back through /var/log/syslog, or /var/log/syslog.1 if a new syslog has been started. Another possibility is that the machine rebooted, again you should be able to see that in syslog. Also the command top will tell you how long he system has been up.

Do things get activated if you just restart node-red? If not then do they get re-activated if you reboot (assuming node-red auto runs on boot)?

Yes. Things activate on boot and deploy. The full computer did not reboot, but something must be crashing just node-red. It is weird that it is usually between 2-4 in the morning.

Do these mean anything?
ERROR: apport (pid 244021) Fri Aug 26 02:43:34 2022: executable: /usr/local/bin/node (command line "node-red")
ERROR: apport (pid 244021) Fri Aug 26 02:43:34 2022: executable does not belong to a package, ignoring

Hard to say until we have context (i.e. the node-red logs)

What should i specifically be looking for? I see this which seems to be the cause: FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory

Correct?

How do I determine what is using the memory?

Yes, that will kill node-red.


searching the forum for "Reached heap limit"

searching the forum for "JavaScript heap out of memory"
https://discourse.nodered.org/search?q=JavaScript%20heap%20out%20of%20memory

Lots of reasons and lots of discussions, solutions and answers - start there.

Show us all the node-red related messages leading up to the failure.
cat /var/log/syslog |grep -i "node-red"
might do it

This keeps repeating itself.

Aug 26 05:41:14 desktop node-red-pi[1365]: Unhandled rejection TypeError: Cannot read properties of null (reading 'name')
Aug 26 05:41:14 desktop node-red-pi[1365]: at ServerNode.sendChanged (/home/reischman/.node-red/node_modules/node-red-contrib-life/nodes/server.js:55:56)
Aug 26 05:41:14 desktop node-red-pi[1365]: at /home/reischman/.node-red/node_modules/node-red-contrib-life/nodes/server.js:76:26
Aug 26 05:41:14 desktop node-red-pi[1365]: at /home/reischman/.node-red/node_modules/node-red-contrib-life/nodes/server.js:83:17
Aug 26 05:41:14 desktop node-red-pi[1365]: at tryCatcher (/home/reischman/.node-red/node_modules/bluebird/js/release/util.js:16:23)
Aug 26 05:41:14 desktop node-red-pi[1365]: at Promise._settlePromiseFromHandler (/home/reischman/.node-red/node_modules/bluebird/js/release/promise.js:547:31)
Aug 26 05:41:14 desktop node-red-pi[1365]: at Promise._settlePromise (/home/reischman/.node-red/node_modules/bluebird/js/release/promise.js:604:18)
Aug 26 05:41:14 desktop node-red-pi[1365]: at Promise._settlePromise0 (/home/reischman/.node-red/node_modules/bluebird/js/release/promise.js:649:10)
Aug 26 05:41:14 desktop node-red-pi[1365]: at Promise._settlePromises (/home/reischman/.node-red/node_modules/bluebird/js/release/promise.js:729:18)
Aug 26 05:41:14 desktop node-red-pi[1365]: at _drainQueueStep (/home/reischman/.node-red/node_modules/bluebird/js/release/async.js:93:12)
Aug 26 05:41:14 desktop node-red-pi[1365]: at _drainQueue (/home/reischman/.node-red/node_modules/bluebird/js/release/async.js:86:9)
Aug 26 05:41:14 desktop node-red-pi[1365]: at Async._drainQueues (/home/reischman/.node-red/node_modules/bluebird/js/release/async.js:102:5)
Aug 26 05:41:14 desktop node-red-pi[1365]: at Immediate.Async.drainQueues [as _onImmediate] (/home/reischman/.node-red/node_modules/bluebird/js/release/async.js:15:14)
Aug 26 05:41:14 desktop node-red-pi[1365]: at processImmediate (node:internal/timers:466:21)

node-red-contrib-life is most likely buggy and needs fixing. Unfortunately, the author did not put a link to github in his package.

You could try the forked version node-red-contrib-life360

Since that node has no documentation and has not been updated in years I suggest trying an alternative way of solving your problem.
If you want to try to work round it then look carefully at messages that you are sending it in case they may causing the problem.

What does "forked version" mean?

It means someone else duplicated the original node and made it better :slight_smile:

Tagging @hepcat72 (forks author) for reference

1 Like

Thanks to all 3 of you!

I also see the output-at-startup tick box that was likely driving me nuts also.

You might also want to improve the flows so that things do not get activated on a node-red restart. I solve this by using Retained topics in MQTT to keep all my system state information (which lights are on, temperature setpoints and so on). Then when node-red restarts it automatically gets given the previous state so there is minimum change on a reboot. Another way is to use persistent context to keep the state, and then have a flow that is run on startup that picks that up and does whatever is necessary to initialise node-red so it is in sync with the real world. I find that MQTT is the more convenient way though.

Do you have a "getting started with MQTT" link? I have an idea how it works and that it could probably simplify what I am doing. My flows have gotten massive. I started out programming one way in node-red and stuck with it. Probably nor really the best. I have no programming training so I just mess with things long enough to get them working. Then hope they never stop working. I suspect that everything I have going is sub-optimal.

-Chris

This will tell you pretty much all you need to know about MQTT. MQTT Essentials - All Core Concepts explained

It seems complex, but in fact is very easy to install and use.

Thanks again!