NodeRed Stops Flow

Dear All,

I installed two times nodeRed (using dockers). On installation was on a test PI 3 and the other on productive PI 4. Both are same image with Bullseye.

On my test PI 3, everything went fine and is running since two weeks without any interruption.
After this successful test, I also installed it on my second PI. But unfortunately, after each 4-5 minutes, the flow stops. Even without any interaction, the flow stops (suddenly).

Even with logging on trace level, I couldn't find any additional information about it.

Do you have any hints for me?

ct-smart-home-nodered-1 | 1 Mar 06:08:27 - [info] [zigbee2mqtt-out:CUPBOARD-0] Published to mqtt topic: zigbee2mqtt/CupBoard/set : {"brightness":0}
ct-smart-home-nodered-1 | 1 Mar 06:10:02 - [info] Stopping flows
ct-smart-home-nodered-1 | 1 Mar 06:10:02 - [debug] red/nodes/flows.stop : stopping flow : a88cac3fd58ce8fe
ct-smart-home-nodered-1 | 1 Mar 06:10:02 - [trace] [flow:a88cac3fd58ce8fe] stop flow
ct-smart-home-nodered-1 | 1 Mar 06:10:02 - [trace] Stopping node mqtt in:841199a2dc8370ca

Can you show us the full log from start to stop please?
How are you starting node red?

Sure, I can provide the startup sequenze.

ct-smart-home-nodered-1  | 1 Mar 08:15:02 - [trace] Stopped node mqtt-broker:ac124d006a3810b2 (109ms)
ct-smart-home-nodered-1  | 1 Mar 08:15:02 - [info] Stopped flows
ct-smart-home-nodered-1  | Starting dbus daemon
ct-smart-home-nodered-1  | Starting Avahi daemon
ct-smart-home-nodered-1  | Starting Node-Red
ct-smart-home-nodered-1  | 
ct-smart-home-nodered-1  | > ctnodered@1.5.0 start /usr/src/node-red
ct-smart-home-nodered-1  | > node $NODE_OPTIONS node_modules/node-red/red.js -v $FLOWS "--userDir" "/data"
ct-smart-home-nodered-1  | 
ct-smart-home-nodered-1  | 1 Mar 08:15:07 - [trace] git["--version"]
ct-smart-home-nodered-1  | 1 Mar 08:15:07 - [trace] git["config","--global","user.name"]
ct-smart-home-nodered-1  | 1 Mar 08:15:08 - [trace] git["config","--global","user.email"]
ct-smart-home-nodered-1  | 1 Mar 08:15:08 - [trace] git init: {"version":"2.34.1"}
ct-smart-home-nodered-1  | 1 Mar 08:15:08 - [info] 
ct-smart-home-nodered-1  | 
ct-smart-home-nodered-1  | Welcome to Node-RED
ct-smart-home-nodered-1  | ===================
ct-smart-home-nodered-1  | 
ct-smart-home-nodered-1  | 1 Mar 08:15:08 - [info] Node-RED version: v2.2.2
ct-smart-home-nodered-1  | 1 Mar 08:15:08 - [info] Node.js  version: v12.22.10
ct-smart-home-nodered-1  | 1 Mar 08:15:08 - [info] Linux 5.10.92-v8+ arm LE
ct-smart-home-nodered-1  | 1 Mar 08:15:08 - [info] Loading palette nodes
ct-smart-home-nodered-1  | 1 Mar 08:15:09 - [debug] Module: node-red-contrib-controltimer 0.1.0 /data/node_modules/node-red-contrib-controltimer
ct-smart-home-nodered-1  | 1 Mar 08:15:09 - [debug] Module: node-red-contrib-samsung-tv-control 1.3.15 /data/node_modules/node-red-contrib-samsung-tv-control
ct-smart-home-nodered-1  | 1 Mar 08:15:09 - [debug] Module: node-red-contrib-timerswitch 1.4.0 /data/node_modules/node-red-contrib-timerswitch
ct-smart-home-nodered-1  | 1 Mar 08:15:09 - [debug] Module: node-red-contrib-zigbee2mqtt 2.1.3 /data/node_modules/node-red-contrib-zigbee2mqtt
ct-smart-home-nodered-1  | 1 Mar 08:15:09 - [debug] Module: node-red-contrib-fritz 1.4.5 /usr/src/node-red/node_modules/node-red-contrib-fritz
ct-smart-home-nodered-1  | 1 Mar 08:15:09 - [debug] Module: node-red-contrib-homekit-bridged 1.4.3 /usr/src/node-red/node_modules/node-red-contrib-homekit-bridged
ct-smart-home-nodered-1  | 1 Mar 08:15:09 - [debug] Module: node-red-contrib-noble-bluetooth 0.9.1 /usr/src/node-red/node_modules/node-red-contrib-noble-bluetooth
ct-smart-home-nodered-1  | 1 Mar 08:15:09 - [debug] Module: node-red-contrib-tado-client 0.9.8 /usr/src/node-red/node_modules/node-red-contrib-tado-client
ct-smart-home-nodered-1  | 1 Mar 08:15:09 - [debug] Module: node-red-dashboard 3.1.6 /usr/src/node-red/node_modules/node-red-dashboard
ct-smart-home-nodered-1  | 1 Mar 08:15:13 - [info] Dashboard version 3.1.6 started at /ui
ct-smart-home-nodered-1  | 1 Mar 08:15:13 - [info] Settings file  : /data/settings.js
ct-smart-home-nodered-1  | 1 Mar 08:15:13 - [info] Context store  : 'default' [module=memory]
ct-smart-home-nodered-1  | 1 Mar 08:15:13 - [info] User directory : /data
ct-smart-home-nodered-1  | 1 Mar 08:15:13 - [info] Projects directory: /data/projects

It looks like the node red was stopped and started for some reason. It didn't crash.

I did ask for the full log from start to stop.

Also can you look in /var/log/syslog and see what is there around 08:15, in case there is a clue there.

Also can you monitor the exact times that it restarts and see if there is a pattern please. I note that the first one you posted was at 06:10:02 and this one is at 08:15:02, which suggests that something is restarting it every five minutes. How are you starting node-red?

Syslog seems pretty fine. As I mention, every 4-5 minutes, the stop of the flows happen.

Mar  1 08:14:04 rasp02 systemd[2582]: run-docker-runtime\x2drunc-moby-3f723bdfddceb7aab158d6fbcaef3cd6b0503982dc8fbb42075faf17f556e9ad-runc.sqh0Es.mount: Succeeded.
Mar  1 08:14:04 rasp02 systemd[1]: run-docker-runtime\x2drunc-moby-3f723bdfddceb7aab158d6fbcaef3cd6b0503982dc8fbb42075faf17f556e9ad-runc.sqh0Es.mount: Succeeded.
Mar  1 08:14:04 rasp02 systemd[1171]: run-docker-runtime\x2drunc-moby-3f723bdfddceb7aab158d6fbcaef3cd6b0503982dc8fbb42075faf17f556e9ad-runc.sqh0Es.mount: Succeeded.
Mar  1 08:14:34 rasp02 systemd[1]: run-docker-runtime\x2drunc-moby-3f723bdfddceb7aab158d6fbcaef3cd6b0503982dc8fbb42075faf17f556e9ad-runc.uwPPYT.mount: Succeeded.
Mar  1 08:14:34 rasp02 systemd[2582]: run-docker-runtime\x2drunc-moby-3f723bdfddceb7aab158d6fbcaef3cd6b0503982dc8fbb42075faf17f556e9ad-runc.uwPPYT.mount: Succeeded.
Mar  1 08:14:34 rasp02 systemd[1171]: run-docker-runtime\x2drunc-moby-3f723bdfddceb7aab158d6fbcaef3cd6b0503982dc8fbb42075faf17f556e9ad-runc.uwPPYT.mount: Succeeded.
Mar  1 08:15:02 rasp02 systemd[1]: docker-3f723bdfddceb7aab158d6fbcaef3cd6b0503982dc8fbb42075faf17f556e9ad.scope: Succeeded.
Mar  1 08:15:02 rasp02 systemd[1]: docker-3f723bdfddceb7aab158d6fbcaef3cd6b0503982dc8fbb42075faf17f556e9ad.scope: Consumed 16.032s CPU time.
Mar  1 08:15:02 rasp02 dockerd[612]: time="2022-03-01T08:15:02.574030163+01:00" level=info msg="ignoring event" container=3f723bdfddceb7aab158d6fbcaef3cd6b0503982dc8fbb42075faf17f556e9ad module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Mar  1 08:15:02 rasp02 containerd[648]: time="2022-03-01T08:15:02.574600328+01:00" level=info msg="shim disconnected" id=3f723bdfddceb7aab158d6fbcaef3cd6b0503982dc8fbb42075faf17f556e9ad
Mar  1 08:15:02 rasp02 systemd[2582]: var-lib-docker-overlay2-aab3f1ccb046423f88a631c5b3ab9dabece63ead5c9e88c74c755073d6449f08-merged.mount: Succeeded.
Mar  1 08:15:02 rasp02 systemd[1]: var-lib-docker-overlay2-aab3f1ccb046423f88a631c5b3ab9dabece63ead5c9e88c74c755073d6449f08-merged.mount: Succeeded.
Mar  1 08:15:02 rasp02 systemd[1171]: var-lib-docker-overlay2-aab3f1ccb046423f88a631c5b3ab9dabece63ead5c9e88c74c755073d6449f08-merged.mount: Succeeded.
Mar  1 08:15:03 rasp02 containerd[648]: time="2022-03-01T08:15:03.479258095+01:00" level=info msg="starting signal loop" namespace=moby path=/run/containerd/io.containerd.runtime.v2.task/moby/3f723bdfddceb7aab158d6fbcaef3cd6b0503982dc8fbb42075faf17f556e9ad pid=26652
Mar  1 08:15:03 rasp02 systemd[2582]: run-docker-runtime\x2drunc-moby-3f723bdfddceb7aab158d6fbcaef3cd6b0503982dc8fbb42075faf17f556e9ad-runc.bbyDvM.mount: Succeeded.
Mar  1 08:15:03 rasp02 systemd[1]: run-docker-runtime\x2drunc-moby-3f723bdfddceb7aab158d6fbcaef3cd6b0503982dc8fbb42075faf17f556e9ad-runc.bbyDvM.mount: Succeeded.
Mar  1 08:15:03 rasp02 systemd[1]: Started libcontainer container 3f723bdfddceb7aab158d6fbcaef3cd6b0503982dc8fbb42075faf17f556e9ad.
Mar  1 08:15:33 rasp02 systemd[1]: run-docker-runtime\x2drunc-moby-3f723bdfddceb7aab158d6fbcaef3cd6b0503982dc8fbb42075faf17f556e9ad-runc.gxNXpR.mount: Succeeded.
Mar  1 08:15:33 rasp02 systemd[1171]: run-docker-runtime\x2drunc-moby-3f723bdfddceb7aab158d6fbcaef3cd6b0503982dc8fbb42075faf17f556e9ad-runc.gxNXpR.mount: Succeeded.
Mar  1 08:15:33 rasp02 systemd[2582]: run-docker-runtime\x2drunc-moby-3f723bdfddceb7aab158d6fbcaef3cd6b0503982dc8fbb42075faf17f556e9ad-runc.gxNXpR.mount: Succeeded.
Mar  1 08:16:04 rasp02 systemd[2582]: run-docker-runtime\x2drunc-moby-3f723bdfddceb7aab158d6fbcaef3cd6b0503982dc8fbb42075faf17f556e9ad-runc.T3pEmg.mount: Succeeded.
Mar  1 08:16:04 rasp02 systemd[1]: run-docker-runtime\x2drunc-moby-3f723bdfddceb7aab158d6fbcaef3cd6b0503982dc8fbb42075faf17f556e9ad-runc.T3pEmg.mount: Succeeded.
Mar  1 08:16:04 rasp02 systemd[1171]: run-docker-runtime\x2drunc-moby-3f723bdfddceb7aab158d6fbcaef3cd6b0503982dc8fbb42075faf17f556e9ad-runc.T3pEmg.mount: Succeeded.
Mar  1 08:16:34 rasp02 systemd[2582]: run-docker-runtime\x2drunc-moby-3f723bdfddceb7aab158d6fbcaef3cd6b0503982dc8fbb42075faf17f556e9ad-runc.vNTJ54.mount: Succeeded.
Mar  1 08:16:34 rasp02 systemd[1]: run-docker-runtime\x2drunc-moby-3f723bdfddceb7aab158d6fbcaef3cd6b0503982dc8fbb42075faf17f556e9ad-runc.vNTJ54.mount: Succeeded.
Mar  1 08:16:34 rasp02 systemd[1171]: run-docker-runtime\x2drunc-moby-3f723bdfddceb7aab158d6fbcaef3cd6b0503982dc8fbb42075faf17f556e9ad-runc.vNTJ54.mount: Succeeded.

For startup, I using on both Pis following script:

I am not an expert in docker, but it looks to me as if something is not right there. I think you need someone who knows about docker to tell us what is happening.

As I mention, why it is working on the other PI very well without any interruption?

Also a native installation (on the problem PI) has the same effect (behavior and logging).

OK, that is good. Show us the log from that. Run
node-red-stop
node-red-start
and post the full output here, up to the first restart.

Follow output is coming from the native installation:

1 Mar 11:48:41 - [info] Linux 5.10.92-v8+ arm LE
1 Mar 11:48:43 - [info] Loading palette nodes
1 Mar 11:48:45 - [info] Settings file  : /home/pi/.node-red/settings.js
1 Mar 11:48:45 - [info] Context store  : 'default' [module=memory]
1 Mar 11:48:45 - [info] User directory : /home/pi/.node-red
1 Mar 11:48:45 - [warn] Projects disabled : editorTheme.projects.enabled=false
1 Mar 11:48:45 - [info] Flows file     : /home/pi/.node-red/flows.json
1 Mar 11:48:45 - [info] Creating new flow file
1 Mar 11:48:45 - [warn]
---------------------------------------------------------------------
Your flow credentials file is encrypted using a system-generated key.
If the system-generated key is lost for any reason, your credentials
file will not be recoverable, you will have to delete it and re-enter
your credentials.
You should set your own key using the 'credentialSecret' option in
your settings file. Node-RED will then re-encrypt your credentials
file using your chosen key the next time you deploy a change.
---------------------------------------------------------------------
1 Mar 11:48:45 - [info] Server now running at http://127.0.0.1:1880/
1 Mar 11:48:45 - [info] Starting flows
1 Mar 11:48:45 - [info] Started flows
1 Mar 11:50:01 - [info] Stopping flows
1 Mar 11:50:01 - [info] Stopped flows
nodered.service: Succeeded.
nodered.service: Consumed 6.647s CPU time.

syslog:

Mar  1 11:49:10  rasp02 dhcpcd[521]: wlan0: Router Advertisement from fe80::xxxxxxxxxxxx
Mar  1 11:50:01 rasp02 Node-RED[31159]: 1 Mar 11:50:01 - [info] Stopping flows
Mar  1 11:50:01 rasp02 Node-RED[31159]: 1 Mar 11:50:01 - [info] Stopped flows
Mar  1 11:50:01 rasp02 systemd[1]: nodered.service: Succeeded.

Please post the full output from (and including) the commands
node-red-stop
node-red-start
up to the first restart. Or does it not restart itself?

Notice again it stopped at exactly a five minute boundary. Have you got anything in the flows that does anything through the node red admin api? Have you got anything that runs every 5 minutes?

I did it. No self restart was happen.

user@rasp02:~$ node-red-start

Start Node-RED

Once Node-RED has started, point a browser at http://172.17.0.69:1880
On Pi Node-RED works better with the Firefox or Chrome browser

Use node-red-stop to stop Node-RED
Use node-red-start to start Node-RED again
Use node-red-log to view the recent log output
Use sudo systemctl enable nodered.service to autostart Node-RED at every boot
Use sudo systemctl disable nodered.service to disable autostart on boot

To find more nodes and example flows - go to http://flows.nodered.org

Starting as a systemd service.
1 Mar 11:48:41 - [info]
Welcome to Node-RED

1 Mar 11:48:41 - [info] Node-RED version: v2.1.6
1 Mar 11:48:41 - [info] Node.js version: v12.22.5
1 Mar 11:48:41 - [info] Linux 5.10.92-v8+ arm LE
1 Mar 11:48:43 - [info] Loading palette nodes
1 Mar 11:48:45 - [info] Settings file : /home/pi/.node-red/settings.js
1 Mar 11:48:45 - [info] Context store : 'default' [module=memory]
1 Mar 11:48:45 - [info] User directory : /home/pi/.node-red
1 Mar 11:48:45 - [warn] Projects disabled : editorTheme.projects.enabled=false
1 Mar 11:48:45 - [info] Flows file : /home/pi/.node-red/flows.json
1 Mar 11:48:45 - [info] Creating new flow file
1 Mar 11:48:45 - [warn]

Your flow credentials file is encrypted using a system-generated key.
If the system-generated key is lost for any reason, your credentials
file will not be recoverable, you will have to delete it and re-enter
your credentials.
You should set your own key using the 'credentialSecret' option in
your settings file. Node-RED will then re-encrypt your credentials
file using your chosen key the next time you deploy a change.

1 Mar 11:48:45 - [info] Server now running at xxxxxfadf:1880/
1 Mar 11:48:45 - [info] Starting flows
1 Mar 11:48:45 - [info] Started flows
1 Mar 11:50:01 - [info] Stopping flows
1 Mar 11:50:01 - [info] Stopped flows
nodered.service: Succeeded.
nodered.service: Consumed 6.647s CPU time.

^C
user@rasp02:~$

The flow (is empty) and configuration is default. No changes from my side happen.

This is very odd. Can you post the contents of /lib/systemd/system/nodered.service.

Note once again that it is on a 5 minute boundary. Something is stopping node red.

Please find the content of nodered.service here:

root@raps02:~#  cat /lib/systemd/system/nodered.service 
# systemd service file to start Node-RED

[Unit]
Description=Node-RED graphical event wiring tool
Wants=network.target
Documentation=http://nodered.org/docs/hardware/raspberrypi.html

[Service]
Type=simple
# Run as normal pi user - change to the user name you wish to run Node-RED as
User=pi
Group=pi
WorkingDirectory=/home/pi

Environment="NODE_OPTIONS=--max_old_space_size=512"
# define an optional environment file in Node-RED's user directory to set custom variables externally
EnvironmentFile=-/home/pi/.node-red/environment
# uncomment and edit next line if you need an http proxy
#Environment="HTTP_PROXY=my.httpproxy.server.address"
# uncomment the next line for a more verbose log output
#Environment="NODE_RED_OPTIONS=-v"
# uncomment next line if you need to wait for time sync before starting
#ExecStartPre=/bin/bash -c '/bin/journalctl -b -u systemd-timesyncd | /bin/grep -q "systemd-timesyncd.* Synchronized to time server"'

ExecStart=/usr/bin/env node-red-pi $NODE_OPTIONS $NODE_RED_OPTIONS
#ExecStart=/usr/bin/env node $NODE_OPTIONS red.js $NODE_RED_OPTIONS
# Use SIGINT to stop
KillSignal=SIGINT
# Auto restart on crash
Restart=on-failure
RestartSec=20
# Tag things in the log
SyslogIdentifier=Node-RED
#StandardOutput=syslog

[Install]
WantedBy=multi-user.target
root@raps02:~#

Admin edit, added triple backticks round the file to make it easier to read.

That looks ok. One think that confuses me, I got the impression that this was a new installation, yet it is not using the latest version of node-red. The recommended install method would have installed the latest version.

I also notice that you have setup at least one other user and also the root account, which is not generally recommended. I wonder if the problem is something to do with other changes that have been made.

Would it be possible to create a fresh install of Pi OS on a new card, install node red using the recommended method and checking that there is no problem with that? Then we will know if it is something to do with your OS setup. Though I have no idea what.

I found now this issue. A cleanup script from other application, kills the node process.

Which presumably was running every 5 minutes.

1 Like

You are right!

1 Like

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.