Hi, I had unexplained restarts of Node-Red yesterday and today of several devices:
VM #2:
Jul 08 22:57:02 webserver Node-RED[684]: 8 Jul 22:57:02 - [info] [mqtt-broker:FlashMQ] Verbindung zum Broker NodeRed@mqtt://localhost:1883 aufgebaut
Jul 15 06:48:22 webserver Node-RED[684]: 15 Jul 06:48:22 - [info] [mqtt-broker:FlashMQ] Verbindung zum Broker NodeRed@mqtt://localhost:1883 abgebaut
Jul 15 06:48:22 webserver systemd[1]: Stopping nodered.service - Node-RED graphical event wiring tool...
Jul 15 06:48:22 webserver Node-RED[684]: 15 Jul 06:48:22 - [info] Flows werden gestoppt
Jul 15 06:48:22 webserver Node-RED[684]: 15 Jul 06:48:22 - [info] [udp in:SMA HM] UDP-Empfangsbereitschaft gestoppt
Jul 15 06:48:22 webserver Node-RED[684]: 15 Jul 06:48:22 - [info] Flows sind gestoppt
Jul 15 06:48:22 webserver systemd[1]: Stopped nodered.service - Node-RED graphical event wiring tool.
Jul 15 06:48:22 webserver systemd[1]: Started nodered.service - Node-RED graphical event wiring tool.
Jul 15 06:48:23 webserver Node-RED[375486]: 15 Jul 06:48:23 - [info]
Jul 15 06:48:23 webserver Node-RED[375486]: Willkommen bei Node-RED
Jul 15 06:48:23 webserver Node-RED[375486]: ===================
Jul 15 06:48:23 webserver Node-RED[375486]: 15 Jul 06:48:23 - [info] Node-RED Version: v4.0.9
Jul 15 06:48:23 webserver Node-RED[375486]: 15 Jul 06:48:23 - [info] Node.js Version: v20.19.3
VM #1:
Jul 15 06:40:15 ubuntu24 Node-RED[687]: 15 Jul 06:40:15 - [info] [mqtt-broker:Ubuntu] Verbindung zum Broker lokal@mqtt://localhost:1883 abgebaut
Jul 15 06:40:15 ubuntu24 Node-RED[687]: 15 Jul 06:40:15 - [info] Flows werden gestoppt
Jul 15 06:40:15 ubuntu24 Node-RED[687]: 15 Jul 06:40:15 - [info] [udp in:SMA HM] UDP-Empfangsbereitschaft gestoppt
Jul 15 06:40:15 ubuntu24 systemd[1]: Stopping nodered.service - Node-RED graphical event wiring tool...
Jul 15 06:40:15 ubuntu24 Node-RED[687]: 15 Jul 06:40:15 - [info] [mqtt-broker:Webserver] Verbindung zum Broker UbuntuVM@mqtt://192.168.168.250:1883 abgebaut
Jul 15 06:40:15 ubuntu24 Node-RED[687]: 15 Jul 06:40:15 - [info] [mqtt-broker:NibePi] Verbindung zum Broker Ubuntu@mqtt://192.168.168.6:1883 abgebaut
Jul 15 06:40:15 ubuntu24 Node-RED[687]: 15 Jul 06:40:15 - [info] [mqtt-broker:Venus] Verbindung zum Broker Ubuntu@mqtt://192.168.168.7:1883 abgebaut
Jul 15 06:40:17 ubuntu24 Node-RED[687]: 15 Jul 06:40:17 - [info] Flows sind gestoppt
Jul 15 06:40:17 ubuntu24 systemd[1]: Stopped nodered.service - Node-RED graphical event wiring tool.
Jul 15 06:40:17 ubuntu24 systemd[1]: Started nodered.service - Node-RED graphical event wiring tool.
Jul 15 06:40:17 ubuntu24 Node-RED[377346]: 15 Jul 06:40:17 - [info]
Jul 15 06:40:17 ubuntu24 Node-RED[377346]: Willkommen bei Node-RED
Jul 15 06:40:17 ubuntu24 Node-RED[377346]: ===================
Jul 15 06:40:17 ubuntu24 Node-RED[377346]: 15 Jul 06:40:17 - [info] Node-RED Version: v4.0.9
Jul 15 06:40:17 ubuntu24 Node-RED[377346]: 15 Jul 06:40:17 - [info] Node.js Version: v20.19.3
Jul 15 06:40:17 ubuntu24 Node-RED[377346]: 15 Jul 06:40:17 - [info] Linux 6.8.0-63-generic x64 LE
Jul 15 06:40:18 ubuntu24 Node-RED[377346]: 15 Jul 06:40:18 - [info] Paletten-Nodes werden geladen
VenusOS:
@40000000686d862706aab7ec 8 Jul 22:57:01 - [info] [mqtt-broker:Webserver] Connected to broker: Venus@mqtt://192.168.168.250:1883
@40000000687485d41fa13c94 14 Jul 06:21:30 - [info] [mqtt-broker:Venus] Disconnected from broker: lokal@mqtt://localhost:1883
@40000000687485d50f59dcec 14 Jul 06:21:31 - [info] Stopping flows
@40000000687485d514c953c4 14 Jul 06:21:31 - [info] [udp in:SMA HM] udp listener stopped
@40000000687485d51a4810c4 14 Jul 06:21:31 - [info] [mqtt-broker:Webserver] Disconnected from broker: Venus@mqtt://192.168.168.250:1883
@40000000687485d715b4c91c 14 Jul 06:21:33 - [info] Stopped flows
@40000000687485d71b2c1bac *** starting node-red-venus ***
@40000000687485d71e9a05c4 *** Waiting for localsettings...
@40000000687485d71fdc5194 *** Starting in normal mode
@40000000687485da102f68e4 [info]: loading /data/home/nodered/.node-red/settings-venus.js failed
@40000000687485da1076e5ac [info]: loading /data/home/nodered/.node-red/settings-user.js failed
@40000000687485da2d967e7c 14 Jul 06:21:36 - [info]
@40000000687485da2d969204
@40000000687485da2d9699d4 Welcome to Node-RED
@40000000687485da2d96a1a4 ===================
@40000000687485da2d96a974
@40000000687485da2dbe2ea4 14 Jul 06:21:36 - [info] Node-RED version: v3.1.15
@40000000687485da2dd50264 14 Jul 06:21:36 - [info] Node.js version: v20.18.2
@40000000687485da2df40f9c 14 Jul 06:21:36 - [info] Linux 5.10.110 arm LE
never had this before !
The scary thing is: At this time I was still asleep.
a broken connection to a mqtt broker must not trigger a reboot !?
Greetings
Edit:
Jul 15 00:00:41 ubuntu24 systemd[1]: Reloaded flashmq.service - FlashMQ MQTT server.
Jul 15 06:40:15 ubuntu24 systemd[1]: Stopping flashmq.service - FlashMQ MQTT server...
Jul 15 06:40:15 ubuntu24 systemd[1]: flashmq.service: Deactivated successfully.
Jul 15 06:40:15 ubuntu24 systemd[1]: Stopped flashmq.service - FlashMQ MQTT server.
Jul 15 06:40:15 ubuntu24 systemd[1]: flashmq.service: Consumed 42min 20.564s CPU time, 13.9M memory peak, 0B memory swap peak.
Jul 15 06:40:15 ubuntu24 systemd[1]: Started flashmq.service - FlashMQ MQTT server.
Jul 15 06:40:15 ubuntu24 flashmq[377281]: [2025-07-15 06:40:15.539] [NOTICE] [main] Loading config. Reload: false.
Jul 15 06:40:15 ubuntu24 flashmq[377281]: [2025-07-15 06:40:15.551] [NOTICE] [main] Creating IPv4 non-SSL TCP listener on [0.0.0.0]:1883
Jul 15 06:40:15 ubuntu24 flashmq[377281]: [2025-07-15 06:40:15.552] [NOTICE] [main] Creating IPv6 non-SSL TCP listener on [::]:1883
Colin
15 July 2025 09:27
2
Node red did not crash, it was stopped and restarted.
Is your system accessible from the internet? If so then you may have been hacked.
1 Like
Colin
15 July 2025 09:42
3
Alternatively, check in /var/log/syslog in the VMs around the restart time to see if there was something else going on at that time.
I looked in journalctl and saw, that several services got stopped at 06:39:
Jul 15 06:39:01 ubuntu24 CRON[375774]: pam_unix(cron:session): session opened for user root(uid=0) by root(uid=0)
Jul 15 06:39:01 ubuntu24 CRON[375775]: (root) CMD ( [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
Jul 15 06:39:01 ubuntu24 CRON[375774]: pam_unix(cron:session): session closed for user root
Jul 15 06:39:04 ubuntu24 systemd[1]: Starting phpsessionclean.service - Clean php session files...
Jul 15 06:39:04 ubuntu24 systemd[1]: phpsessionclean.service: Deactivated successfully.
Jul 15 06:39:04 ubuntu24 systemd[1]: Finished phpsessionclean.service - Clean php session files.
Jul 15 06:39:41 ubuntu24 systemd[1]: Starting apt-daily-upgrade.service - Daily apt upgrade and clean activities...
Jul 15 06:39:48 ubuntu24 apt.systemd.daily[375883]: /usr/bin/unattended-upgrade:567: DeprecationWarning: This process (pid=375883) is multi-threaded, use of fork() may lead to deadlocks in the child.
Jul 15 06:39:48 ubuntu24 apt.systemd.daily[375883]: pid = os.fork()
Jul 15 06:39:50 ubuntu24 systemd[1]: Reloading requested from client PID 376095 ('systemctl') (unit apt-daily-upgrade.service)...
Jul 15 06:39:50 ubuntu24 systemd[1]: Reloading...
Jul 15 06:39:50 ubuntu24 systemd[1]: Reloading finished in 235 ms.
Jul 15 06:39:51 ubuntu24 systemd[1]: Reloading requested from client PID 376142 ('systemctl') (unit apt-daily-upgrade.service)...
Jul 15 06:39:51 ubuntu24 systemd[1]: Reloading...
Jul 15 06:39:51 ubuntu24 systemd[1]: Reloading finished in 211 ms.
Jul 15 06:39:51 ubuntu24 systemd[1]: Stopping apport.service - automatic crash report generation...
Jul 15 06:39:51 ubuntu24 systemd[1]: apport.service: Deactivated successfully.
Jul 15 06:39:51 ubuntu24 systemd[1]: Stopped apport.service - automatic crash report generation.
Jul 15 06:39:51 ubuntu24 systemd[1]: Starting apport.service - automatic crash report generation...
Jul 15 06:39:51 ubuntu24 systemd[1]: Finished apport.service - automatic crash report generation.
Jul 15 06:39:52 ubuntu24 systemd[1]: Reloading requested from client PID 376230 ('systemctl') (unit apt-daily-upgrade.service)...
Jul 15 06:39:52 ubuntu24 systemd[1]: Reloading...
Jul 15 06:39:52 ubuntu24 systemd[1]: Reloading finished in 206 ms.
Jul 15 06:39:52 ubuntu24 systemd[1]: apport-autoreport.path - Process error reports when automatic reporting is enabled (file watch) was skipped because of an unmet condition check (ConditionPathExists=/var/lib/apport/autoreport).
Jul 15 06:39:52 ubuntu24 systemd[1]: apport-autoreport.timer - Process error reports when automatic reporting is enabled (timer based) was skipped because of an unmet condition check (ConditionPathExists=/var/lib/apport/autoreport).
Jul 15 06:39:52 ubuntu24 systemd[1]: apport-forward.socket - Unix socket for apport crash forwarding was skipped because of an unmet condition check (ConditionVirtualization=container).
Jul 15 06:39:53 ubuntu24 dbus-daemon[680]: [system] Activating via systemd: service name='org.freedesktop.PackageKit' unit='packagekit.service' requested by ':1.50' (uid=0 pid=376291 comm="/usr/bin/gdbus call --system --dest org.freedeskto" label="unconfined")
Jul 15 06:39:53 ubuntu24 systemd[1]: Starting packagekit.service - PackageKit Daemon...
Jul 15 06:39:53 ubuntu24 PackageKit[376295]: daemon start
Jul 15 06:39:53 ubuntu24 dbus-daemon[680]: [system] Successfully activated service 'org.freedesktop.PackageKit'
Jul 15 06:39:53 ubuntu24 systemd[1]: Started packagekit.service - PackageKit Daemon.
Jul 15 06:39:55 ubuntu24 apt.systemd.daily[375883]: /usr/bin/unattended-upgrade:567: DeprecationWarning: This process (pid=375883) is multi-threaded, use of fork() may lead to deadlocks in the child.
Jul 15 06:39:55 ubuntu24 apt.systemd.daily[375883]: pid = os.fork()
Jul 15 06:39:57 ubuntu24 systemd[1]: Reexecuting requested from client PID 376492 ('systemctl') (unit apt-daily-upgrade.service)...
Jul 15 06:39:57 ubuntu24 systemd[1]: Reexecuting.
Jul 15 06:39:57 ubuntu24 systemd[1]: systemd 255.4-1ubuntu8.8 running in system mode (+PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY +P11KIT +QRENCOD
E +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
Jul 15 06:39:57 ubuntu24 systemd[1]: Detected virtualization kvm.
Jul 15 06:39:57 ubuntu24 systemd[1]: Detected architecture x86-64.
Jul 15 06:40:02 ubuntu24 grafana[824]: logger=cleanup t=2025-07-15T06:40:02.970260841+02:00 level=info msg="Completed cleanup jobs" duration=30.622585ms
Jul 15 06:40:03 ubuntu24 grafana[824]: logger=plugins.update.checker t=2025-07-15T06:40:03.094205475+02:00 level=info msg="Update check succeeded" duration=43.436449ms
Jul 15 06:40:15 ubuntu24 systemd[1]: Stopping cron.service - Regular background program processing daemon...
Jul 15 06:40:15 ubuntu24 systemd[1]: cron.service: Deactivated successfully.
Jul 15 06:40:15 ubuntu24 systemd[1]: Stopped cron.service - Regular background program processing daemon.
Jul 15 06:40:15 ubuntu24 systemd[1]: cron.service: Consumed 51.761s CPU time, 556.2M memory peak, 0B memory swap peak.
Jul 15 06:40:15 ubuntu24 systemd[1]: Started cron.service - Regular background program processing daemon.
Jul 15 06:40:15 ubuntu24 (cron)[377218]: cron.service: Referenced but unset environment variable evaluates to an empty string: EXTRA_OPTS
Jul 15 06:40:15 ubuntu24 systemd[1]: Stopping flashmq.service - FlashMQ MQTT server...
Jul 15 06:40:15 ubuntu24 cron[377218]: (CRON) INFO (pidfile fd = 3)
Jul 15 06:40:15 ubuntu24 cron[377218]: (CRON) INFO (Skipping @reboot jobs -- not system startup)
Jul 15 06:40:15 ubuntu24 systemd[1]: Stopping influxdb.service - InfluxDB is an open-source, distributed, time series database
Jul 15 06:40:15 ubuntu24 influxd[684]: ts=2025-07-15T04:40:15.479743Z lvl=info msg="Signal received, initializing clean shutdown..." log_id=0xd8DOW0000
Jul 15 06:40:15 ubuntu24 influxd[684]: ts=2025-07-15T04:40:15.479804Z lvl=info msg="Waiting for clean shutdown..." log_id=0xd8DOW0000
Jul 15 06:40:15 ubuntu24 influxd[684]: ts=2025-07-15T04:40:15.479886Z lvl=info msg="Listener closed" log_id=0xd8DOW0000 service=snapshot
Jul 15 06:40:15 ubuntu24 influxd[684]: ts=2025-07-15T04:40:15.479882Z lvl=info msg="Shutting down monitor service" log_id=0xd8DOW0000 service=monitor
Jul 15 06:40:15 ubuntu24 influxd[684]: ts=2025-07-15T04:40:15.479930Z lvl=info msg="Terminating storage of statistics" log_id=0xd8DOW0000 service=monitor
Jul 15 06:40:15 ubuntu24 influxd[684]: ts=2025-07-15T04:40:15.479955Z lvl=info msg="Terminating precreation service" log_id=0xd8DOW0000 service=shard-precreation
Jul 15 06:40:15 ubuntu24 influxd[684]: ts=2025-07-15T04:40:15.479967Z lvl=info msg="Terminating continuous query service" log_id=0xd8DOW0000 service=continuous_querier
Jul 15 06:40:15 ubuntu24 influxd[684]: ts=2025-07-15T04:40:15.480009Z lvl=info msg="Closing retention policy enforcement service" log_id=0xd8DOW0000 service=retention
Jul 15 06:40:15 ubuntu24 multipathd[362]: multipathd: shut down
Jul 15 06:40:15 ubuntu24 systemd[1]: Stopping multipathd.service - Device-Mapper Multipath Device Controller...
Jul 15 06:40:15 ubuntu24 Node-RED[687]: 15 Jul 06:40:15 - [info] [mqtt-broker:Ubuntu] Verbindung zum Broker lokal@mqtt://localhost:1883 abgebaut
Jul 15 06:40:15 ubuntu24 Node-RED[687]: 15 Jul 06:40:15 - [info] Flows werden gestoppt
Jul 15 06:40:15 ubuntu24 Node-RED[687]: 15 Jul 06:40:15 - [info] [udp in:SMA HM] UDP-Empfangsbereitschaft gestoppt
Jul 15 06:40:15 ubuntu24 systemd[1]: Stopping nodered.service - Node-RED graphical event wiring tool...
Jul 15 06:40:15 ubuntu24 systemd[1]: Stopping packagekit.service - PackageKit Daemon...
Jul 15 06:40:15 ubuntu24 systemd[1]: Stopping polkit.service - Authorization Manager...
Jul 15 06:40:15 ubuntu24 systemd[1]: packagekit.service: Deactivated successfully.
Jul 15 06:40:15 ubuntu24 systemd[1]: Stopped packagekit.service - PackageKit Daemon.
Jul 15 06:40:15 ubuntu24 systemd[1]: Starting packagekit.service - PackageKit Daemon...
Jul 15 06:40:15 ubuntu24 systemd[1]: Stopping qemu-guest-agent.service - QEMU Guest Agent...
Jul 15 06:40:15 ubuntu24 systemd[1]: qemu-guest-agent.service: Deactivated successfully.
Jul 15 06:40:15 ubuntu24 systemd[1]: Stopped qemu-guest-agent.service - QEMU Guest Agent.
Jul 15 06:40:15 ubuntu24 systemd[1]: polkit.service: Deactivated successfully.
Jul 15 06:40:15 ubuntu24 systemd[1]: Stopped polkit.service - Authorization Manager.
Jul 15 06:40:15 ubuntu24 systemd[1]: polkit.service: Consumed 19.152s CPU time, 2.5M memory peak, 0B memory swap peak.
Jul 15 06:40:15 ubuntu24 PackageKit[377255]: daemon start
Jul 15 06:40:15 ubuntu24 systemd[1]: Starting polkit.service - Authorization Manager...
Jul 15 06:40:15 ubuntu24 influxd[684]: ts=2025-07-15T04:40:15.510233Z lvl=info msg="Closed service" log_id=0xd8DOW0000 service=subscriber
Jul 15 06:40:15 ubuntu24 influxd[684]: ts=2025-07-15T04:40:15.512642Z lvl=info msg="Server shutdown completed" log_id=0xd8DOW0000
Jul 15 06:40:15 ubuntu24 systemd[1]: Started qemu-guest-agent.service - QEMU Guest Agent.
Jul 15 06:40:15 ubuntu24 sshd[1155]: Received signal 15; terminating.
Jul 15 06:40:15 ubuntu24 systemd[1]: Stopping ssh.service - OpenBSD Secure Shell server...
Jul 15 06:40:15 ubuntu24 systemd[1]: multipathd.service: Deactivated successfully.
Jul 15 06:40:15 ubuntu24 systemd[1]: Stopped multipathd.service - Device-Mapper Multipath Device Controller.
Influx, FlashMQ, Node-Red, ...
grep "upgrade " /var/log/dpkg.log
2025-07-15 06:39:49 upgrade python3-problem-report:all 2.28.1-0ubuntu3.7 2.28.1-0ubuntu3.8
2025-07-15 06:39:49 upgrade python3-apport:all 2.28.1-0ubuntu3.7 2.28.1-0ubuntu3.8
2025-07-15 06:39:49 upgrade apport-core-dump-handler:all 2.28.1-0ubuntu3.7 2.28.1-0ubuntu3.8
2025-07-15 06:39:49 upgrade apport:all 2.28.1-0ubuntu3.7 2.28.1-0ubuntu3.8
2025-07-15 06:39:56 upgrade libc-devtools:amd64 2.39-0ubuntu8.4 2.39-0ubuntu8.5
2025-07-15 06:39:56 upgrade libc6-dev:amd64 2.39-0ubuntu8.4 2.39-0ubuntu8.5
2025-07-15 06:39:56 upgrade libc-dev-bin:amd64 2.39-0ubuntu8.4 2.39-0ubuntu8.5
2025-07-15 06:39:56 upgrade libc6:amd64 2.39-0ubuntu8.4 2.39-0ubuntu8.5
2025-07-15 06:39:57 upgrade libc-bin:amd64 2.39-0ubuntu8.4 2.39-0ubuntu8.5
2025-07-15 06:39:58 upgrade locales:all 2.39-0ubuntu8.4 2.39-0ubuntu8.5
2025-07-15 06:40:18 upgrade libgnutls30t64:amd64 3.8.3-1.1ubuntu3.3 3.8.3-1.1ubuntu3.4
2025-07-15 07:58:10 upgrade firmware-sof-signed:all 2023.12.1-1ubuntu1.5 2023.12.1-1ubuntu1.6
it was an automatic update from "unattended-upgrade" - but I never noticed before a restart of Node-Red since years !
Colin
15 July 2025 12:22
5
I can see how upgrading packages can restart apps like influxdb, if packages on which it depends are updated. Generally, however, node-red is not installed using the package manager so I would not have expected to see it restarted. How was node-red installed?
bash <(curl -sL https://raw.githubusercontent.com/node-red/linux-installers/master/deb/update-nodejs-and-nodered)
as recommended
Colin
15 July 2025 20:59
7
What does this command showapt policy node-red
Janvi
15 July 2025 21:29
8
Becker:
VenusOS:
Had the same problem last week using Venus OS. Victrons hardware uses insufficient input circuit without optocouples. All input terminals go to 3 Volt uC pins for Cerbo GX, Ekrano and the new IO Expander directly. This causes serious ESD problems as soon as you connect the input terminals to a real hardware sensor with significant cable length. Connecting anything is typically done together with the use of NodeRed to process this peripherals. Before using NodeRed, the system run stable for 2 years only rebooted for updates. After using Node Red, several reboots happened every day. Indeed, it was not NodeRed but a hardware ESD problem. I used this optocouples to connect the Input signals. Although its easy to crash NodeRed / node.js with a single line of JS, everything works fine now. No reboots for over one week.
1 Like
N: Paket node-red kann nicht gefunden werden.
@Janvi I use a RPi4
Janvi
16 July 2025 05:17
10
might be same if you connect anything to the GPIO there
