Unexplained reboots of NR

Hi, I had unexplained restarts of Node-Red yesterday and today of several devices:

VM #2:

Jul 08 22:57:02 webserver Node-RED[684]: 8 Jul 22:57:02 - [info] [mqtt-broker:FlashMQ] Verbindung zum Broker NodeRed@mqtt://localhost:1883 aufgebaut
Jul 15 06:48:22 webserver Node-RED[684]: 15 Jul 06:48:22 - [info] [mqtt-broker:FlashMQ] Verbindung zum Broker NodeRed@mqtt://localhost:1883 abgebaut
Jul 15 06:48:22 webserver systemd[1]: Stopping nodered.service - Node-RED graphical event wiring tool...
Jul 15 06:48:22 webserver Node-RED[684]: 15 Jul 06:48:22 - [info] Flows werden gestoppt
Jul 15 06:48:22 webserver Node-RED[684]: 15 Jul 06:48:22 - [info] [udp in:SMA HM] UDP-Empfangsbereitschaft gestoppt
Jul 15 06:48:22 webserver Node-RED[684]: 15 Jul 06:48:22 - [info] Flows sind gestoppt
Jul 15 06:48:22 webserver systemd[1]: Stopped nodered.service - Node-RED graphical event wiring tool.
Jul 15 06:48:22 webserver systemd[1]: Started nodered.service - Node-RED graphical event wiring tool.
Jul 15 06:48:23 webserver Node-RED[375486]: 15 Jul 06:48:23 - [info]
Jul 15 06:48:23 webserver Node-RED[375486]: Willkommen bei Node-RED
Jul 15 06:48:23 webserver Node-RED[375486]: ===================
Jul 15 06:48:23 webserver Node-RED[375486]: 15 Jul 06:48:23 - [info] Node-RED Version: v4.0.9
Jul 15 06:48:23 webserver Node-RED[375486]: 15 Jul 06:48:23 - [info] Node.js  Version: v20.19.3

VM #1:

Jul 15 06:40:15 ubuntu24 Node-RED[687]: 15 Jul 06:40:15 - [info] [mqtt-broker:Ubuntu] Verbindung zum Broker lokal@mqtt://localhost:1883 abgebaut
Jul 15 06:40:15 ubuntu24 Node-RED[687]: 15 Jul 06:40:15 - [info] Flows werden gestoppt
Jul 15 06:40:15 ubuntu24 Node-RED[687]: 15 Jul 06:40:15 - [info] [udp in:SMA HM] UDP-Empfangsbereitschaft gestoppt
Jul 15 06:40:15 ubuntu24 systemd[1]: Stopping nodered.service - Node-RED graphical event wiring tool...
Jul 15 06:40:15 ubuntu24 Node-RED[687]: 15 Jul 06:40:15 - [info] [mqtt-broker:Webserver] Verbindung zum Broker UbuntuVM@mqtt://192.168.168.250:1883 abgebaut
Jul 15 06:40:15 ubuntu24 Node-RED[687]: 15 Jul 06:40:15 - [info] [mqtt-broker:NibePi] Verbindung zum Broker Ubuntu@mqtt://192.168.168.6:1883 abgebaut
Jul 15 06:40:15 ubuntu24 Node-RED[687]: 15 Jul 06:40:15 - [info] [mqtt-broker:Venus] Verbindung zum Broker Ubuntu@mqtt://192.168.168.7:1883 abgebaut
Jul 15 06:40:17 ubuntu24 Node-RED[687]: 15 Jul 06:40:17 - [info] Flows sind gestoppt
Jul 15 06:40:17 ubuntu24 systemd[1]: Stopped nodered.service - Node-RED graphical event wiring tool.
Jul 15 06:40:17 ubuntu24 systemd[1]: Started nodered.service - Node-RED graphical event wiring tool.
Jul 15 06:40:17 ubuntu24 Node-RED[377346]: 15 Jul 06:40:17 - [info]
Jul 15 06:40:17 ubuntu24 Node-RED[377346]: Willkommen bei Node-RED
Jul 15 06:40:17 ubuntu24 Node-RED[377346]: ===================
Jul 15 06:40:17 ubuntu24 Node-RED[377346]: 15 Jul 06:40:17 - [info] Node-RED Version: v4.0.9
Jul 15 06:40:17 ubuntu24 Node-RED[377346]: 15 Jul 06:40:17 - [info] Node.js  Version: v20.19.3
Jul 15 06:40:17 ubuntu24 Node-RED[377346]: 15 Jul 06:40:17 - [info] Linux 6.8.0-63-generic x64 LE
Jul 15 06:40:18 ubuntu24 Node-RED[377346]: 15 Jul 06:40:18 - [info] Paletten-Nodes werden geladen

VenusOS:

@40000000686d862706aab7ec 8 Jul 22:57:01 - [info] [mqtt-broker:Webserver] Connected to broker: Venus@mqtt://192.168.168.250:1883
@40000000687485d41fa13c94 14 Jul 06:21:30 - [info] [mqtt-broker:Venus] Disconnected from broker: lokal@mqtt://localhost:1883
@40000000687485d50f59dcec 14 Jul 06:21:31 - [info] Stopping flows
@40000000687485d514c953c4 14 Jul 06:21:31 - [info] [udp in:SMA HM] udp listener stopped
@40000000687485d51a4810c4 14 Jul 06:21:31 - [info] [mqtt-broker:Webserver] Disconnected from broker: Venus@mqtt://192.168.168.250:1883
@40000000687485d715b4c91c 14 Jul 06:21:33 - [info] Stopped flows
@40000000687485d71b2c1bac *** starting node-red-venus ***
@40000000687485d71e9a05c4 *** Waiting for localsettings...
@40000000687485d71fdc5194 *** Starting in normal mode
@40000000687485da102f68e4 [info]: loading /data/home/nodered/.node-red/settings-venus.js failed
@40000000687485da1076e5ac [info]: loading /data/home/nodered/.node-red/settings-user.js failed
@40000000687485da2d967e7c 14 Jul 06:21:36 - [info] 
@40000000687485da2d969204 
@40000000687485da2d9699d4 Welcome to Node-RED
@40000000687485da2d96a1a4 ===================
@40000000687485da2d96a974 
@40000000687485da2dbe2ea4 14 Jul 06:21:36 - [info] Node-RED version: v3.1.15
@40000000687485da2dd50264 14 Jul 06:21:36 - [info] Node.js  version: v20.18.2
@40000000687485da2df40f9c 14 Jul 06:21:36 - [info] Linux 5.10.110 arm LE

never had this before !

The scary thing is: At this time I was still asleep.

a broken connection to a mqtt broker must not trigger a reboot !?

Greetings

Edit:
FlashMQ has rebooted also:

Jul 15 00:00:41 ubuntu24 systemd[1]: Reloaded flashmq.service - FlashMQ MQTT server.
Jul 15 06:40:15 ubuntu24 systemd[1]: Stopping flashmq.service - FlashMQ MQTT server...
Jul 15 06:40:15 ubuntu24 systemd[1]: flashmq.service: Deactivated successfully.
Jul 15 06:40:15 ubuntu24 systemd[1]: Stopped flashmq.service - FlashMQ MQTT server.
Jul 15 06:40:15 ubuntu24 systemd[1]: flashmq.service: Consumed 42min 20.564s CPU time, 13.9M memory peak, 0B memory swap peak.
Jul 15 06:40:15 ubuntu24 systemd[1]: Started flashmq.service - FlashMQ MQTT server.
Jul 15 06:40:15 ubuntu24 flashmq[377281]: [2025-07-15 06:40:15.539] [NOTICE] [main] Loading config. Reload: false.
Jul 15 06:40:15 ubuntu24 flashmq[377281]: [2025-07-15 06:40:15.551] [NOTICE] [main] Creating IPv4 non-SSL TCP listener on [0.0.0.0]:1883
Jul 15 06:40:15 ubuntu24 flashmq[377281]: [2025-07-15 06:40:15.552] [NOTICE] [main] Creating IPv6 non-SSL TCP listener on [::]:1883

Node red did not crash, it was stopped and restarted.

Is your system accessible from the internet? If so then you may have been hacked.

1 Like

Alternatively, check in /var/log/syslog in the VMs around the restart time to see if there was something else going on at that time.

I looked in journalctl and saw, that several services got stopped at 06:39:


Jul 15 06:39:01 ubuntu24 CRON[375774]: pam_unix(cron:session): session opened for user root(uid=0) by root(uid=0)
Jul 15 06:39:01 ubuntu24 CRON[375775]: (root) CMD (  [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
Jul 15 06:39:01 ubuntu24 CRON[375774]: pam_unix(cron:session): session closed for user root
Jul 15 06:39:04 ubuntu24 systemd[1]: Starting phpsessionclean.service - Clean php session files...
Jul 15 06:39:04 ubuntu24 systemd[1]: phpsessionclean.service: Deactivated successfully.
Jul 15 06:39:04 ubuntu24 systemd[1]: Finished phpsessionclean.service - Clean php session files.
Jul 15 06:39:41 ubuntu24 systemd[1]: Starting apt-daily-upgrade.service - Daily apt upgrade and clean activities...
Jul 15 06:39:48 ubuntu24 apt.systemd.daily[375883]: /usr/bin/unattended-upgrade:567: DeprecationWarning: This process (pid=375883) is multi-threaded, use of fork() may lead to deadlocks in the child.
Jul 15 06:39:48 ubuntu24 apt.systemd.daily[375883]:   pid = os.fork()
Jul 15 06:39:50 ubuntu24 systemd[1]: Reloading requested from client PID 376095 ('systemctl') (unit apt-daily-upgrade.service)...
Jul 15 06:39:50 ubuntu24 systemd[1]: Reloading...
Jul 15 06:39:50 ubuntu24 systemd[1]: Reloading finished in 235 ms.
Jul 15 06:39:51 ubuntu24 systemd[1]: Reloading requested from client PID 376142 ('systemctl') (unit apt-daily-upgrade.service)...
Jul 15 06:39:51 ubuntu24 systemd[1]: Reloading...
Jul 15 06:39:51 ubuntu24 systemd[1]: Reloading finished in 211 ms.
Jul 15 06:39:51 ubuntu24 systemd[1]: Stopping apport.service - automatic crash report generation...
Jul 15 06:39:51 ubuntu24 systemd[1]: apport.service: Deactivated successfully.
Jul 15 06:39:51 ubuntu24 systemd[1]: Stopped apport.service - automatic crash report generation.
Jul 15 06:39:51 ubuntu24 systemd[1]: Starting apport.service - automatic crash report generation...
Jul 15 06:39:51 ubuntu24 systemd[1]: Finished apport.service - automatic crash report generation.
Jul 15 06:39:52 ubuntu24 systemd[1]: Reloading requested from client PID 376230 ('systemctl') (unit apt-daily-upgrade.service)...
Jul 15 06:39:52 ubuntu24 systemd[1]: Reloading...
Jul 15 06:39:52 ubuntu24 systemd[1]: Reloading finished in 206 ms.
Jul 15 06:39:52 ubuntu24 systemd[1]: apport-autoreport.path - Process error reports when automatic reporting is enabled (file watch) was skipped because of an unmet condition check (ConditionPathExists=/var/lib/apport/autoreport).
Jul 15 06:39:52 ubuntu24 systemd[1]: apport-autoreport.timer - Process error reports when automatic reporting is enabled (timer based) was skipped because of an unmet condition check (ConditionPathExists=/var/lib/apport/autoreport).
Jul 15 06:39:52 ubuntu24 systemd[1]: apport-forward.socket - Unix socket for apport crash forwarding was skipped because of an unmet condition check (ConditionVirtualization=container).
Jul 15 06:39:53 ubuntu24 dbus-daemon[680]: [system] Activating via systemd: service name='org.freedesktop.PackageKit' unit='packagekit.service' requested by ':1.50' (uid=0 pid=376291 comm="/usr/bin/gdbus call --system --dest org.freedeskto" label="unconfined")
Jul 15 06:39:53 ubuntu24 systemd[1]: Starting packagekit.service - PackageKit Daemon...
Jul 15 06:39:53 ubuntu24 PackageKit[376295]: daemon start
Jul 15 06:39:53 ubuntu24 dbus-daemon[680]: [system] Successfully activated service 'org.freedesktop.PackageKit'
Jul 15 06:39:53 ubuntu24 systemd[1]: Started packagekit.service - PackageKit Daemon.
Jul 15 06:39:55 ubuntu24 apt.systemd.daily[375883]: /usr/bin/unattended-upgrade:567: DeprecationWarning: This process (pid=375883) is multi-threaded, use of fork() may lead to deadlocks in the child.
Jul 15 06:39:55 ubuntu24 apt.systemd.daily[375883]:   pid = os.fork()
Jul 15 06:39:57 ubuntu24 systemd[1]: Reexecuting requested from client PID 376492 ('systemctl') (unit apt-daily-upgrade.service)...
Jul 15 06:39:57 ubuntu24 systemd[1]: Reexecuting.
Jul 15 06:39:57 ubuntu24 systemd[1]: systemd 255.4-1ubuntu8.8 running in system mode (+PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY +P11KIT +QRENCOD
E +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
Jul 15 06:39:57 ubuntu24 systemd[1]: Detected virtualization kvm.
Jul 15 06:39:57 ubuntu24 systemd[1]: Detected architecture x86-64.
Jul 15 06:40:02 ubuntu24 grafana[824]: logger=cleanup t=2025-07-15T06:40:02.970260841+02:00 level=info msg="Completed cleanup jobs" duration=30.622585ms
Jul 15 06:40:03 ubuntu24 grafana[824]: logger=plugins.update.checker t=2025-07-15T06:40:03.094205475+02:00 level=info msg="Update check succeeded" duration=43.436449ms
Jul 15 06:40:15 ubuntu24 systemd[1]: Stopping cron.service - Regular background program processing daemon...
Jul 15 06:40:15 ubuntu24 systemd[1]: cron.service: Deactivated successfully.
Jul 15 06:40:15 ubuntu24 systemd[1]: Stopped cron.service - Regular background program processing daemon.
Jul 15 06:40:15 ubuntu24 systemd[1]: cron.service: Consumed 51.761s CPU time, 556.2M memory peak, 0B memory swap peak.
Jul 15 06:40:15 ubuntu24 systemd[1]: Started cron.service - Regular background program processing daemon.
Jul 15 06:40:15 ubuntu24 (cron)[377218]: cron.service: Referenced but unset environment variable evaluates to an empty string: EXTRA_OPTS
Jul 15 06:40:15 ubuntu24 systemd[1]: Stopping flashmq.service - FlashMQ MQTT server...
Jul 15 06:40:15 ubuntu24 cron[377218]: (CRON) INFO (pidfile fd = 3)
Jul 15 06:40:15 ubuntu24 cron[377218]: (CRON) INFO (Skipping @reboot jobs -- not system startup)
Jul 15 06:40:15 ubuntu24 systemd[1]: Stopping influxdb.service - InfluxDB is an open-source, distributed, time series database
Jul 15 06:40:15 ubuntu24 influxd[684]: ts=2025-07-15T04:40:15.479743Z lvl=info msg="Signal received, initializing clean shutdown..." log_id=0xd8DOW0000
Jul 15 06:40:15 ubuntu24 influxd[684]: ts=2025-07-15T04:40:15.479804Z lvl=info msg="Waiting for clean shutdown..." log_id=0xd8DOW0000
Jul 15 06:40:15 ubuntu24 influxd[684]: ts=2025-07-15T04:40:15.479886Z lvl=info msg="Listener closed" log_id=0xd8DOW0000 service=snapshot
Jul 15 06:40:15 ubuntu24 influxd[684]: ts=2025-07-15T04:40:15.479882Z lvl=info msg="Shutting down monitor service" log_id=0xd8DOW0000 service=monitor
Jul 15 06:40:15 ubuntu24 influxd[684]: ts=2025-07-15T04:40:15.479930Z lvl=info msg="Terminating storage of statistics" log_id=0xd8DOW0000 service=monitor
Jul 15 06:40:15 ubuntu24 influxd[684]: ts=2025-07-15T04:40:15.479955Z lvl=info msg="Terminating precreation service" log_id=0xd8DOW0000 service=shard-precreation
Jul 15 06:40:15 ubuntu24 influxd[684]: ts=2025-07-15T04:40:15.479967Z lvl=info msg="Terminating continuous query service" log_id=0xd8DOW0000 service=continuous_querier
Jul 15 06:40:15 ubuntu24 influxd[684]: ts=2025-07-15T04:40:15.480009Z lvl=info msg="Closing retention policy enforcement service" log_id=0xd8DOW0000 service=retention
Jul 15 06:40:15 ubuntu24 multipathd[362]: multipathd: shut down
Jul 15 06:40:15 ubuntu24 systemd[1]: Stopping multipathd.service - Device-Mapper Multipath Device Controller...
Jul 15 06:40:15 ubuntu24 Node-RED[687]: 15 Jul 06:40:15 - [info] [mqtt-broker:Ubuntu] Verbindung zum Broker lokal@mqtt://localhost:1883 abgebaut
Jul 15 06:40:15 ubuntu24 Node-RED[687]: 15 Jul 06:40:15 - [info] Flows werden gestoppt
Jul 15 06:40:15 ubuntu24 Node-RED[687]: 15 Jul 06:40:15 - [info] [udp in:SMA HM] UDP-Empfangsbereitschaft gestoppt
Jul 15 06:40:15 ubuntu24 systemd[1]: Stopping nodered.service - Node-RED graphical event wiring tool...
Jul 15 06:40:15 ubuntu24 systemd[1]: Stopping packagekit.service - PackageKit Daemon...
Jul 15 06:40:15 ubuntu24 systemd[1]: Stopping polkit.service - Authorization Manager...
Jul 15 06:40:15 ubuntu24 systemd[1]: packagekit.service: Deactivated successfully.
Jul 15 06:40:15 ubuntu24 systemd[1]: Stopped packagekit.service - PackageKit Daemon.
Jul 15 06:40:15 ubuntu24 systemd[1]: Starting packagekit.service - PackageKit Daemon...
Jul 15 06:40:15 ubuntu24 systemd[1]: Stopping qemu-guest-agent.service - QEMU Guest Agent...
Jul 15 06:40:15 ubuntu24 systemd[1]: qemu-guest-agent.service: Deactivated successfully.
Jul 15 06:40:15 ubuntu24 systemd[1]: Stopped qemu-guest-agent.service - QEMU Guest Agent.
Jul 15 06:40:15 ubuntu24 systemd[1]: polkit.service: Deactivated successfully.
Jul 15 06:40:15 ubuntu24 systemd[1]: Stopped polkit.service - Authorization Manager.
Jul 15 06:40:15 ubuntu24 systemd[1]: polkit.service: Consumed 19.152s CPU time, 2.5M memory peak, 0B memory swap peak.
Jul 15 06:40:15 ubuntu24 PackageKit[377255]: daemon start
Jul 15 06:40:15 ubuntu24 systemd[1]: Starting polkit.service - Authorization Manager...
Jul 15 06:40:15 ubuntu24 influxd[684]: ts=2025-07-15T04:40:15.510233Z lvl=info msg="Closed service" log_id=0xd8DOW0000 service=subscriber
Jul 15 06:40:15 ubuntu24 influxd[684]: ts=2025-07-15T04:40:15.512642Z lvl=info msg="Server shutdown completed" log_id=0xd8DOW0000
Jul 15 06:40:15 ubuntu24 systemd[1]: Started qemu-guest-agent.service - QEMU Guest Agent.
Jul 15 06:40:15 ubuntu24 sshd[1155]: Received signal 15; terminating.
Jul 15 06:40:15 ubuntu24 systemd[1]: Stopping ssh.service - OpenBSD Secure Shell server...
Jul 15 06:40:15 ubuntu24 systemd[1]: multipathd.service: Deactivated successfully.
Jul 15 06:40:15 ubuntu24 systemd[1]: Stopped multipathd.service - Device-Mapper Multipath Device Controller.

Influx, FlashMQ, Node-Red, ...

grep "upgrade " /var/log/dpkg.log

2025-07-15 06:39:49 upgrade python3-problem-report:all 2.28.1-0ubuntu3.7 2.28.1-0ubuntu3.8
2025-07-15 06:39:49 upgrade python3-apport:all 2.28.1-0ubuntu3.7 2.28.1-0ubuntu3.8
2025-07-15 06:39:49 upgrade apport-core-dump-handler:all 2.28.1-0ubuntu3.7 2.28.1-0ubuntu3.8
2025-07-15 06:39:49 upgrade apport:all 2.28.1-0ubuntu3.7 2.28.1-0ubuntu3.8
2025-07-15 06:39:56 upgrade libc-devtools:amd64 2.39-0ubuntu8.4 2.39-0ubuntu8.5
2025-07-15 06:39:56 upgrade libc6-dev:amd64 2.39-0ubuntu8.4 2.39-0ubuntu8.5
2025-07-15 06:39:56 upgrade libc-dev-bin:amd64 2.39-0ubuntu8.4 2.39-0ubuntu8.5
2025-07-15 06:39:56 upgrade libc6:amd64 2.39-0ubuntu8.4 2.39-0ubuntu8.5
2025-07-15 06:39:57 upgrade libc-bin:amd64 2.39-0ubuntu8.4 2.39-0ubuntu8.5
2025-07-15 06:39:58 upgrade locales:all 2.39-0ubuntu8.4 2.39-0ubuntu8.5
2025-07-15 06:40:18 upgrade libgnutls30t64:amd64 3.8.3-1.1ubuntu3.3 3.8.3-1.1ubuntu3.4
2025-07-15 07:58:10 upgrade firmware-sof-signed:all 2023.12.1-1ubuntu1.5 2023.12.1-1ubuntu1.6

it was an automatic update from "unattended-upgrade" - but I never noticed before a restart of Node-Red since years !

I can see how upgrading packages can restart apps like influxdb, if packages on which it depends are updated. Generally, however, node-red is not installed using the package manager so I would not have expected to see it restarted. How was node-red installed?

bash <(curl -sL https://raw.githubusercontent.com/node-red/linux-installers/master/deb/update-nodejs-and-nodered)

as recommended

What does this command show
apt policy node-red

Had the same problem last week using Venus OS. Victrons hardware uses insufficient input circuit without optocouples. All input terminals go to 3 Volt uC pins for Cerbo GX, Ekrano and the new IO Expander directly. This causes serious ESD problems as soon as you connect the input terminals to a real hardware sensor with significant cable length. Connecting anything is typically done together with the use of NodeRed to process this peripherals. Before using NodeRed, the system run stable for 2 years only rebooted for updates. After using Node Red, several reboots happened every day. Indeed, it was not NodeRed but a hardware ESD problem. I used this optocouples to connect the Input signals. Although its easy to crash NodeRed / node.js with a single line of JS, everything works fine now. No reboots for over one week.

1 Like
N: Paket node-red kann nicht gefunden werden.

@Janvi I use a RPi4

might be same if you connect anything to the GPIO there

In that case I have no idea why package updates should restart node-red.

Unattended upgrades can be set to reboot if required.
What does the uptime command show?

If the VM had been restarted then that should have been obvious from syslog.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.