There´s no fixed time, but node-red is not reachable.
Putty -> Network error: Connection timed out
Powershell -> connect to host: connection timed out
Node-Red via browser -> site not reachable
ping -> host not reachable
nc -zv 22 -> connect failed: No route to host
nc -zv 1880 -> connect failed: No route to host
sudo nmap -sS -p22 -> Host seems down. If it´s really up, but blocking out our ping probes, try -Pn
sudo nmap -Pn -> 0 hosts up
Look into router -> no active connection
router messages -> device not answering
So I´ve looked into the Pi logfiles.
9 minutes before Node-Red wasn´t reachable the update and cleaning service was running, but Node-Red was able to reboot.
6 minutes later Node-Red was down again.
I´m alway getting this message:
"nodered.service: Main process exited, code=killed, status=11/SEGV"
In Node-Red is only running 1 flow. You can find it attached below. flows.json (62.9 KB)
I´m running headless Raspberry OS current version on a Raspberry Pi Zero.
This happens on both versions, Zero1 and Zero2.
The Pi isn´t reachable by WLAN oder LAN.
I´ve installed NR with the script.
I can´t restart NR, because I´ve to switch off/on the Pi.
The Pi isn´t reachable by SSH. Maybe I can write a script.
This is from the logfile:
Jan 10 09:01:15 PiZero Node-RED[295]: 10 Jan 09:01:15 - [error] SyntaxError: Unexpected identifier
Jan 10 09:01:15 PiZero Node-RED[295]: at Object.createReadStream (fs.js:2071:10)
Jan 10 09:01:15 PiZero Node-RED[295]: at SendStream.stream (/usr/lib/node_modules/node-red/node_modules/send/index.js:796:19)
Jan 10 09:01:15 PiZero Node-RED[295]: at SendStream.send (/usr/lib/node_modules/node-red/node_modules/send/index.js:707:8)
Jan 10 09:01:15 PiZero Node-RED[295]: at onstat (/usr/lib/node_modules/node-red/node_modules/send/index.js:729:10)
Jan 10 09:01:15 PiZero Node-RED[295]: at FSReqCallback.oncomplete (fs.js:193:5)
Jan 10 09:01:15 PiZero systemd[1]: nodered.service: Main process exited, code=exited, status=1/FAILURE
Jan 10 09:01:15 PiZero systemd[1]: nodered.service: Failed with result 'exit-code'.
Jan 10 09:01:21 PiZero kernel: [49447.160789] w1_master_driver w1_bus_master1: Attaching one wire slave 00.adc000000000 crc e6
Jan 10 09:01:21 PiZero kernel: [49447.186729] w1_master_driver w1_bus_master1: Family 0 for 00.adc000000000.e6 is not registered.
Jan 10 09:01:36 PiZero systemd[1]: nodered.service: Service RestartSec=20s expired, scheduling restart.
Jan 10 09:01:36 PiZero systemd[1]: nodered.service: Scheduled restart job, restart counter is at 1.
Jan 10 09:01:36 PiZero systemd[1]: Stopped Node-RED graphical event wiring tool.
Jan 10 09:01:36 PiZero systemd[1]: Started Node-RED graphical event wiring tool.
Jan 10 09:01:37 PiZero Node-RED[3778]: internal/fs/rimraf.js:251
Jan 10 09:01:37 PiZero Node-RED[3778]: for (let i = 1; i <= tries; i+)'ì õC÷Uy#003ì Áq°J#021>53a#006x3a#006x3M'ì õC÷Uy#003ì r°J#021>53a#006x3a#006x3q'ì õC÷Uy#003ì ár°J#021>53a#006x3a#006x3'ì õC÷Uy#003ì !Ï°J#021>53a#006x3a#006x3¹'ì õC÷Uy#003ì Ï°J#021>53a#006x3a#006x3Ý'ì õC÷Uy#003ì áÏ°J#021>53a#006x3a#006x3#001(ì õC÷Uy#003ì !l°J#021>53a#006x3a#006x3%(ì õC÷Uy#003ì Am°J#021>53a#006x3a#006x3I(ì õC÷Uy#003ì !r°J
Jan 10 09:01:37 PiZero Node-RED[3778]: ^
Jan 10 09:01:37 PiZero Node-RED[3778]: SyntaxError: Unexpected token ')'
Jan 10 09:01:37 PiZero Node-RED[3778]: at NativeModule.compileForInternalLoader (internal/bootstrap/loaders.js:285:18)
Jan 10 09:01:37 PiZero Node-RED[3778]: at nativeModuleRequire (internal/bootstrap/loaders.js:315:14)
Jan 10 09:01:37 PiZero Node-RED[3778]: at internal/fs/promises.js:45:28
Jan 10 09:01:37 PiZero Node-RED[3778]: at NativeModule.compileForInternalLoader (internal/bootstrap/loaders.js:286:7)
Jan 10 09:01:37 PiZero Node-RED[3778]: at nativeModuleRequire (internal/bootstrap/loaders.js:315:14)
Jan 10 09:01:37 PiZero Node-RED[3778]: at internal/modules/esm/get_source.js:14:12
Jan 10 09:01:37 PiZero Node-RED[3778]: at NativeModule.compileForInternalLoader (internal/bootstrap/loaders.js:286:7)
Jan 10 09:01:37 PiZero Node-RED[3778]: at nativeModuleRequire (internal/bootstrap/loaders.js:315:14)
Jan 10 09:01:37 PiZero Node-RED[3778]: at internal/modules/esm/loader.js:33:30
Jan 10 09:01:37 PiZero Node-RED[3778]: at NativeModule.compileForInternalLoader (internal/bootstrap/loaders.js:286:7)
Jan 10 09:01:37 PiZero systemd[1]: nodered.service: Main process exited, code=exited, status=1/FAILURE
Jan 10 09:01:38 PiZero systemd[1]: nodered.service: Failed with result 'exit-code'.
Jan 10 09:01:58 PiZero systemd[1]: nodered.service: Service RestartSec=20s expired, scheduling restart.
Jan 10 09:01:58 PiZero systemd[1]: nodered.service: Scheduled restart job, restart counter is at 2.
type or paste code here
This really doesn't help and won't help anyone in the future since releases come out at different times. If the Pi is not rechable, it sounds like you may have other issues.
You should attach a monitor, keyboard and mouse to make sure the Pi is booting, then make sure you hav SSH enabled. Once you have the pi up and running run node-red-stop and then run `node-red --safe' which will load NR but not start running it until you do a deploy.
I suspect you have a hardware problem of some sort, either the Pi, the SD card, or the PSU.
After you have power cycled the pi have a look and see what is in the log immediately before the reboot. The timestamps in syslog can be confusing at the start of the reboot because the pi does not know what the correct time is, so you have to look for the start of the reboot (look at the kernel timestamps) and then go back a bit to find the start of the boot.
@zenofmud You might not understand. After rebooting or restarting the Pi, SSH is working and node-red is running. And than a hour, a day or a week later SSH and NR are not running anymore und I´m not able to reach the Pi by WLAN or LAN.
And it´s totally uncomfortable to place a monitor and a keyboard where he´s running, because it´s outside.
@Colin
I´ve changed the PSU. It´s happening with 2 different PSU´s. The Pi is completely new, but it´s also happening with another Pi.
In the syslog at 6:47 you can see what´s happening. syslog.txt (230.1 KB)
Node-Red is running and suddenly I´m getting those messages:
Jan 15 06:53:59 PiZero systemd[1]: nodered.service: Main process exited, code=killed, status=11/SEGV
Jan 15 06:53:59 PiZero systemd[1]: nodered.service: Failed with result 'signal'.
Jan 15 08:20:24 PiZero kernel: [50599.309881] w1_master_driver w1_bus_master1: Family 0 for 00.d3c000000000.01 is not registered.
Jan 15 08:21:01 PiZero kernel: [50636.162868] w1_master_driver w1_bus_master1: Attaching one wire slave 00.33c000000000 crc e8
Jan 15 08:21:01 PiZero kernel: [50636.191564] w1_master_driver w1_bus_master1: Family 0 for 00.33c000000000.e8 is not registered.
Jan 15 08:17:25 PiZero systemd-modules-load[81]: Inserted module 'i2c_bcm2708'
Jan 15 08:17:25 PiZero systemd-modules-load[81]: Inserted module 'i2c_dev'
Jan 15 08:17:25 PiZero fake-hwclock[78]: Sa 15. Jan 07:17:01 UTC 2022
Jan 15 08:17:25 PiZero systemd-fsck[97]: e2fsck 1.44.5 (15-Dec-2018)
Jan 15 08:17:25 PiZero systemd-fsck[97]: rootfs: clean, 144280/947232 files, 977876/3822976 blocks
Jan 15 08:17:25 PiZero systemd[1]: Started Remount Root and Kernel File Systems.
Jan 15 08:17:25 PiZero kernel: [ 0.000000] Booting Linux on physical CPU 0x0
Jan 15 08:17:25 PiZero systemd[1]: Starting Create System Users...
Jan 15 08:17:25 PiZero systemd[1]: Condition check resulted in Rebuild Hardware Database being skipped.
Jan 15 08:17:25 PiZero systemd[1]: Starting Flush Journal to Persistent Storage...
Jan 15 08:17:25 PiZero systemd[1]: Starting Load/Save Random Seed...
Jan 15 08:17:25 PiZero systemd[1]: Started Set the console keyboard layout.
Jan 15 08:17:25 PiZero systemd[1]: Started udev Coldplug all Devices.
Jan 15 08:17:25 PiZero systemd[1]: Starting Helper to synchronize boot up for ifupdown...
Jan 15 08:17:25 PiZero systemd[1]: Started Create System Users.
Jan 15 08:17:25 PiZero kernel: [ 0.000000] Linux version 5.10.63+ (dom@buildbot) (arm-linux-gnueabihf-gcc-8 (Ubuntu/Linaro 8.4.0-3ubuntu1) 8.4.0, GNU ld (GNU Binutils for Ubuntu) 2.34) #1457 Tue Sep 28 11:24:51 BST 2021
Jan 15 08:17:25 PiZero kernel: [ 0.000000] CPU: ARMv6-compatible processor [410fb767] revision 7 (ARMv7), cr=00c5387d
Notice how the clock appears to go backwards after 08:21. The last good message in syslog is at 08:21:01. There are no normal shutdown messages after that which means that the Pi stopped suddenly. I don't see anything that says it is anything to do with node-red.
If you look later you will see
Jan 15 08:17:51 PiZero systemd[435]: Starting Sound Service...
Jan 15 09:17:41 PiZero systemd-timesyncd[232]: Synchronized to time server for the first time 162.159.200.1:123 (2.debian.pool.ntp.org).
So at 09:17 the wifi was up so the pi was able to connect to the internet and synchronise its time.
As to why node-red is occasionally killed I have no idea, but that is happening at different times, not when the pi dies.
Are you using 1-wire devices such as the DS18B20? If not then disable 1-wire in raspi config and it will stop the 1-wire messages. If you are using those devices then you have a fault with the wiring or a device. I don't think that is anything to do with the crash though.
What external connections to the pi have you got (wires connected I mean).
I would guess that it ran out of memory and it then crashed... Usual Node-RED reasons for this are dashboard ui-charts having too many points or too rapid data. More generically - handling large data chunk like images etc... but from what you say it sounds more like the former.
At that time I´ve powered off the Pi, because he wasn´t reachable.
So the time jump is because of synchronising the clock.
Those jumps are always when I´m powering off and restarting the Pi .
I´m not using 1-wire, so I´ve disabled it.
I´m using I²C for an analog card and SPI for the LAN adapter.
Do you mean node-red ran out of memory, or the Pi itself ran out of memory?
I would not expect either to cause the pi to stop reacting to Ping requests though.
I don´t know exactly the time when the Pi become unreachable.
I was waking up this morning, checked it and he wasn´t reachable
I´ve made a cronjob and scripts this morning to check, if ssh is running and the router is reachable.
If not the Pi should shut down. mThen I can see exactly the time.
On the Pi is running a flow for my heatpump.
The interval for checking 1 value and sending 1 value via I²C is every 30 seconds.
Additionally I´m getting informations by MQTT, but the MQTT server is running on another Pi, and also sending informations to a InfluxDB, what´s also running on another Pi.
CPU and memory load by running NR is 2-9% and 18,8%.
If the Pi is not responding then the cronjob and scripts will probably not run....
Set up a watchdog. On the Pi that is hanging, add a flow to send a timestamp via mqtt to another computer runnng NR and have that flow log the times. It could also check to see it the time interval has been exceeded and send you an alert.
watchdog is enabled, off course.
I´m not sure, if the Pi is really stucking or if the services or jobs are stopping.
As you can see in the logfile, there´s no reason.
As asked, please post the previous syslog. I want to see if the network failed before the start of the log you posted. It will be called syslog.1 provided the one you posted is still the current one.
[Edit] You can see when it failed by looking in Influx to see when the data stopped.
Is this Pi accessible from the internet?
have you ever installed rimraf.js?
The reason I'm asking is that there seams to be a problem with rimraf.js. Eithor coruption of the file or the code is obfuscated which would make me wonder if you have been hacked.
Once it throws an error, that error is repeated over and over in the log.
Jan 10 09:01:36 PiZero systemd[1]: Started Node-RED graphical event wiring tool.
Jan 10 09:01:37 PiZero Node-RED[3778]: internal/fs/rimraf.js:251
Jan 10 09:01:37 PiZero Node-RED[3778]: for (let i = 1; i <= tries; i+)'ì õC÷Uy#003ì Áq°J#021>53a#006x3a#006x3M'ì õC÷Uy#003ì r°J#021>53a#006x3a#006x3q'ì õC÷Uy#003ì ár°J#021>53a#006x3a#006x3'ì õC÷Uy#003ì !Ï°J#021>53a#006x3a#006x3¹'ì õC÷Uy#003ì Ï°J#021>53a#006x3a#006x3Ý'ì õC÷Uy#003ì áÏ°J#021>53a#006x3a#006x3#001(ì õC÷Uy#003ì !l°J#021>53a#006x3a#006x3%(ì õC÷Uy#003ì Am°J#021>53a#006x3a#006x3I(ì õC÷Uy#003ì !r°J
Jan 10 09:01:37 PiZero Node-RED[3778]: ^
Jan 10 09:01:37 PiZero Node-RED[3778]: SyntaxError: Unexpected token ')'
Jan 10 09:01:37 PiZero Node-RED[3778]: at NativeModule.compileForInternalLoader (internal/bootstrap/loaders.js:285:18)
Jan 10 09:01:37 PiZero Node-RED[3778]: at nativeModuleRequire (internal/bootstrap/loaders.js:315:14)
Jan 10 09:01:37 PiZero Node-RED[3778]: at internal/fs/promises.js:45:28
Jan 10 09:01:37 PiZero Node-RED[3778]: at NativeModule.compileForInternalLoader (internal/bootstrap/loaders.js:286:7)
Jan 10 09:01:37 PiZero Node-RED[3778]: at nativeModuleRequire (internal/bootstrap/loaders.js:315:14)
Jan 10 09:01:37 PiZero Node-RED[3778]: at internal/modules/esm/get_source.js:14:12
Jan 10 09:01:37 PiZero Node-RED[3778]: at NativeModule.compileForInternalLoader (internal/bootstrap/loaders.js:286:7)
Jan 10 09:01:37 PiZero Node-RED[3778]: at nativeModuleRequire (internal/bootstrap/loaders.js:315:14)
Jan 10 09:01:37 PiZero Node-RED[3778]: at internal/modules/esm/loader.js:33:30
Jan 10 09:01:37 PiZero Node-RED[3778]: at NativeModule.compileForInternalLoader (internal/bootstrap/loaders.js:286:7)
Jan 10 09:01:37 PiZero systemd[1]: nodered.service: Main process exited, code=exited, status=1/FAILURE
Jan 10 09:01:38 PiZero systemd[1]: nodered.service: Failed with result 'exit-code'.
Jan 10 09:01:58 PiZero systemd[1]: nodered.service: Service RestartSec=20s expired, scheduling restart.
Jan 10 09:01:58 PiZero systemd[1]: nodered.service: Scheduled restart job, restart counter is at 2.
Jan 10 09:01:58 PiZero systemd[1]: Stopped Node-RED graphical event wiring tool.
Jan 10 09:01:58 PiZero systemd[1]: Started Node-RED graphical event wiring tool.
Jan 10 09:01:59 PiZero Node-RED[3790]: internal/fs/rimraf.js:251
The error looks like it may well be SD card corruption.
Personally I would make sure that my backup of the .node-red folder is up to date (everything except the node_modules folder should be backed up, including hidden files starting with dot), bin the SD card (or at least quarantine it for the moment, install Raspbian on it and install node-red using the recommended script from the node-red docs.
Then run node-red once to make sure it is working, stop it and replace the .node-red folder from the backup.
Finally go into the .node-red folder and run npm install
which will automatically install any contrib nodes that you may have installed, and restart node-red.
