I am using Node-red to listen the data form The things stack (LoRa) with MQTT node and then save the data to SQL and showing it on Dashboard.
The integration works fine but every night node red stops running with the error "Disconnected from broker". The log is below. I have already set the log level to trace to get detailed logs.
4 Dec 00:00:44 - [trace] Stopped node ui_group:820f6eb8cfb19b5f (2217ms)
4 Dec 00:00:44 - [trace] Stopped node file:acd436a6.915b58 (11167ms)
4 Dec 00:00:44 - [trace] Stopped node file:c81b1fa38ca9a1a9 (9780ms)
4 Dec 00:00:44 - [trace] Stopped node persist out:920af82a52d974fe (8162ms)
4 Dec 00:00:44 - [trace] Stopped node persist out:208943d6ca80d596 (8077ms)
4 Dec 00:00:44 - [trace] Stopped node persist out:d864d3592b0a402d (8042ms)
4 Dec 00:00:44 - [trace] Stopped node file:c68e384ebbf990ca (7684ms)
4 Dec 00:00:44 - [trace] Stopped node persist-store:fc55cc01139f2355 (2270ms)
4 Dec 00:00:44 - [trace] Stopped node persist-store:44000ea4a5a8e0cc (2190ms)
4 Dec 00:00:44 - [trace] Stopped node persist-store:5783905d29c6d349 (2188ms)
4 Dec 00:00:45 - [trace] Stopped node MySQLdatabase:8d9e671a.fe43f8 (4813ms)
4 Dec 00:00:46 - [info] [mqtt-broker:pce-rs485-ln] Disconnected from broker: mqtts://thethings.pce-service.net:8883
4 Dec 00:00:46 - [trace] Stopped node mqtt-broker:5a4affde40f9961b (4883ms)
4 Dec 00:00:46 - [info] [mqtt-broker:pce-lht65] Disconnected from broker: mqtts://thethings.pce-service.net:8883
4 Dec 00:00:46 - [trace] Stopped node mqtt-broker:331ab51.7038a4a (4848ms)
4 Dec 00:00:46 - [info] [mqtt-broker:pce-df703] Disconnected from broker: mqtts://thethings.pce-service.net:8883
4 Dec 00:00:46 - [trace] Stopped node mqtt-broker:4ecf1b4c38c80bc6 (5076ms)
4 Dec 00:00:47 - [error] [mqtt in:pce-lht65] Error stopping node: Close timed out
4 Dec 00:00:47 - [debug] undefined
4 Dec 00:00:49 - [error] [mqtt in:pce-df703] Error stopping node: Close timed out
4 Dec 00:00:49 - [debug] undefined
4 Dec 00:00:50 - [error] [mqtt in:pce-rs485-ln] Error stopping node: Close timed out
4 Dec 00:00:50 - [debug] undefined
4 Dec 00:00:50 - [info] Stopped flows
Stopped Node-RED graphical event wiring tool.
I am not able to figure out what is the issue here. Why it is keep happening ever night between midnight and 1 O'clock.
Additional Info:
The Things stack and Node-red are running on the same remote server Ubantu 18.04
It appears from the logs that something is explicitly asking Node-RED to shutdown. As it is we don't have enough information to say what is asking it to shutdown or why.
What we can say is that you have multiple broker config nodes pointing at the same broker which normally isn't required (you can share a broker config node between many mqtt-in/mqtt-out nodes).
Given the timeout errors on some of the broker connections when asked to shutdown it implies that they have shutdown messages configured and it is failing to send them possibly because the broker is also being at the same time.
Given the time window, I would be tempted to look at your logrotate settings to see if it's restarting everything at the start of the new day with a fresh log file.
Is there any way I can get more information about what is asking Node-RED to shutdown?
Yes, I agree with you. I also had the same doubt and tried to use a single MQTT node. But in The Things stack each application has a different ID and Password to connect MQTT. So, I need one MQTT node for each application. (More info here). If you can suggest a way around, I would be happy to implement.
I tried to look into logrotate settings. Here are results with commands:
cat /etc/logrotate.conf
# see "man logrotate" for details
# rotate log files weekly
weekly
# use the syslog group by default, since this is the owning group
# of /var/log/syslog.
su root syslog
# keep 4 weeks worth of backlogs
rotate 4
# create new (empty) log files after rotating old ones
create
# uncomment this if you want your log files compressed
#compress
# packages drop log rotation information into this directory
include /etc/logrotate.d
# no packages own wtmp, or btmp -- we'll rotate them here
/var/log/wtmp {
missingok
monthly
create 0664 root utmp
rotate 1
}
/var/log/btmp {
missingok
monthly
create 0660 root utmp
rotate 1
}
# system-specific logs may be configured here
Is there any way I can get more information about what is asking Node-RED to shutdown?
No, Node-RED will just be responding to a signal sent to it.
The logrotate stuff looks fine, but it was just a guess based on the rough time of day, poke around the crontab for other things set to run at midnight or daily
the MQTT client node stopping error(s) is a known issue if you use a MQTT broker inside of Node-RED - most folk use Mosquitto as a stand alone broker outside of Node-RED (usually on the same box). But that is nothing to do with why it is shutting down in the first place.
Yes, that is what I was trying to explain. The log is from the MQTT broker which is running on server mqtts://thethings.pce-service.net:8883 and not inside Node-RED.
Let me use this opportunity to clear my understanding of MQTT as I have limited experience
MQTT Broker == MQTT server
MQTT Client == MQTT-in node
or am I missing something?
I dont know if it helps but in the past i have faced some cases where node red crashes with uncaught exception deep down somewhere when mqtt server unexpectedly interrupts connection. Could your mqtt broker (server) be disconnecting you somehow?
Is there a way to stop node red programmatically? Just speculating here maybe in the recent versions it started to catch such errors and shut down? Not sure,
It depends what you mean by programmatically. Node red can stop itself certainly, or it could be stopped from the flow using an exec node. Node red will not stop itself without providing a message in the log, unless there is an obscure bug. The whole point of catching exceptions is to allow node-red to continue if possible.
I think I am able to figure it out!!!
Early speculation by @hardillb was right that somethings is asking Node-RED to shutdown. I looked into syslogs and then into crontab files as suggested by @hardillb and @Colin and found out that the server is scheduled to reboot every midnight. I found this is the crontab file 0 0 * * * /sbin/shutdown -r now.
So yesterday I wrote restarting command for Node-RED at reboot in crontab file and today morning it I checked it again and it was working for the whole night.
Thanks all for chipping in. Learned a lot about crontab and syslogs!!
Oh yes, I totally forgot about systemd command. I will modify that.
I am running on Ubuntu 18.04
I have installed node red using Running on Raspberry Pi : Node-RED instructions as root.
The normal Raspberry Pi installation script, also used for [some?] other Linuses sets up systemd to automatically start Node-Red at boot. ie sudo systemctl enable nodered.
Do you have these two files? If so, I suspect that also starting node-red with cron will cause problems.
ls -l /etc/systemd/system/multi-user.target.wants/nodered.service
lrwxrwxrwx 1 root root 35 Nov 15 2020 /etc/systemd/system/multi-user.target.wants/nodered.service -> /lib/systemd/system/nodered.service
ls -l /lib/systemd/system/nodered.service
-rw-r--r-- 1 root root 1282 Nov 21 16:54 /lib/systemd/system/nodered.service
(The second file listed is the systemd service file. The first one, linking to it, is created when you enable the service.)
The fact that yours was not starting when the computer reboots suggests you didn't install it correctly, or something went wrong in the installation.
