Node-red dropping connection to external services extremely often

Suddenly Node-Red is dropping connections to MQTT brokers and an IMAP server extremely often, like once every 10 seconds or so. It's been rock solid for a couple of years prior to this.
I thought it must be the internet connection so I ran a ping to 8.8.8.8 and observed what Node-Red was doing. Node-Red lost connections while the ping continued to return just fine.

Any ideas how I can troubleshoot?

Thank you!

Did you make any changes/updates lately?
What platform(s) are node-red and your mqtt broker running on?
What version of NR are you running?

Also stop node-red and start it again in a command window and see if there are any messages shown. Post the log here if necessary. Copy/paste please, not screenshot.

Interesting. I ran a ping to the host 192.168.194.22 that kept disconnecting and there are some long pings - wonder if the MQTT server just isn't able to maintain a reliable connection?

===================

2 Oct 11:36:21 - [info] Node-RED version: v1.0.2
2 Oct 11:36:21 - [info] Node.js  version: v8.16.0
2 Oct 11:36:21 - [info] Linux 3.10.0-1127.19.1.el7.x86_64 x64 LE
2 Oct 11:36:22 - [info] Loading palette nodes
2 Oct 11:36:28 - [info] Dashboard version 2.22.1 started at /ui
2 Oct 11:36:29 - [info] Settings file  : /root/.node-red/settings.js
2 Oct 11:36:29 - [info] Context store  : 'default' [module=memory]
2 Oct 11:36:29 - [info] User directory : /root/.node-red
2 Oct 11:36:29 - [warn] Projects disabled : editorTheme.projects.enabled=false
2 Oct 11:36:29 - [info] Flows file     : /root/.node-red/flows_freepbx.sangoma.local.json
2 Oct 11:36:29 - [info] Server now running at http://127.0.0.1:1880/
2 Oct 11:36:29 - [warn]

---------------------------------------------------------------------
Your flow credentials file is encrypted using a system-generated key.

If the system-generated key is lost for any reason, your credentials
file will not be recoverable, you will have to delete it and re-enter
your credentials.

You should set your own key using the 'credentialSecret' option in
your settings file. Node-RED will then re-encrypt your credentials
file using your chosen key the next time you deploy a change.
---------------------------------------------------------------------

2 Oct 11:36:29 - [info] Starting flows
2 Oct 11:36:29 - [info] Started flows
2 Oct 11:36:30 - [info] [mqtt-broker:hass] Connected to broker: node-red@mqtt://192.168.2.30:1883
2 Oct 11:36:30 - [info] [mqtt-broker:homehass] Connected to broker: noderedmacmini@mqtt://192.168.194.22:1883
2 Oct 11:36:45 - [info] [mqtt-broker:homehass] Disconnected from broker: noderedmacmini@mqtt://192.168.194.22:1883
2 Oct 11:37:00 - [info] [mqtt-broker:homehass] Connected to broker: noderedmacmini@mqtt://192.168.194.22:1883
2 Oct 11:37:15 - [info] [mqtt-broker:homehass] Disconnected from broker: noderedmacmini@mqtt://192.168.194.22:1883
2 Oct 11:37:30 - [info] [mqtt-broker:homehass] Connected to broker: noderedmacmini@mqtt://192.168.194.22:1883
2 Oct 11:37:42 - [info] Stopping flows
2 Oct 11:37:42 - [info] Stopped flows
64 bytes from 192.168.194.22: icmp_seq=10 ttl=64 time=235 ms
64 bytes from 192.168.194.22: icmp_seq=11 ttl=64 time=163 ms
64 bytes from 192.168.194.22: icmp_seq=12 ttl=64 time=78.5 ms
64 bytes from 192.168.194.22: icmp_seq=13 ttl=64 time=101 ms
64 bytes from 192.168.194.22: icmp_seq=14 ttl=64 time=100 ms
64 bytes from 192.168.194.22: icmp_seq=15 ttl=64 time=53.6 ms
64 bytes from 192.168.194.22: icmp_seq=16 ttl=64 time=50.4 ms
64 bytes from 192.168.194.22: icmp_seq=17 ttl=64 time=201 ms
64 bytes from 192.168.194.22: icmp_seq=18 ttl=64 time=131 ms
64 bytes from 192.168.194.22: icmp_seq=19 ttl=64 time=150 ms
64 bytes from 192.168.194.22: icmp_seq=20 ttl=64 time=180 ms
64 bytes from 192.168.194.22: icmp_seq=21 ttl=64 time=115 ms
64 bytes from 192.168.194.22: icmp_seq=22 ttl=64 time=87.4 ms
64 bytes from 192.168.194.22: icmp_seq=23 ttl=64 time=128 ms
64 bytes from 192.168.194.22: icmp_seq=24 ttl=64 time=164 ms
64 bytes from 192.168.194.22: icmp_seq=25 ttl=64 time=206 ms
64 bytes from 192.168.194.22: icmp_seq=26 ttl=64 time=65.0 ms
64 bytes from 192.168.194.22: icmp_seq=27 ttl=64 time=158 ms
64 bytes from 192.168.194.22: icmp_seq=29 ttl=64 time=78.1 ms
64 bytes from 192.168.194.22: icmp_seq=30 ttl=64 time=111 ms
64 bytes from 192.168.194.22: icmp_seq=31 ttl=64 time=61.2 ms
64 bytes from 192.168.194.22: icmp_seq=32 ttl=64 time=103 ms
64 bytes from 192.168.194.22: icmp_seq=33 ttl=64 time=74.9 ms
64 bytes from 192.168.194.22: icmp_seq=34 ttl=64 time=111 ms
64 bytes from 192.168.194.22: icmp_seq=35 ttl=64 time=81.1 ms
64 bytes from 192.168.194.22: icmp_seq=36 ttl=64 time=50.9 ms
64 bytes from 192.168.194.22: icmp_seq=37 ttl=64 time=50.6 ms
64 bytes from 192.168.194.22: icmp_seq=39 ttl=64 time=55.5 ms
64 bytes from 192.168.194.22: icmp_seq=40 ttl=64 time=56.3 ms
64 bytes from 192.168.194.22: icmp_seq=41 ttl=64 time=53.6 ms
64 bytes from 192.168.194.22: icmp_seq=42 ttl=64 time=66.9 ms
64 bytes from 192.168.194.22: icmp_seq=43 ttl=64 time=96.4 ms
64 bytes from 192.168.194.22: icmp_seq=44 ttl=64 time=153 ms
64 bytes from 192.168.194.22: icmp_seq=45 ttl=64 time=76.4 ms
64 bytes from 192.168.194.22: icmp_seq=46 ttl=64 time=55.7 ms
64 bytes from 192.168.194.22: icmp_seq=47 ttl=64 time=149 ms
64 bytes from 192.168.194.22: icmp_seq=48 ttl=64 time=167 ms
64 bytes from 192.168.194.22: icmp_seq=49 ttl=64 time=204 ms
64 bytes from 192.168.194.22: icmp_seq=50 ttl=64 time=60.1 ms
64 bytes from 192.168.194.22: icmp_seq=51 ttl=64 time=94.6 ms
64 bytes from 192.168.194.22: icmp_seq=52 ttl=64 time=174 ms
64 bytes from 192.168.194.22: icmp_seq=53 ttl=64 time=109 ms
64 bytes from 192.168.194.22: icmp_seq=54 ttl=64 time=138 ms
64 bytes from 192.168.194.22: icmp_seq=55 ttl=64 time=115 ms
64 bytes from 192.168.194.22: icmp_seq=56 ttl=64 time=50.7 ms
64 bytes from 192.168.194.22: icmp_seq=57 ttl=64 time=61.0 ms
64 bytes from 192.168.194.22: icmp_seq=58 ttl=64 time=58.4 ms
64 bytes from 192.168.194.22: icmp_seq=59 ttl=64 time=50.9 ms
64 bytes from 192.168.194.22: icmp_seq=60 ttl=64 time=144 ms
64 bytes from 192.168.194.22: icmp_seq=61 ttl=64 time=56.0 ms
64 bytes from 192.168.194.22: icmp_seq=62 ttl=64 time=82.0 ms
64 bytes from 192.168.194.22: icmp_seq=63 ttl=64 time=101 ms
64 bytes from 192.168.194.22: icmp_seq=64 ttl=64 time=59.4 ms
64 bytes from 192.168.194.22: icmp_seq=65 ttl=64 time=67.9 ms
64 bytes from 192.168.194.22: icmp_seq=66 ttl=64 time=111 ms
64 bytes from 192.168.194.22: icmp_seq=67 ttl=64 time=133 ms
64 bytes from 192.168.194.22: icmp_seq=68 ttl=64 time=57.9 ms
64 bytes from 192.168.194.22: icmp_seq=69 ttl=64 time=80.9 ms
64 bytes from 192.168.194.22: icmp_seq=70 ttl=64 time=61.3 ms
64 bytes from 192.168.194.22: icmp_seq=71 ttl=64 time=95.2 ms
64 bytes from 192.168.194.22: icmp_seq=72 ttl=64 time=50.6 ms
64 bytes from 192.168.194.22: icmp_seq=73 ttl=64 time=57.9 ms
64 bytes from 192.168.194.22: icmp_seq=74 ttl=64 time=127 ms
64 bytes from 192.168.194.22: icmp_seq=75 ttl=64 time=208 ms
64 bytes from 192.168.194.22: icmp_seq=76 ttl=64 time=76.4 ms
64 bytes from 192.168.194.22: icmp_seq=77 ttl=64 time=75.7 ms
64 bytes from 192.168.194.22: icmp_seq=78 ttl=64 time=108 ms
64 bytes from 192.168.194.22: icmp_seq=79 ttl=64 time=49.8 ms
64 bytes from 192.168.194.22: icmp_seq=80 ttl=64 time=51.0 ms
64 bytes from 192.168.194.22: icmp_seq=81 ttl=64 time=145 ms
64 bytes from 192.168.194.22: icmp_seq=82 ttl=64 time=55.0 ms
64 bytes from 192.168.194.22: icmp_seq=83 ttl=64 time=49.5 ms
64 bytes from 192.168.194.22: icmp_seq=84 ttl=64 time=54.1 ms
64 bytes from 192.168.194.22: icmp_seq=85 ttl=64 time=54.2 ms
64 bytes from 192.168.194.22: icmp_seq=86 ttl=64 time=56.9 ms
64 bytes from 192.168.194.22: icmp_seq=87 ttl=64 time=158 ms
64 bytes from 192.168.194.22: icmp_seq=88 ttl=64 time=208 ms
64 bytes from 192.168.194.22: icmp_seq=89 ttl=64 time=122 ms
64 bytes from 192.168.194.22: icmp_seq=90 ttl=64 time=177 ms
64 bytes from 192.168.194.22: icmp_seq=91 ttl=64 time=149 ms
64 bytes from 192.168.194.22: icmp_seq=92 ttl=64 time=70.9 ms

Haven't made any changes recently
Running Node-Red on a FreePBX CentOS server.
Running NR 1.0.2

Have a look at the mqtt server log on the server. If it is mosquitto it may be /var/log/mosquitto/mosquitto.log

When you say you have not made any changes do you mean literally that? No changes to flows or anything? That sort of regular connect/disconnect can happen if you have multiple connections with the same client id.

Does it make any difference if you disable the mqtt broker on 192.168.2.30?

Is your remote MQTT server running Mosquitto or something else? Is it running on bare metal or in a container or VM?

For Mosquitto, you may need to turn up the logging on the remote MQTT server.

Also check for errors and performance issues on the remote MQTT server.

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.