Why connection failed to broker?

General
,
1

I'm running a mosquitto broker and node-red as docker images and all was fine until this morning.
I've restarted both docker container for node-red and mosquitto, then I restarted the whole server. And finally I updated node-red to the latest docker image, but all to no avail.
I can _pub and _sub with the credentials of the node-red user. The broker seems fine, but it does not see any longer connection attempts from node-red?!
I'm really at a loss what could be the issue?

node-klomp  | > node-red-docker@2.2.2 start /usr/src/node-red
node-klomp  | > node $NODE_OPTIONS node_modules/node-red/red.js $FLOWS "--userDir" "/data"
node-klomp  | 
node-klomp  | 24 Apr 22:36:25 - [info] 
node-klomp  | 
node-klomp  | Welcome to Node-RED
node-klomp  | ===================
node-klomp  | 
node-klomp  | 24 Apr 22:36:25 - [info] Node-RED version: v2.2.2
node-klomp  | 24 Apr 22:36:25 - [info] Node.js  version: v14.18.2
node-klomp  | 24 Apr 22:36:25 - [info] Linux 4.4.0-210-generic x64 LE
node-klomp  | 24 Apr 22:36:25 - [info] Loading palette nodes
node-klomp  | 24 Apr 22:36:25 - [debug] Module: node-red-contrib-bigtimer 2.7.5 /data/node_modules/node-red-contrib-bigtimer
node-klomp  | 24 Apr 22:36:25 - [debug] Module: node-red-contrib-config 1.2.1 /data/node_modules/node-red-contrib-config
node-klomp  | 24 Apr 22:36:25 - [debug] Module: node-red-contrib-telegrambot 8.10.0 /data/node_modules/node-red-contrib-telegrambot
node-klomp  | 24 Apr 22:36:25 - [debug] Module: node-red-dashboard 3.1.1 /data/node_modules/node-red-dashboard
node-klomp  | 24 Apr 22:36:26 - [info] Dashboard version 3.1.1 started at /ui
node-klomp  | 24 Apr 22:36:26 - [info] Settings file  : /data/settings.js
node-klomp  | 24 Apr 22:36:26 - [info] Context store  : 'default' [module=memory]
node-klomp  | 24 Apr 22:36:26 - [info] User directory : /data
node-klomp  | 24 Apr 22:36:26 - [warn] Projects disabled : editorTheme.projects.enabled=false
node-klomp  | 24 Apr 22:36:26 - [info] Flows file     : /data/flows.json
node-klomp  | 24 Apr 22:36:26 - [info] Server now running at https://127.0.0.1:1880/
node-klomp  | 24 Apr 22:36:26 - [debug] loaded flow revision: 058975407b276111eac9baacefc7d7c8
node-klomp  | 24 Apr 22:36:26 - [debug] red/runtime/nodes/credentials.load : user provided key
node-klomp  | 24 Apr 22:36:26 - [debug] red/runtime/nodes/credentials.load : keyType=user
node-klomp  | 24 Apr 22:36:26 - [info] Starting flows
node-klomp  | 24 Apr 22:36:26 - [debug] red/nodes/flows.start : starting flow : global
node-klomp  | 24 Apr 22:36:26 - [debug] red/nodes/flows.start : starting flow : 2f499eea.89e0d2
node-klomp  | 24 Apr 22:36:26 - [debug] red/nodes/flows.start : starting flow : 9eac7f36.55cd1
node-klomp  | 24 Apr 22:36:26 - [debug] red/nodes/flows.start : starting flow : c666a417.dfbfd8
node-klomp  | 24 Apr 22:36:26 - [debug] red/nodes/flows.start : starting flow : de4e0039.dd2ca
node-klomp  | 24 Apr 22:36:26 - [debug] red/nodes/flows.start : starting flow : a78d9fa1.954e28
node-klomp  | 24 Apr 22:36:26 - [debug] red/nodes/flows.start : starting flow : ed26102b.57edf
node-klomp  | 24 Apr 22:36:26 - [debug] [inject:Off at 22:30] crontab = 30 22 * * *
node-klomp  | 24 Apr 22:36:26 - [debug] [inject:Toggle on 08:00] crontab = 00 08 * * *
node-klomp  | 24 Apr 22:36:26 - [debug] [inject:reset to auto] crontab = 01 00 * * *
node-klomp  | 24 Apr 22:36:26 - [debug] [inject:At 22:00] crontab = 00 22 * * *
node-klomp  | 24 Apr 22:36:26 - [debug] [inject:On at 5:20] crontab = 20 05 * * 1,2,3,4,5
node-klomp  | 24 Apr 22:36:26 - [debug] [inject:Off at 7:00] crontab = 00 07 * * 1,2,3,4,5
node-klomp  | 24 Apr 22:36:26 - [debug] [inject:reset to auto] crontab = 01 12 * * *
node-klomp  | 24 Apr 22:36:26 - [debug] [inject:On at 6:45] crontab = 45 06 * * 1,2,3,4,5
node-klomp  | 24 Apr 22:36:26 - [debug] [inject:Daily at 22:00] crontab = 00 22 * * *
node-klomp  | 24 Apr 22:36:26 - [info] Started flows
node-klomp  | 24 Apr 22:36:26 - [info] [mqtt-broker:mqtt] Connection failed to broker: node-red@mqtt://mqtt:51883
node-klomp  | 24 Apr 22:36:41 - [info] [mqtt-broker:mqtt] Connection failed to broker: node-red@mqtt://mqtt:51883
node-klomp  | 24 Apr 22:36:56 - [info] [mqtt-broker:MQTT-tls] Connection failed to broker: node-klomp@mqtts://study.klomp.ca:51883
node-klomp  | 24 Apr 22:36:56 - [info] [mqtt-broker:mqtt] Connection failed to broker: node-red@mqtt://mqtt:51883
node-klomp  | 24 Apr 22:37:11 - [info] [mqtt-broker:mqtt] Connection failed to broker: node-red@mqtt://mqtt:51883
node-klomp  | 24 Apr 22:37:26 - [info] [mqtt-broker:mqtt] Connection failed to broker: node-red@mqtt://mqtt:51883
node-klomp  | 24 Apr 22:37:41 - [info] [mqtt-broker:MQTT-tls] Connection failed to broker: node-klomp@mqtts://study.klomp.ca:51883

Mosquitto log:

1650807586: Received PUBLISH from node-klomp (d0, q0, r0, m0, 'jlight/set', ... (12 bytes))
1650807586: Sending PUBLISH to node-klomp (d0, q0, r0, m0, 'jlight/set', ... (12 bytes))
1650807586: Received PUBLISH from node-klomp (d0, q0, r0, m0, 'jlight/set', ... (12 bytes))
1650807586: Sending PUBLISH to node-klomp (d0, q0, r0, m0, 'jlight/set', ... (12 bytes))
1650807609: Client node-klomp disconnected: Protocol error.

In fact I am running two brokers, one with and one without tls. Both stopped connecting.
The non-tls is on a different host.

2

Well, having two brokers running may be a problem.

Are they both on the same port?
If so.... I'd say that is a problem.

Moving on:

On the machine running the/a broker, can you open a CLI / TERMINAL and send/receive messages?

Because until that is proven, nothing can be said.

P.S.

In/on the config nodes of the machines trying to use the MQTT:
Have you set the correct IP address for the broker?

Take a screen shot of the MQTT CONFIG node showing the broker IP address and name.
Paste it and that may go a long way to helping also.

3

Both brokers have a different hostname/IP address, thus that does not give connection issues. Like I implied it used to run fine for weeks. Just out of the blue it broke this morning.
And yes I can publish and subscribe to both brokers from the CLI.

4

mqtt-tls
mqtt-nontls
Both used to work as is shown.

5

In the second picture you have:

server then you have mqtt.

I think that need to be the IP address of the machine. Not the name of MQTT.

And you do know the ports are 1883 not 51883.
Just saying. (Well, usually. Unless you have changed the port numbers in the brokers also)

6 
Non-authoritative answer:
Name:	mqtt.klomp.ca
Address: 192.168.0.77

Non-authoritative answer:
Name:	study.klomp.ca
Address: 192.168.0.60

And yes, I deliberately changed the configured ports for both.

7

Sorry, I've been there many times when I type a number I put the wrong one in.
But as it was me who typed it I can't see the mistake.

So in that bit with the names and IP.....

You didn't cover mqtt as an IP address.

d607b38695d80fabf9e12a3a5e22f26c566a0f24

(you have LIMITED posts if you are new here so don't squander them)

Why are you using names rather than IP addresses?
And if they are 192.x.x.x..... they are LOCAL addresses and won't be going too far out of your LAN.

8

Wondering if there is at the docker level a way to trace the traffic leaving the node-red container?
(going to sleep on that for now...)

9

the default search domain is klomp.ca, thus mqtt and mqtt.klomp.ca are the same host.
l'm using names as to more easily move work loads.

I think I really need to look at the docker plumbing, to see what's going on.

10

No problems. I am at the bottom of the learning curve anyway.

But it may not hurt to do it just to make sure that the IP numbers will (or won't) make a difference.

11

Unless you've opened the port, docker won't let you communicate.

When using Docker, it is best to always assume that any problem is a Docker configuration issue. Because 99% of the time, it is. That's why a lot of us don't like using Docker for something as straight-forward as Node-RED. It adds a significant layer of complexity for very little reward or purpose for most people.

Docker is certainly useful for more complex environments that someone else has kindly packaged up :slight_smile: I use it for several things. But Node-RED is so easy to install, upgrade and manage in any case.

If you are going to use Docker, I recommend always using Docker Compose as it is an easy way to maintain your Docker configuration.

12

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.