MQTT suddenly crashed

Hi all,

We have a machine protocol that connect to ActiveMq via MQTT nodes.
we are communicating via topic exchange.
We are facing 2 main issue:

  1. On suddenly disconnect from ActiveMQ node-red crashes
  2. Sometimes MQTT nodes showing connected but messages not recieving - after a long time without messages exchange.

For general info: I'm using a docker with latest image of nodered/node-red:minimal

Did someone faced this issues?

Finally i was able to reproduce the errors on terminal:*

This happens when i'm using the default TTL : 60s and not V on clean session.
"Uncaught exepction" : this current node-red using only 1 MQTT node and 1 debug node.

The first thing to do is to run node-red in a terminal and see what it shows when it crashes. Post the full output from the terminal here please, including the node-red startup.

[Edit] For question 2 install MQTT explorer and connect it to the broker, then you will see whether it is node-red that is the problem.

Have a look with MQTT explorer at your mqtt broker, it looks like it gets floods with to much messages.

A message queue server should NEVER crash! That is a severe failure of your infrastructure. MQ's should be one of the most stable services that you have since their whole purpose is to act as "glue" between systems.

However, I note that ActiveMQ is one of Apache's creations and written in Java - far from my favourite language and something that I personally advise people to avoid if at all possible.

Java has a number of ongoing structural issues that make it a rather expensive platform to maintain. For example, running a Java-based application on a device with limited memory is quite likely to result in issues. Java needs careful memory management both from a developer perspective and from an operational one.

If you are only using ActiveMQ for MQTT, I strongly advise you to move to Mosquitto instead. It is rock solid and uses minimal resources.

Otherwise, carefully check the memory utilisation of ActiveMQ.

I understood that it is node-red that is alleged to be crashing, not the broker.

Ah, my bad, I misread that.

I would still check memory utilisation though since Java is such a hog.

Hi Colin,

It happens on docker containers.
I'm trying to reproduce the issue again.
Something weird that if i use the default config of MQTT nodes, so i do'n't get the error/connections status from the node.
If i change to keep-alive to 0 and use clean session , i suddenly get the:
client error Error: connect ECONNREFUSED
and timeout.

For question 2 : Unfortunately i can't because it's connected to out source company ActiveMQ.

Hi there,

Thank you for the reply.
The ActiveMQ is used by outsource company that we are sending messages to.
We can't change to current infra.
As i see node-red is pretty stable but we have the above issues once a while.

Thx, Nono

Why would you want to do that? I don't know if that is even legal.

Just because you are connecting to an outsourced broker does not stop you using MQTT Explorer. Run it on your PC and connect to the broker, then you will see whether problem 2 is in node-red or in the publish or broker.

We need to see the full log, there should be more after that. Copy/paste the text please, not a screenshot. Use the </> button at the top of the forum entry window when pasting it in.

I will look into the MQTT Explorer.
Regards the keep alive i just wanted to let you know that it current default settings i don't recieve events messages from MQTT , only if i change i recieve MQTT events messages. (at least in the logs)
This is the logs:

2021-07-19 11:34:21	
npm ERR!     /data/.npm/_logs/2021-07-19T08_34_21_554Z-debug.log
2021-07-19 11:34:21	
npm ERR! A complete log of this run can be found in:
2021-07-19 11:34:21	
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
2021-07-19 11:34:21	
npm ERR! Failed at the node-red-docker@1.3.5 start script.
2021-07-19 11:34:21	
npm ERR! 
2021-07-19 11:34:21	
npm ERR! Exit status 1
2021-07-19 11:34:21	
npm ERR! node-red-docker@1.3.5 start: `node $NODE_OPTIONS node_modules/node-red/red.js $FLOWS "--userDir" "/data" "node-red" "flows.json"`
2021-07-19 11:34:21	
npm ERR! errno 1
2021-07-19 11:34:21	
npm ERR! code ELIFECYCLE
2021-07-19 11:34:21	
    at TCP.onStreamRead (internal/stream_base_commons.js:111:27)
2021-07-19 11:34:21	
19 Jul 08:34:21 - Error: read ECONNRESET
2021-07-19 11:34:21	
19 Jul 08:34:21 - [red] Uncaught Exception:
2021-07-19 11:33:44	
19 Jul 08:33:44 - [info] [mqtt-broker:baff5da1.7789b] Disconnected from broker: mqtt://*******
2021-07-19 11:25:35	
19 Jul 08:25:35 - [info] [mqtt-broker:baff5da1.7789b] Connected to broker: mqtt://*****
2021-07-19 11:22:50	
19 Jul 08:22:50 - [info] [mqtt-broker:baff5da1.7789b] Disconnected from broker: mqtt://******

Unfortunately that doesnt give us a lot to go on, but that is often the case with uncaught exceptions.

Just to confirm, this was with just an MQTT and inject node?

There must be some code path that can lead to the error handler being missed. if there's any more information/observations you can share about when this happens (for example, the log suggests this happens a while after a disconnect?) That would be helpful.

Did you mentioned this started happening on 1.3.5? If so, what previous version were you using?

Hi Nick,

Old version was 1.2.2 (i'm not completely sure that the error don't accour there)
The process was to shutdown the ActiveMQ server and then waiting a while , and sudden crash accours.
Like mentioned when i using the default settings i don't get the Event logs , i tried to change (now i can't reproduce it , but when i changed the keep alive to 0 suddenly dissconnections and reset and other events are being logged)

What do you mean by Event logs? Is that the question 2 in your first post?

The events of reconnections or ecconect refused and etc of MQTT