# MQTT Nodes Connection Behaviour

**URL:** <https://discourse.nodered.org/t/mqtt-nodes-connection-behaviour/45954>\
**Category:** General\
**Created:** [20 May 2021 01:48 UTC](https://discourse.nodered.org/t/mqtt-nodes-connection-behaviour/45954 "2021-05-20T01:48:21Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![Siothrun](https://avatars.discourse-cdn.com/v4/letter/s/aeb1de/32.png) [@Siothrun](https://discourse.nodered.org/u/Siothrun)\
**Post date:** [20 May 2021 01:48 UTC](https://discourse.nodered.org/t/mqtt-nodes-connection-behaviour/45954/1 "2021-05-20T01:48:21Z")

</div>

Dear Forum

I wanted to post my observations of the MQTT nodes connection behaviour.

1. There is an approximately 20 minute delay from when internet access is lost (SIM card removed) to when the MQTT nodes show disconnected.
2. There is no data sent via MQTT out nodes for approximately 20 minutes if internet access is lost (SIM card removed) even if internet access is restored (SIM card replaced) almost immediately.  
NB: The modem is being restarted immediately after the SIM card is replaced, but this is with a new IP address as is the nature of mobile/cellular connections.
3. MQTT out nodes backfill is only for this approximately 20 minute period from when internet access is lost (SIM Card Removed) to when the MQTT nodes show disconnected. If the internet access outage is longer than 20 minutes then data sent after this time is lost.  
NB: I have implemented a contributed Node-RED node (node-red-contrib-msg-queue), that queues undeliverable messages to file for later delivery to avoid this data being lost.

This has raised the following questions for me.

1. Are others observing the same connection behaviour from their MQTT nodes?  
**2. When internet access is lost and then restored is it possible to force the MQTT nodes to disconnect and reconnect? If so, can someone help give me some guidance in doing this?**

Configuration details...  
Hardware: MTX-GTW-II  
OS: Linux, ARM, 4.1.15-1.2.0+gda74a32  
Node-RED 1.2.9  
Node.js 10.24.0  
A SIM card is used for internet access from the device.

Test Case details...  
14/05/2021  
Test 1: Internet disconnection for ~31 minutes.  
This test uses keep alive time: 120s, clean session and auto generated client ID.  
10:43:30 SIM Card Removed & MQTT Shows Connected  
11:01:45 MQTT Shows Disconnected  
11:02:00 MQTT Shows Connecting  
11:12:00 SIM Card Replaced  
11:14:51 Connection Restored  
11:15:40 MQTT Shows Connected

Test 2: Internet disconnection for ~28 minutes.  
This test uses keep alive time: 30s, no clean session and client ID:NR\_Hanmer.  
13:32:15 SIM Card Removed & MQTT Shows Connected  
13:52:23 MQTT Shows Disconnected  
13:52:37 MQTT Shows Connecting  
13:58:34 SIM Card Replaced  
14:00:10 Connection Restored  
14:01:26 MQTT Shows Connected

Test 3: Internet disconnection for ~7 minutes.  
This test uses keep alive time: 30s, no clean session and client ID:NR\_Hanmer.  
14:09:33 SIM Card Removed & MQTT Shows Connected  
14:10:00 SIM Card Replaced  
14:16:13 Connection Restored  
14:27:59 MQTT Shows Disconnected  
14:28:10 MQTT Shows Connecting  
14:29:17 MQTT Shows Connected

Any assistance users can provide in answering my questions is appreciated.

Regards  
Siothrun

---

<div class="post-metadata">

**Author:** ![Colin](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/colin/32/17040_2.png) [@Colin](https://discourse.nodered.org/u/Colin)\
**Post date:** [20 May 2021 06:34 UTC](https://discourse.nodered.org/t/mqtt-nodes-connection-behaviour/45954/2 "2021-05-20T06:34:37Z")

</div>

Welcome to the forum @Siothrun

There have been some significant updates to the mqtt nodes since the old version of node red you are using. If possible it would be a good idea to upgrade node red and see if that improves it.

---

<div class="post-metadata">

**Author:** ![Steve-Mcl](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/steve-mcl/32/4826_2.png) [@Steve-Mcl](https://discourse.nodered.org/u/Steve-Mcl)\
**Post date:** [20 May 2021 06:53 UTC](https://discourse.nodered.org/t/mqtt-nodes-connection-behaviour/45954/3 "2021-05-20T06:53:46Z")

</div>

> [@Siothrun](#):
>
> Are others observing the same connection behaviour from their MQTT nodes

No, it should only take around 15 secs (default settings.js setting is 15000)

```auto
     // Retry time in milliseconds for MQTT connections
    mqttReconnectTime: 15000,

```

  

> [@Siothrun](#):
>
> When internet access is lost and then restored is it possible to force the MQTT nodes to disconnect and reconnect

Not yet. there is an open PR for a new node called `mqtt-control` that permits this.  
It is still to be reviewed and accepted but I am hopeful it will see its way into the next major release.

  

in this period of outage, have you tried ...

- forcing a publish (use an inject node)
- connecting via a different broker (e.g. mqtt explorer) to verify the broker is actually reachable?

---

<div class="post-metadata">

**Author:** ![Colin](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/colin/32/17040_2.png) [@Colin](https://discourse.nodered.org/u/Colin)\
**Post date:** [20 May 2021 10:38 UTC](https://discourse.nodered.org/t/mqtt-nodes-connection-behaviour/45954/4 "2021-05-20T10:38:53Z")

</div>

> [@Steve-Mcl](#):
>
> it should only take around 15 secs (default settings.js setting is 15000

I think that is a different issue to the one here where it is apparently taking 20 minutes for the MQTT node to report that the network is disconnected. I believe that is set by the Keep Alive setting in the MQTT connection node.  
@Siothrun what have you got Keep Alive set to in the node?

---

<div class="post-metadata">

**Author:** ![Siothrun](https://avatars.discourse-cdn.com/v4/letter/s/aeb1de/32.png) [@Siothrun](https://discourse.nodered.org/u/Siothrun)\
**Post date:** [20 May 2021 13:18 UTC](https://discourse.nodered.org/t/mqtt-nodes-connection-behaviour/45954/5 "2021-05-20T13:18:45Z")

</div>

Thanks for the replies @Colin & @Steve-Mcl!

> [@Colin](#):
>
> If possible it would be a good idea to upgrade node red and see if that improves it.

Thanks for this feedback I'll certainly look to take this action.

> [@Steve-Mcl](#):
>
> Not yet. there is an open PR for a new node called `mqtt-control` that permits this.  
> It is still to be reviewed and accepted but I am hopeful it will see its way into the next major release.

Thanks for the confirmation that this control is not yet possible and also the hope that it will soon be available via an 'mqtt-control' node.

> [@Steve-Mcl](#):
>
> in this period of outage, have you tried ...
> 
> - forcing a publish (use an inject node)
> - connecting via a different broker (e.g. mqtt explorer) to verify the broker is actually reachable?

In my testing a publish was being triggered every 10 seconds.  
I'm not sure I understand the next test you mention trying. I have been connecting to a Mosquitto broker, but I'd assumed the MQTT client connection behaviour I am seeing would be broker independent. I will though have a separate client communicate with the same broker to rule out potential broker issues with reconnecting.

> [@Colin](#):
>
> @Siothrun what have you got Keep Alive set to in the node?

I've tested the connection behaviour with keep alive times of 120s and 30s. This setting had no impact on the approximately 20 minute delay.  
As I understand it the keep alive time is registered with the broker by the client when it establishes the connection. When the broker has not heard from the client for this period (plus some extra time) it will then issue the Last Will and Testament (LWT) message. This worked as expected when I tested it, except for the fact that there is an undefined amount of extra time that the broker waits.

---

<div class="post-metadata">

**Author:** ![Colin](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/colin/32/17040_2.png) [@Colin](https://discourse.nodered.org/u/Colin)\
**Post date:** [20 May 2021 14:06 UTC](https://discourse.nodered.org/t/mqtt-nodes-connection-behaviour/45954/6 "2021-05-20T14:06:49Z")

</div>

> [@Siothrun](#):
>
> When the broker has not heard from the client for this period (plus some extra time) it will then issue the Last Will and Testament (LWT) message.

That is what happens at the broker, but it is the client (which is node-red) that initiates the regular keep alive ping. If the node red node does not get the ping reply then it should show disconnected so that mechanism is not working as it should. I think you need to upgrade node-red so that at least you can rule out that possibility.

---

<div class="post-metadata">

**Author:** ![Siothrun](https://avatars.discourse-cdn.com/v4/letter/s/aeb1de/32.png) [@Siothrun](https://discourse.nodered.org/u/Siothrun)\
**Post date:** [21 May 2021 00:15 UTC](https://discourse.nodered.org/t/mqtt-nodes-connection-behaviour/45954/7 "2021-05-21T00:15:00Z")

</div>

> [@Colin](#):
>
> That is what happens at the broker, but it is the client (which is node-red) that initiates the regular keep alive ping. If the node red node does not get the ping reply then it should show disconnected so that mechanism is not working as it should. I think you need to upgrade node-red so that at least you can rule out that possibility.

Thanks for that explanation Colin. I didn't realise that there was a keep alive ping from the client node. I'd incorrectly assumed there was a requirement to send messages within this keep alive time to avoid the issue of the LWT message. I will take your advice and look to upgrade Node-RED and Node.js to see if this changes the connection behaviour I am seeing.

---

<div class="post-metadata">

**Author:** ![Siothrun](https://avatars.discourse-cdn.com/v4/letter/s/aeb1de/32.png) [@Siothrun](https://discourse.nodered.org/u/Siothrun)\
**Post date:** [24 May 2021 06:04 UTC](https://discourse.nodered.org/t/mqtt-nodes-connection-behaviour/45954/8 "2021-05-24T06:04:37Z")

</div>

My further testing has raised the following question...  
**Should the Keep Alive time be less than the shortest time between messages?**

The results of my further testing...

**Upgraded**  
I upgraded as recommended...  
Node-Red 1.3.5  
Node.js 10.24.1

Unfortunately this didn't change the MQTT nodes connection behaviour.

**Different Broker**  
I connected using a different broker...  
_Mosquitto_ using TLS on port 8883 (Existing Broker)  
_HiveMQ_ on port 1883 (Different Broker)

They both had a Keep Alive Time of 30s and didn't use a clean session. The message type and frequency being sent to each of these brokers was different. The internet connection was interrupted for a couple of minutes.  
The _Mosquitto_ broker connected MQTT node continued to show connected for approximately 20 minutes without sending data before finally reconnecting and resumed sending data.  
The _HiveMQ_ broker connected MQTT node almost immediately reconnected and resumed sending data.

This difference prompted me to look closer at the _Mosquitto_ broker.

**PINGREQ & PINGRESP**  
The Mosquitto broker logs showed "Received PINGREQ" from a different client and the broker "Sending PINGRESP". The broker logs showed no "Received PINGREQ" from my client!

[The Keep Alive timer, measured in seconds, defines the maximum time interval between  
messages received from a client. It enables the server to detect that the network  
connection to a client has dropped, without having to wait for the long TCP/IP timeout.  
The client has a responsibility to send a message within each Keep Alive time period. In  
the absence of a data-related message during the time period, the client sends a  
PINGREQ message, which the server acknowledges with a PINGRESP message.](http://public.dhe.ibm.com/software/dw/webservices/ws-mqtt/MQTT_V3.1_Protocol_Specific.pdf)

My client is sending a message (watchdog pulse, QoS 2) every 10 seconds and I am also setup to receive the message being sent so I can display it in a debug node. As a result I do not send a PINGREQ message. So I was curious if the behaviour would change if I selected a Keep Alive time that would force my client to send a PINGREQ. Selecting a Keep Alive time of 9 seconds and monitoring the broker logs showed "Received PINGREQ" from my client. Interrupting the internet connection for a couple of minutes now has delivered me mixed results. A number of times it has disconnected quickly and then reconnected once the connection has been re-established. However, I've also experienced the ~20 minute disconnection delay again, but I'm wondering if this now the exception and not the rule.

**Disconnection Delay**  
I suspect now that the ~20 minute disconnection delay is due to the TCP/IP timeout, which is dictated by the following file on Linux.

> /proc/sys/net/ipv4/tcp\_retries2

The default value is 15, which corresponds to a duration of approximately between 13 to 30 minutes.  
A hypothetical timeout of 924.6 seconds is a lower bound for the effective timeout.

---

<div class="post-metadata">

**Author:** ![dceejay](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/dceejay/32/38_2.png) [@dceejay](https://discourse.nodered.org/u/dceejay)\
**Post date:** [24 May 2021 07:43 UTC](https://discourse.nodered.org/t/mqtt-nodes-connection-behaviour/45954/9 "2021-05-24T07:43:45Z")

</div>

As I understand it - I thought that any received message (at the broker) would act the same as a ping message and indicate that the client was still alive - and likewise at the client end there was no need to send a keepalive message until the set time after the last transmitted real message. I.E. you only needed to send keepalives if you weren't sending data for a period of time greater than the keepalive timeout.

At least that is how I thought it was supposed to work...

---

<div class="post-metadata">

**Author:** ![Siothrun](https://avatars.discourse-cdn.com/v4/letter/s/aeb1de/32.png) [@Siothrun](https://discourse.nodered.org/u/Siothrun)\
**Post date:** [25 May 2021 04:58 UTC](https://discourse.nodered.org/t/mqtt-nodes-connection-behaviour/45954/10 "2021-05-25T04:58:22Z")

</div>

> [@dceejay](#):
>
> As I understand it - I thought that any received message (at the broker) would act the same as a ping message and indicate that the client was still alive - and likewise at the client end there was no need to send a keepalive message until the set time after the last transmitted real message. I.E. you only needed to send keepalives if you weren't sending data for a period of time greater than the keepalive timeout.

Based on my testing I think this is how it does work. The problem is when the internet connection is interrupted AND the MQTT node continues receiving messages more frequently than the Keep Alive time to transmit. It therefore has no need to send the PINGREQ. If it did send the PINGREQ it would time out waiting for the PINGRESP and disconnect.

What I can't reconcile now though is that I get this behaviour from the Mosquitto broker connection, but not from the HiveMQ broker connection.

---

<div class="post-metadata">

**Author:** ![Colin](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/colin/32/17040_2.png) [@Colin](https://discourse.nodered.org/u/Colin)\
**Post date:** [25 May 2021 06:28 UTC](https://discourse.nodered.org/t/mqtt-nodes-connection-behaviour/45954/11 "2021-05-25T06:28:41Z")

</div>

> [@Siothrun](#):
>
> The problem is when the internet connection is interrupted AND the MQTT node continues receiving messages

How can it continue receiving messages if the connection is interrupted?

---

<div class="post-metadata">

**Author:** ![Siothrun](https://avatars.discourse-cdn.com/v4/letter/s/aeb1de/32.png) [@Siothrun](https://discourse.nodered.org/u/Siothrun)\
**Post date:** [25 May 2021 07:21 UTC](https://discourse.nodered.org/t/mqtt-nodes-connection-behaviour/45954/12 "2021-05-25T07:21:16Z")

</div>

You only quoted half of my sentence! 🙂  
The problem is when the internet connection is interrupted AND the MQTT node continues receiving messages more frequently than the Keep Alive time **to transmit**.

I may not be using the best terminology here and I'm happy to be corrected, but worded differently...  
The MQTT out node is not sending messages, but it is receiving messages from the flow to transmit to the broker.  
The MQTT in node is not receiving messages.

---

<div class="post-metadata">

**Author:** ![zenofmud](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/zenofmud/32/316_2.png) [@zenofmud](https://discourse.nodered.org/u/zenofmud)\
**Post date:** [25 May 2021 08:14 UTC](https://discourse.nodered.org/t/mqtt-nodes-connection-behaviour/45954/13 "2021-05-25T08:14:20Z")

</div>

@colin here is a case: you have a broker on the same device as NR. NR sends/receives msgs to other devices on the network AND sends/receives msgs to other tabs/locations within the flow.

So you could loose network connection but the mqtt msgs between the tabs/locations in the flow would(should) still work (I think 🤔)

---

<div class="post-metadata">

**Author:** ![Colin](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/colin/32/17040_2.png) [@Colin](https://discourse.nodered.org/u/Colin)\
**Post date:** [25 May 2021 20:47 UTC](https://discourse.nodered.org/t/mqtt-nodes-connection-behaviour/45954/14 "2021-05-25T20:47:33Z")

</div>

> [@zenofmud](#):
>
> you have a broker on the same device as NR

I don't think that is the situation here. @Siothrun can you confirm that node red is running on one device and the broker is across the internet?

---

<div class="post-metadata">

**Author:** ![Colin](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/colin/32/17040_2.png) [@Colin](https://discourse.nodered.org/u/Colin)\
**Post date:** [25 May 2021 20:50 UTC](https://discourse.nodered.org/t/mqtt-nodes-connection-behaviour/45954/15 "2021-05-25T20:50:38Z")

</div>

> [@Siothrun](#):
>
> The MQTT out node is not sending messages, but it is receiving messages from the flow to transmit to the broker.

Messages passed to the mqtt node from the flow are of no consequence in whether ping requests will be sent. It is purely whether data is received from the broker.

What QoS have you specified in the MQTT nodes? If you are using zero does it make a difference if you change it to 1?

---

<div class="post-metadata">

**Author:** ![Siothrun](https://avatars.discourse-cdn.com/v4/letter/s/aeb1de/32.png) [@Siothrun](https://discourse.nodered.org/u/Siothrun)\
**Post date:** [25 May 2021 21:05 UTC](https://discourse.nodered.org/t/mqtt-nodes-connection-behaviour/45954/16 "2021-05-25T21:05:29Z")

</div>

> [@Colin](#):
>
> @Siothrun can you confirm that node red is running on one device and the broker is across the internet?

I can confirm that Node-RED and the MQTT broker are on different devices.

MTX-GTW II AUS is running Node-RED.  
Windows VM is running the Mosquitto MQTT broker.

---

<div class="post-metadata">

**Author:** ![Siothrun](https://avatars.discourse-cdn.com/v4/letter/s/aeb1de/32.png) [@Siothrun](https://discourse.nodered.org/u/Siothrun)\
**Post date:** [25 May 2021 21:21 UTC](https://discourse.nodered.org/t/mqtt-nodes-connection-behaviour/45954/17 "2021-05-25T21:21:38Z")

</div>

> [@Colin](#):
>
> Messages passed to the mqtt node from the flow are of no consequence in whether ping requests will be sent. It is purely whether data is received from the broker.

The MQTT in node is also receiving messages from the broker every 10 seconds (when the internet connection is active). So if this node stops receiving messages for longer than the Keep Alive time when the internet connection is interrupted it should send a PINGREQ, but then fail to receive a PINGRESP. When it fails to receive the PINGRESP within 1.5 x Keep Alive time it should disconnect? I’m certainly not seeing this behaviour.

> [@Colin](#):
>
> What QoS have you specified in the MQTT nodes? If you are using zero does it make a difference if you change it to 1?

MQTT in nodes have QoS 0.  
MQTT out nodes have QoS 2.  
I can experiment with using a different QoS, but ultimately my application requires these QoS settings.

---

<div class="post-metadata">

**Author:** ![Colin](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/colin/32/17040_2.png) [@Colin](https://discourse.nodered.org/u/Colin)\
**Post date:** [26 May 2021 12:59 UTC](https://discourse.nodered.org/t/mqtt-nodes-connection-behaviour/45954/18 "2021-05-26T12:59:42Z")

</div>

Why does it require QoS 0?

Have you upgraded node red yet?

---

<div class="post-metadata">

**Author:** ![Siothrun](https://avatars.discourse-cdn.com/v4/letter/s/aeb1de/32.png) [@Siothrun](https://discourse.nodered.org/u/Siothrun)\
**Post date:** [26 May 2021 21:12 UTC](https://discourse.nodered.org/t/mqtt-nodes-connection-behaviour/45954/19 "2021-05-26T21:12:05Z")

</div>

> [@Siothrun](#):
>
> **Upgraded**  
> I upgraded as recommended...  
> Node-Red 1.3.5  
> Node.js 10.24.1

Node-RED in my application is being used in a control application to interface a cloud HMI to a PLC.

When reading values from the PLC to display on the HMI QoS 2 is desired. The values read are time stamped and the history of what is happening is important. This is why the MQTT backfill is relevant to me.

When writing values from the HMI to the PLC QoS 0 is required. If the internet connection is interrupted I don’t want a value/command being buffered and written to the PLC at a later date.

---

<div class="post-metadata">

**Author:** ![Colin](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/colin/32/17040_2.png) [@Colin](https://discourse.nodered.org/u/Colin)\
**Post date:** [27 May 2021 08:31 UTC](https://discourse.nodered.org/t/mqtt-nodes-connection-behaviour/45954/20 "2021-05-27T08:31:24Z")

</div>

> [@Siothrun](#):
>
> When writing values from the HMI to the PLC QoS 0 is required. If the internet connection is interrupted I don’t want a value/command being buffered and written to the PLC at a later date.

I suspect that using QoS 0 does not absolutely guarantee that the value will not be buffered in the mqtt driver and sent later. If it is essential that you do not use old data then I suggest adding a timestamp and check at the receiving end that it is not stale.

> [@Siothrun](#):
>
> When reading values from the PLC to display on the HMI QoS 2 is desired. The values read are time stamped and the history of what is happening is important. This is why the MQTT backfill is relevant to me.

I suggest writing the receiving end so that it ignores repeated messages. That will make your life much simpler. Then, if what you are after is guaranteeing that you do not lose any messages you can use something like this flow to send the data. The flow can even buffer the data over a node-red restart if that is required.

> [@Node-red-contrib-msg-queue - Only storing not forwarding w/ active connection](https://discourse.nodered.org/t/node-red-contrib-msg-queue-only-storing-not-forwarding-w-active-connection/44274/11):
>
> Give this a go. It uses the Guaranteed Delivery subflow from [here](https://flows.nodered.org/flow/05e6d61f14ef6af763ec4cfd1049ab61) with the logic to make it work with MQTT. Configure the MQTT node as required and send the messages to be sent in via the top left link node. It uses a Complete node attached to the MQTT node to know when the message has been sent successfully but there is no way to know when it has failed, so it uses a 5 second timeout to indicate that. When it fails it retries every 60 seconds. That can be configured in the subflow settings…

[Next page](https://discourse.nodered.org/t/mqtt-nodes-connection-behaviour/45954.md?page=2)