Understanding MQTT v5 retain options

General
1

Hi, I'm trying to fully understand this two option.

  • keep retain flag of original publish
  • retained message handling
    Can someone please summarize them or share a good guide to follow? Googling I haven't find a clear one, and even making several test it's not that clear to me

image
thanks so mucj

2

Keep retain is as it describes. What you may not be aware of is that in MQTT version 3, the retain flag is cleared before sending the message. This option allows the incoming messages to KEEP the flag.

The send retained messages are also pretty much as described. The 1st option is send retained messages. The 2nd option is send retained for only new subscriptions and the 3rd option is don't send retained messages.
What may make this clearer is this is about retained messages and therefore about what happens when the client (node-red) connects or subscribes to a topic. Re-read the options with that in mind.

3

Thanks @Steve-Mcl

  • Sorry but for the flag I don't understand it. If I make a test, no matter if it's checked or not, the received message always contains the property retain = true
  • For the retain message handling, at the end these are the instruction for the broker? Or node-red anyway receives the retained messages, but based on this settings are either or not sent to the output?
4

I'll see if I can find the demo flows I had when adding MQTT v5 support.

5

Not quite true… it is cleared if the message is passed directly to a subscriber (ie not retained on the broker). This is so the subscriber can tell if the message is live or not. This flag overrides that and just passes it through always.

6

Thanks for the reminder Dave. It is amazing (and concerning) how much of the spec I have forgotten since doing the mqtt5 stuff.

@Lupin_III I'll not be up and running until Monday. I'll try to find those demo flows

7

Thanks so much to all.
@Steve-Mcl It will help a lot. I've tried some more test even for the retained message handling, but I cannot see any difference. When node-red starts, or client disconnects and reconnects, or restarting the flow, it always get the retained messages. So, maybe the only difference could be for dynamic subscription that you can manage once node-red is running. But even in that case, if for whatever reason the client get disconnected and reconnected, it always gets the retained message.
But maybe I've set the test in the wrong way.

8

Yes, it always gets the retained messages, it is the retained flag (msg.retained) that is affected. That can be used to identify whether the message is being sent because it is retained, or because a new value has just been published.

9

Thanks @Colinn. Nothing, I always get the retained flag (msg.retained) as true. Probably the way I do the test. But conceptually I think I have understood it
Yes a flow could help for sure to better understand it. Thanks so much

10

There does seem to be an issue here, or I too am misunderstanding it.
I have an Out node publishing with the Retain flag set, and two In nodes, one with Keep set and one with it not set. As I understand it, on publishing a message then the subscriber should see msg.retain set according to the Keep flag, but it is set in both cases

image
image983×275 42.6 KB 

[{"id":"310782544a09a97e","type":"mqtt out","z":"bdd7be38.d3b55","name":"","topic":"test/testretained","qos":"0","retain":"true","respTopic":"","contentType":"","userProps":"","correl":"","expiry":"","broker":"adf2dac3.b20fe8","x":520,"y":2120,"wires":[]},{"id":"53ea2ca73a91f2eb","type":"mqtt in","z":"bdd7be38.d3b55","name":"Don't keep retained flag","topic":"test/testretained","qos":"2","datatype":"auto-detect","broker":"adf2dac3.b20fe8","nl":false,"rap":false,"rh":0,"inputs":0,"x":240,"y":2180,"wires":[["9061500bbf32ed93"]]},{"id":"9061500bbf32ed93","type":"debug","z":"bdd7be38.d3b55","name":"debug 99","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":420,"y":2180,"wires":[]},{"id":"bab8ee4c7cda13cf","type":"inject","z":"bdd7be38.d3b55","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":140,"y":2120,"wires":[["310782544a09a97e"]]},{"id":"1bbd8c1f2edfbeb9","type":"mqtt in","z":"bdd7be38.d3b55","name":"Keep retained flag","topic":"test/testretained","qos":"2","datatype":"auto-detect","broker":"adf2dac3.b20fe8","nl":false,"rap":true,"rh":0,"inputs":0,"x":230,"y":2240,"wires":[["860c3b9007cf85c4"]]},{"id":"860c3b9007cf85c4","type":"debug","z":"bdd7be38.d3b55","name":"debug 100","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":420,"y":2240,"wires":[]},{"id":"adf2dac3.b20fe8","type":"mqtt-broker","name":"localhost V5","broker":"localhost","port":"1883","clientid":"","autoConnect":true,"usetls":false,"protocolVersion":"5","keepalive":"60","cleansession":true,"birthTopic":"tigger/LWT","birthQos":"0","birthPayload":"Online","birthMsg":{},"closeTopic":"tigger/LWT","closeQos":"0","closePayload":"Offline","closeMsg":{},"willTopic":"tigger.LWT","willQos":"0","willPayload":"Offline","willMsg":{},"userProps":"","sessionExpiry":""}]

I have also had it lock into the situation where neither of the received messages have that flag set, but I haven't worked out what triggers that. A node red restart fixed that.

Node red 3.0.2 with nodejs 18

11

Further to the above, I think it is having two In nodes, one keeping and one not that is causing the problem

12

@Lupin_III If you import my flow, but delete one pair of MQTT In and debug nodes, does it work for you? You should see the received Retain flag change as you change the setting in the MQTT In node.

13

Thanks @Colin Yes, it works and it is probably why I didn't see any dfference with my test.
So in order to summirize it, even for the other readers, the "Keep retain flag of original publish" works in this way, as even @dceejay said:

  • if it's NOT flagged and the message is received on the fly, then retain: false
  • if it's NOT flagged and the message is received on connection, then retain: true (doesn't matter if it was already received before the client got disconnected)
  • if it's flagged then always retain: true

Now I'm still having some doubts on the "retained message handling" option. I have tested all possible combination but I always get the message and its retain property as true, a part from the "Do not send" option, whne I don't receive the message at all, as espected.

14

This is probably a nicer explanation: MQTT 5.0 Features | Retain Message | by EMQ Technologies | Medium

15

Here is another from Steve's internet site:

Retained Message Control

Retained messages still work as in 3.1.1 but subscribe options have been added to control what the client receives.:

0 -Send retained messages when client subscribes. Same as in MQTT 3.1.1
1 -Send retained messages when client subscribes. if the subscription does not currently exist.
2 – Don’t send retained messages when client subscribes.

Key part being "when client subscribes"

So ordinarily you won't see this as node-red subscribes at point of start up. To see this in action you will need to trigger subscribe/unsubscribe actions dynamically and watch the FULL msg coming from the MQTT in node.

Sorry but I couldn't find my Dev/test flows (was over 1y since I did this work)

16

You can see it more easily by configuring the debug node to output to the console, then monitor the log and do a full deploy (or restart flows).

@Steve-Mcl did you try my test with two In nodes with different settings, and confirm that it does not work in this case if the two nodes have the flag configured differently?

17

Those two sentences appear to be in contradiction with each other.

18

Not yet. Not had the time Colin.

19

I suspect the two subscriptions, with and without the flag, are merged into one subscription to the broker, instead of being kept separate.

20

That is a very strong possibility. And the first place I will look.

Would you be kind enough to raise an issue with your demo flow (or a link back) please?