Node-RED Mqtt 5 support

General
1

I have an existing project using Mqtt 5 (user properties, etc) and would like to tie in Node Red. However, without Mqtt 5 support, I'm not able to do this. Any recommendations?

2

Help migrate the existing node ?

4 Likes
3

Thanks, and that's something I'd definitely consider. I've searched for information regarding MQTT 5 support, but came up empty. While not new to Node-RED, I am new to the community, so I thought I'd start by checking to make sure I wasn't missing a major roadblock or reinventing the wheel.

4

MQTT.js seems to be a popular node.js library but it's v5 feature is claimed as experimental though it seems to have a number of properties dedicated to it.

Mosquitto also seems to support it as a broker.

I guess a good starting point is to think about what is different in v5? Does it need different settings? Or is it transparent?

What would be needed in the NR client? Just another check-box like for 3.1.1?

5

Is there any update to support MQTT 5.0 in NodeRed?

6

The 1.2 release updates the underlying MQTT library to the latest version, which brings v5 support. However it has not yet been exposed in the nodes. One step closer, but not there yet.

3 Likes
7

thx a lot.
Do you know any upcoming details in NodeRed core to expose this settings to the node?

8

It's something we'd like to do, but no-one is actively working on it. In absence of anyone volunteering to work on it directly, it's one item on a long list of items waiting to be prioritised.

9

I am personally very interested in the user properties aspect of MQTT 5 - I think it could be a quick win.

FWIW, it (user properties) works with the updated lib now included in node-red beta.

I did a quick test with beta.1. I overrode protocolVersion setting it to 5 & added a properties.userProperties object to the options object (in the publish call) - they return in the packet :+1:

image
image987Ă—461 44.6 KB

While this simple test proves V5 capabilities are now possible, the UI implementation would need to be understood. Is this the time and place thread to discuss this?

My questions on this would be ...

  • Would it be ok to cherry pick V5 features or must we have a road map before beginning any work?
    • I personally would like to see user properties sooner rather than later - so long as the implementation slots in nicely with design considerations for the other V5 features.
  • Is it enough that to set userProperties in the msg object or should we use a typedInput to permit the user to map any msg property to userProperties
  • is it right to return the user properties in msg.userProperties considering user properties are just one part of the MQTT V5 packet.properties object
  • How would we permit V5 type connection
    • Perhaps change the [x] Use legacy MQTT 3.1 support check box to a drop down with the allowable protocol versions 3, 3.1.1, 5 (defaulting to V3.1.1) ?
2 Likes
10

All good questions that go to answer your first point - no, we can't just cherry pick features without considering the wider requirements of what v5 brings and how those features can be exposed.

If you're asking me to answer these questions here and now, then you're effectively asking me to jump this item to the top of the priority list and spend time doing the design work.

It would be great to see a proposal for v5 that has already taken these things into consideration and offers up a complete design.

5 Likes
11

Is this the correct place to continue the discussion of the proposal then?

It would be a shame not to leverage Steve's enthusiasm at this point :smiley:

12

I'm more than happy for you guys to discuss it here - that would be great. I just need to set myself some boundaries otherwise I'll spend a chunk of my time on this when I have some much more urgent matters to deal with this week.

2 Likes
13

So this one is already answered by Nick :smiley:

I'll admit that I have no real ideas what v5 introduces so I have no feel currently for how big this will be.

Sounds like a sensible place to start coding anyway. We just need to recognise that if there are many aspects to v5, we need to spend some time getting all of it done.

Could setting from the msg object be considered the Minimum Viable Product (MVP)? Again, that sounds like a sensible place to start. But I doubt anyone could make a sensible answer to this without understanding what other complexities v5 brings.

Can you expand on this, not quite sure what you mean?

That seems sensible to me, especially if we are going to add other complexities to the node.

It would also allow us to hide anything v5 only if v5 isn't being chosen. It will also allow v5+ options to be easily added in the future.

1 Like
14

Looking at the options for the underlying library, there are a number of new properties that we should consider supporting I think?

Client setup options

options.properties : properties MQTT 5.0. object that supports the following properties:

  • sessionExpiryInterval : representing the Session Expiry Interval in seconds number ,
  • receiveMaximum : representing the Receive Maximum value number ,
  • maximumPacketSize : representing the Maximum Packet Size the Client is willing to accept number ,
  • topicAliasMaximum : representing the Topic Alias Maximum value indicates the highest value that the Client will accept as a Topic Alias sent by the Server number ,
  • requestResponseInformation : The Client uses this value to request the Server to return Response Information in the CONNACK boolean ,
  • requestProblemInformation : The Client uses this value to indicate whether the Reason String or User Properties are sent in the case of failures boolean ,
  • userProperties : The User Property is allowed to appear multiple times to represent multiple name, value pairs object ,
  • authenticationMethod : the name of the authentication method used for extended authentication string ,
  • authenticationData : Binary Data containing authentication data binary

options.will.properties : properties of will by MQTT 5.0:

  • willDelayInterval : representing the Will Delay Interval in seconds number ,
  • payloadFormatIndicator : Will Message is UTF-8 Encoded Character Data or not boolean ,
  • messageExpiryInterval : value is the lifetime of the Will Message in seconds and is sent as the Publication Expiry Interval when the Server publishes the Will Message number ,
  • contentType : describing the content of the Will Message string ,
  • responseTopic : String which is used as the Topic Name for a response message string ,
  • correlationData : The Correlation Data is used by the sender of the Request Message to identify which request the Response Message is for when it is received binary ,
  • userProperties : The User Property is allowed to appear multiple times to represent multiple name, value pairs object

Client Events

Event 'connect'

The connack object is returned - does this contain the new v5 features such as the capability flags? Probably doesn't need any changes to the Node though?

Event 'disconnect'

New v5 feature so does this needs handling somehow?

Client Publish Options

properties : MQTT 5.0 properties object

  • payloadFormatIndicator : Payload is UTF-8 Encoded Character Data or not boolean ,
  • messageExpiryInterval : the lifetime of the Application Message in seconds number ,
  • topicAlias : value that is used to identify the Topic instead of using the Topic Name number ,
  • responseTopic : String which is used as the Topic Name for a response message string ,
  • correlationData : used by the sender of the Request Message to identify which request the Response Message is for when it is received binary ,
  • userProperties : The User Property is allowed to appear multiple times to represent multiple name, value pairs object ,
  • subscriptionIdentifier : representing the identifier of the subscription number ,
  • contentType : String describing the content of the Application Message string

CLient Subscribe options

options is the options to subscribe with, including:

  • qos QoS subscription level, default 0
  • nl No Local MQTT 5.0 flag (If the value is true, Application Messages MUST NOT be forwarded to a connection with a ClientID equal to the ClientID of the publishing connection)
  • rap Retain as Published MQTT 5.0 flag (If true, Application Messages forwarded using this subscription keep the RETAIN flag they were published with. If false, Application Messages forwarded using this subscription have the RETAIN flag set to 0.)
  • rh Retain Handling MQTT 5.0 (This option specifies whether retained messages are sent when the subscription is established.)
  • properties : object
    • subscriptionIdentifier : representing the identifier of the subscription number ,
    • userProperties : The User Property is allowed to appear multiple times to represent multiple name, value pairs object

Client unsubscribe and end

Both have properties.userProperties in the options object.

Better error handling

Do we need to consider the new error handling/reporting features in v5? Or is that all exposed for us by the library?

Payload format indicators

This looks interesting and potentially REALLY useful for Node-RED. Mime-Type headers that help describe the content being returned - does the client pass these back?

That appears to be it. Phew, quite a bit there.

2 Likes
15

Hi Julian.

Firstly, a blog post that nicely sums up the majority of V5 in a 2 minute read: Hello MQTT Version 5.0!

Ok...

The issue I have with is this approach is getting lumbered with legacy code (to detect if the UI field is empty and the msg contains the property). Suppose we design it to say "Put your user properties in msg.userProperties" instead of adding a typedInput then we will always have to handle this. If we start clean and decide user properties (and perhaps some other v5 stuff) is done via a typedInput from the get go, we dont have this legacy code checking to see if a value was passed in the msg

So, most of the (publish) additional features in V5 are passed into the publish call via a properties object attached to the options object.

image
image834Ă—392 30.7 KB

Reference: Publish properties

Likewise, upon receiving a subscription event, any v5 properties will be returned in packet.properties.
image
Reference: packet

So, to be clearer, I said "is it right to return the user properties in msg.userProperties considering user properties are just one part of the MQTT V5 packet.properties object"

Perhaps it might be clear to say...

"Should we simply pass the properties object from the packet to the msg or extract the properties of packet.properties into the msg selectively"

hope that makes sense?

Yeah - it looks ok & makes future changes easy...
image
(3.1.1 is selected by default)

additional thoughts...

There is a lot of stuff in V5 that needs thought (shared topics, responseTopic etc) but after playing with some of the features, I can see no reason why we wouldnt be able to stage the additions incrementally.

Please keep the discussion coming.

1 Like
16

My apologies. I completely missed this post (hadn't refreshee browser before posting my previous reply)

I will read and absorb you post (probably tomorrow)

Bfn.

17

SUBJECT: MQTT on client connect

Maybe so maybe not. Warning - more questions than answers ahead.

When connecting to a V5 MQTT broker, we get a bunch of properties in the returned packet...

image

  • Should we be keeping a copy of these properties and be verifying users operations against the capabilities of the broker?
    • e.g. suppose a user sends a packet > 1048576 - should the mqtt node-red node nip this in the bud OR let it go to the broker (and fail)
    • e.g. suppose a user sends a publish with retain set should we prevent it being sent to the broker if retainAvailable is false?
  • or perhaps - we could provide some means of accessing these properties in node-red to permit the user to to check before publish/subscribe? (perhaps via an mqtt service or mqtt properties node?)

more questions to follow as i explore V5

18

SUBJECT: Payload format indicators

Julian,
Are you referring to payloadFormatIndicator AND / OR contentType?
Both are quite related IMO. REF: MQTT Payload Format Description and Content Type – MQTT 5 Essentials Part 8

Payload Format Indicator

The Payload Format Indicator is part of any MQTT packet that can contain a payload. Specifically, a CONNECT packet that contains a WILL message or a PUBLISH packet. The indicator is an optional byte value. A value of 0 indicates an “unspecified byte stream”. A value of 1 indicates a “UTF-8 encoded payload”. If no Payload Format Indicator is provided, the default value is 0.

Content Type

Similar to the Payload Format Indicator, the Content Type is optional and can be part of a CONNECT that contains a WILL message or of any PUBLISH. The Content Type must be a UTF-8 encoded string. The Content Type value identifies the kind of UTF-8 encoded payload. When the Payload Format Indicator is set to 1, a MIME content type descriptor is expected (but not mandatory). Any valid UTF-8 String can be used.

payloadFormatIndicator

I have not tested this, but it seems it needs to be a number of 1 or 0 (0 being the default)
QUESTION: Does anyone understand the significance of this?
QUESTION: Would this be a checkbox on the publish node and the LWT settings?

Content Type

I have tested contentType - if sent in options.properties.contentType it is simply returned in packet.properties -
QUESTION: Should this be an input or a typedInput along side payload on the publish node?
QUESTION: Should this be an input along side payload on the connection config node for LWT messages?

19

SUBJECT: Client Publish Options - messageExpiryInterval

I have tested this & it works well with my broker (EMQX) (have yet to test against a mosquitto broker)

I published a payload to topic test/test with options.messageExpiryInterval to 60 and retain to true. I disconnected and re-connected a client and the packet was returned. I tried again after 60 seconds and the packet was gone from the broker.

QUESTION: Would you hope to see this as an input on the publish UI (perhaps as a typedInput) (or just a msg property)?

I am starting to realises there is a TON to go through :smiley:

However, I am still positive we can impliemnent certain features independant of others.

As it stands, userProperties, messageExpiryInterval and contentType are easy to impliment, easy to understand and as far as i can see, implementable without too much consideration to things like topic aliases, extended errors etc.

Thoughts?

20

SUBJECT: Client Publish Options - topicAlias

About...

Topic Aliases are an integer value that can be used as a substitute for topic names. A sender can set the Topic Alias value in the PUBLISH message, following the topic name. The message receiver then processes the message like any other PUBLISH and persists a mapping between the Integer (Topic Alias) and String (Topic Name). Any following PUBLISH message for the same topic name can then be sent with an empty topic name, only using the defined Topic Alias.

Testing

I have tested topicAlias as follows...

PUB1

{"topic":"test/test-alias/long/topic/indeed","payload":"first pub with topic","topicAlias":1}

SUB

{"topic":"test/test-alias/long/topic/indeed","payload":"first pub with topic","qos":0,"retain":false,"topicAlias":1}

PUB2

{"payload":"publishing to topicAlias 1","topicAlias":1}

SUB

{"topic":"test/test-alias/long/topic/indeed","payload":"publishing to topicAlias 1","qos":0,"retain":false}

RESULT: Seems to work well. NOTE: when topicAlias is used, topic MUST be an empty string
QUESTION: How to represent this on the UI?