Node-RED Mqtt 5 support

Sorry, I have no answers. There IS a lot to go through. I'm thinking that it will not be feasible to implement everything in one go though. I think that we need a roadmap of v5 enhanced features, picking off the most important.

just a thought: Instead of looking at the list of v5 features and thinking about the easiest way of implementing them, maybe it is worth to think about for what kind of purposes these features could be used within the Node-RED context to determine HOW they should be exposed as options.
Maybe Steve you have already some kind of applications in mind.

3 Likes

My top 3 Cherry picked...

  • User Properties

    • My first use case for this is an installation already up and running and sending number/string type payloads BUT I want to "tag on" additional info (like where the message came from, a time stamp etc) for traceability and easier diagnostics
  • Message Expiry

    • Similar to above (but also for future projects) I would use this to persist values for a maximum time where you want the value to be retained but after 24h, it makes little sense.
  • Payload Format Description

    • this is particularly useful for describing the payload to a third party.

I am working on a feature + effort vs benefit table with notes and links that I will post in due course.

After a fair bit reading, thinking & playing, am now fairly certain (and Julian agrees) that the new features will need to be implemented in isolated / non overlapping phases & I hope to get agreement on the effort vs benefit & some more thoughts about implementation/UI.

Lets see where this goes.

3 Likes

Hi all, so after some time evaluating and making sense of (where i can) the specification & every piece of information i can find around the internet I have some updates to share...

Foreword

  • Many parts of the MQTT V5 additions are covered (but not all). I have listed out most of the features and provided some notes regarding their purpose and status.
  • It was not my intention to actually upgrade the MQTT nodes however in testing how they work and fit into node-red, I ended up pretty much doing that so I hope what i have done is of some use.
  • A lot of this is a brain dump - please be forgiving :slight_smile:

First some demos (ease you in)

Broker Node

Publish node

Subscribe node

User Properties...
image

Topic alias

Request / Response Pattern

no local (and more user properties / content type)

Shared topics
2020-10-14_13-17-00

Details...

Server/Broker options Implementation notes Testing Follow up
Session expiration. Split the cleanup session flag into a new start flag and the session expiration interval can be modified when disconnected. number type input. Type checked and coerced into integer before updating v5 properties object 100%
Flow control. Allow the client and server to specify the number of outstanding reliable messages (QoS> 0) respectively number type input. Type checked and coerced into integer before updating v5 properties object 50% Attempted to test by setting low value (5) but couldnt trigger the disconnect. Suspect this is something I am doing wrong in my tests
Maximum packet length. Allow the client and server to individually specify the maximum message lengths they support number type input. Type checked and coerced into integer before updating v5 properties object 50% Although implemented and tested, it seems a bit odd in that when requesting the msg size to be smaller, a subscription node still receives a message but the topic is empty. So it works but not as expected.
Reason codes of all confirmation packets. All response packets contain reason codes so that the caller can determine whether the requested function was successful. Leave for future update NA In my trials and testing, the 'message' event never seems to contain this! Perhaps we need to handle 'packet' event instead?
Optional server function availability. Inform the client the feature list supported by the server to avoid the client using unsupported features Upon connack, I capture the flags the broker reported. 50% Utilise these flags and determine safe defaults. Not necessary for 1st v5 release. Can easily be utilised at a later for a later revision / when requested?
Enhanced authentication. Provide a mechanism to enable challenge/response style authentication including mutual authentication. Leave for future update NA OK to leave this for a later revision / when requested?
Server Disconnect. The server is allowed to send a DISCONNECT message to indicate why the connection was closed. The event is wired up but unsure how to handle it 50% What to do with this event? The broker disconnects regardless so its probably for client side cleanup / visualisation / logging only?
Server reference / redirect. Allows the server to specify a standby server. Leave for future update NA OK to leave this for a later revision / when requested?
Will delay. Provide the ability to delay the sending of the specified will message after the connection is interrupted Leave for future update NA OK to leave this for a later revision / when requested?
Server keeps connected. Allow the server to specify the keep-alive values it wants the client to use Leave for future update NA OK to leave this for a later revision / when requested?
Subscribe options Implementation notes Testing Follow up / notes
Shared subscription. Load balancing groups (round robin) Already impliemented 100% Round robin tested + working as expected
Subscription Identifier. Allows to specify a digital subscription identifier in the subscription packet and returns this identifier when the message is distributed The value is parsed into integer type and passed to the v5 properties object when > 0. 75% If a subID is present in an incoming message AND this UI option is set, this acts like a secondary filter and actually solves the multiple messages on overlapping wildcard topics. Is this a reasonable approach for NR implementation?
no local. Don’t send published messages back to same client Checkbox flag option. Type checked and coerced into boolean before updating v5 properties object 100%
Retain as Published. Messages forwarded using this subscription keep the RETAIN flag they were published with Checkbox flag option. Type checked and coerced into boolean before updating v5 properties object 100% Works as described (though I am not certain of usefulness the feature).
Retain Handling. 0: always get retained messages, 1: get retained messages on new subscription, 2: never get retained messages Select option. Type checked and coerced into integer 0 ~ 2 before updating v5 properties object 50% I have verified correct value is passed to v5 properties. Seems to work but needs thought around option 1.
User Properties. Support user-defined attributes and transmission of additional custom information to expand more application scenarios Permit JSON and flow/global. Under the hood, a helper function ensures only string pair values are sent / received 100% Unsure of the purpose of user properties in the subscribe node! I suspect they arrive in the suback packet for use by other broker/client implementations.
Publish Options Implementation notes Testing Follow up / notes
User Properties. Support user-defined attributes and transmission of additional custom information to expand more application scenarios TypedInput permits JSON and msg/flow/global. Under the hood, a helper function ensures ony string pair values are sent / received Done Should this be an editableList (or is typedInput acceptable for initial version)
Topic alias. Reduce the overhead of MQTT messages by shortening the topic name to a small integer. Can be passed in msg.topicAlias if empty. NOTE Broker disconnects if topicAlias is used without first being set. This is per spec Done Should this be a typed input? (or is std input acceptable in first release)
Message expiration. Allows messages to be published with an expiration interval Can be passed in msg.sessionExpiryInterval if empty Done Should this be a typed input? (or is std input acceptable in first release)
Payload format and content type. Allows to specify payload format (binary, text) and MIME-style content types when publishing messages Can be passed in msg.contentType and msg.payloadFormatIndicator if fields are empty. 50% The contentType is simple (passed back in the message) but unsure how to handle utf8. Is the binary/utf8 transmission handled by the client? Cant find clear details
Request / Response Pattern. Specify the MQTT request / response mode, provide the response topic and compare data attributes, and control the response message to be routed back to the request publisher Can be passed in msg.responseTopic and and msg.correlationData if fields are empty. Done For this implimentation, correlationData must be a buffer (as per spec). however, should we permit string/other value types in correlationData & coerce this to a buffer in the backend? The issue arises when we try to do the reverse on an incoming message. Is it acceptable to leave this as a buffer in first release?
Shared subscription. support for shared subscriptions to allow load balancing for multiple subscription consumers Tested 3 client connections and tested the round robin effect. Done
client identifier. If the server has assigned a client identifier, this client identifier is returned to the client. Leave for future update NA Cant find clear information on this. OK to leave this for a later revision / when requested?
Subscription Identifier. An ID associated with the subscription The value is parsed into integer type and passed to the v5 properties object when > 0 (as per spec) Done I cant for the life of me understand how Sub ID operates at the publish side. Oasis desc on Subscription Identifier for publish packet is unclear.

Notes

  • V5 options are hidden by default and show when a user selects V5 option in the broker config node.
  • Publishing or Subscribing with incorrect properties value types or incorrect combinations of values can crash node-red. A few helper functions that check bounds and coerce types were added to contain this.
  • Currently, MQTT Node doesn’t unsubscribe a topic until the node is deleted, in order for changed options to take effect, this probably needs changing (in my works I have changed this)
  • For publish nodes, the v5 properties that can be sent in msg object use the exact name/case/spelling as the MQTT.JS lib for consistency. However, if all items were typedInputs, this would be moot.

Reference: Oasis Spec

Update
More testing and above info updated.

2 Likes

Hi @knolleary & @dceejay

I have waited until your crazy hectic schedule with v1.2.0 release was done and dusted before shouting out - I hope this is now the time to discuss MQTT V5 in node-red?

I have (as I touched on in the post above) done a fair bit work in my play time testing of V5 features and (unintentionally) wound up implementing them in the MQTT nodes as I went along (made sense to see how the features would knit in).

If this work is of use, please give me a shout with how you would like to approach it? PR? other workspace?

I know this is quite a sizeable nugget but hope now is a good time to approach it, nice and early and in time for 1.3.x release?

I think they are still at work on 1.2 - last thing I saw said working on a 1.2.2 release

@Steve-Mcl thanks, I will take a look through what you've done this week.