Sorry, I have no answers. There IS a lot to go through. I'm thinking that it will not be feasible to implement everything in one go though. I think that we need a roadmap of v5 enhanced features, picking off the most important.
just a thought: Instead of looking at the list of v5 features and thinking about the easiest way of implementing them, maybe it is worth to think about for what kind of purposes these features could be used within the Node-RED context to determine HOW they should be exposed as options.
Maybe Steve you have already some kind of applications in mind.
3 Likes
My top 3 Cherry picked...
-
User Properties
- My first use case for this is an installation already up and running and sending number/string type payloads BUT I want to "tag on" additional info (like where the message came from, a time stamp etc) for traceability and easier diagnostics
-
Message Expiry
- Similar to above (but also for future projects) I would use this to persist values for a maximum time where you want the value to be retained but after 24h, it makes little sense.
-
Payload Format Description
- this is particularly useful for describing the payload to a third party.
I am working on a feature + effort vs benefit table with notes and links that I will post in due course.
After a fair bit reading, thinking & playing, am now fairly certain (and Julian agrees) that the new features will need to be implemented in isolated / non overlapping phases & I hope to get agreement on the effort vs benefit & some more thoughts about implementation/UI.
Lets see where this goes.
3 Likes
Hi all, so after some time evaluating and making sense of (where i can) the specification & every piece of information i can find around the internet I have some updates to share...
Foreword
- Many parts of the MQTT V5 additions are covered (but not all). I have listed out most of the features and provided some notes regarding their purpose and status.
- It was not my intention to actually upgrade the MQTT nodes however in testing how they work and fit into node-red, I ended up pretty much doing that so I hope what i have done is of some use.
- A lot of this is a brain dump - please be forgiving
First some demos (ease you in)
Broker Node
Publish node
Subscribe node
User Properties...
Topic alias
Request / Response Pattern
no local (and more user properties / content type)
Shared topics
Details...
| Server/Broker options | Implementation notes | Testing | Follow up |
|---|---|---|---|
| Session expiration. Split the cleanup session flag into a new start flag and the session expiration interval can be modified when disconnected. | number type input. Type checked and coerced into integer before updating v5 properties object | 100% | |
| Flow control. Allow the client and server to specify the number of outstanding reliable messages (QoS> 0) respectively | number type input. Type checked and coerced into integer before updating v5 properties object | 50% | Attempted to test by setting low value (5) but couldnt trigger the disconnect. Suspect this is something I am doing wrong in my tests |
| Maximum packet length. Allow the client and server to individually specify the maximum message lengths they support | number type input. Type checked and coerced into integer before updating v5 properties object | 50% | Although implemented and tested, it seems a bit odd in that when requesting the msg size to be smaller, a subscription node still receives a message but the topic is empty. So it works but not as expected. |
| Reason codes of all confirmation packets. All response packets contain reason codes so that the caller can determine whether the requested function was successful. | Leave for future update | NA | In my trials and testing, the 'message' event never seems to contain this! Perhaps we need to handle 'packet' event instead? |
| Optional server function availability. Inform the client the feature list supported by the server to avoid the client using unsupported features | Upon connack, I capture the flags the broker reported. | 50% | Utilise these flags and determine safe defaults. Not necessary for 1st v5 release. Can easily be utilised at a later for a later revision / when requested? |
| Enhanced authentication. Provide a mechanism to enable challenge/response style authentication including mutual authentication. | Leave for future update | NA | OK to leave this for a later revision / when requested? |
| Server Disconnect. The server is allowed to send a DISCONNECT message to indicate why the connection was closed. | The event is wired up but unsure how to handle it | 50% | What to do with this event? The broker disconnects regardless so its probably for client side cleanup / visualisation / logging only? |
| Server reference / redirect. Allows the server to specify a standby server. | Leave for future update | NA | OK to leave this for a later revision / when requested? |
| Will delay. Provide the ability to delay the sending of the specified will message after the connection is interrupted | Leave for future update | NA | OK to leave this for a later revision / when requested? |
| Server keeps connected. Allow the server to specify the keep-alive values it wants the client to use | Leave for future update | NA | OK to leave this for a later revision / when requested? |
| Subscribe options | Implementation notes | Testing | Follow up / notes |
|---|---|---|---|
| Shared subscription. Load balancing groups (round robin) | Already impliemented | 100% | Round robin tested + working as expected |
| Subscription Identifier. Allows to specify a digital subscription identifier in the subscription packet and returns this identifier when the message is distributed | The value is parsed into integer type and passed to the v5 properties object when > 0. | 75% | If a subID is present in an incoming message AND this UI option is set, this acts like a secondary filter and actually solves the multiple messages on overlapping wildcard topics. Is this a reasonable approach for NR implementation? |
| no local. Donât send published messages back to same client | Checkbox flag option. Type checked and coerced into boolean before updating v5 properties object | 100% | |
| Retain as Published. Messages forwarded using this subscription keep the RETAIN flag they were published with | Checkbox flag option. Type checked and coerced into boolean before updating v5 properties object | 100% | Works as described (though I am not certain of usefulness the feature). |
| Retain Handling. 0: always get retained messages, 1: get retained messages on new subscription, 2: never get retained messages | Select option. Type checked and coerced into integer 0 ~ 2 before updating v5 properties object | 50% | I have verified correct value is passed to v5 properties. Seems to work but needs thought around option 1. |
| User Properties. Support user-defined attributes and transmission of additional custom information to expand more application scenarios | Permit JSON and flow/global. Under the hood, a helper function ensures only string pair values are sent / received | 100% | Unsure of the purpose of user properties in the subscribe node! I suspect they arrive in the suback packet for use by other broker/client implementations. |
| Publish Options | Implementation notes | Testing | Follow up / notes |
|---|---|---|---|
| User Properties. Support user-defined attributes and transmission of additional custom information to expand more application scenarios | TypedInput permits JSON and msg/flow/global. Under the hood, a helper function ensures ony string pair values are sent / received | Done | Should this be an editableList (or is typedInput acceptable for initial version) |
| Topic alias. Reduce the overhead of MQTT messages by shortening the topic name to a small integer. | Can be passed in msg.topicAlias if empty. NOTE Broker disconnects if topicAlias is used without first being set. This is per spec | Done | Should this be a typed input? (or is std input acceptable in first release) |
| Message expiration. Allows messages to be published with an expiration interval | Can be passed in msg.sessionExpiryInterval if empty | Done | Should this be a typed input? (or is std input acceptable in first release) |
| Payload format and content type. Allows to specify payload format (binary, text) and MIME-style content types when publishing messages | Can be passed in msg.contentType and msg.payloadFormatIndicator if fields are empty. | 50% | The contentType is simple (passed back in the message) but unsure how to handle utf8. Is the binary/utf8 transmission handled by the client? Cant find clear details |
| Request / Response Pattern. Specify the MQTT request / response mode, provide the response topic and compare data attributes, and control the response message to be routed back to the request publisher | Can be passed in msg.responseTopic and and msg.correlationData if fields are empty. | Done | For this implimentation, correlationData must be a buffer (as per spec). however, should we permit string/other value types in correlationData & coerce this to a buffer in the backend? The issue arises when we try to do the reverse on an incoming message. Is it acceptable to leave this as a buffer in first release? |
| Shared subscription. support for shared subscriptions to allow load balancing for multiple subscription consumers | Tested 3 client connections and tested the round robin effect. | Done | |
| client identifier. If the server has assigned a client identifier, this client identifier is returned to the client. | Leave for future update | NA | Cant find clear information on this. OK to leave this for a later revision / when requested? |
| Subscription Identifier. An ID associated with the subscription | The value is parsed into integer type and passed to the v5 properties object when > 0 (as per spec) | Done | I cant for the life of me understand how Sub ID operates at the publish side. Oasis desc on Subscription Identifier for publish packet is unclear. |
Notes
- V5 options are hidden by default and show when a user selects V5 option in the broker config node.
- Publishing or Subscribing with incorrect properties value types or incorrect combinations of values can crash node-red. A few helper functions that check bounds and coerce types were added to contain this.
- Currently, MQTT Node doesnât unsubscribe a topic until the node is deleted, in order for changed options to take effect, this probably needs changing (in my works I have changed this)
- For publish nodes, the v5 properties that can be sent in msg object use the exact name/case/spelling as the MQTT.JS lib for consistency. However, if all items were typedInputs, this would be moot.
Reference: Oasis Spec
Update
More testing and above info updated.
5 Likes
Hi @knolleary & @dceejay
I have waited until your crazy hectic schedule with v1.2.0 release was done and dusted before shouting out - I hope this is now the time to discuss MQTT V5 in node-red?
I have (as I touched on in the post above) done a fair bit work in my play time testing of V5 features and (unintentionally) wound up implementing them in the MQTT nodes as I went along (made sense to see how the features would knit in).
If this work is of use, please give me a shout with how you would like to approach it? PR? other workspace?
I know this is quite a sizeable nugget but hope now is a good time to approach it, nice and early and in time for 1.3.x release?
I think they are still at work on 1.2 - last thing I saw said working on a 1.2.2 release
@Steve-Mcl thanks, I will take a look through what you've done this week.
I am really thirst to have MQTT5 features msg.responseTopic and msg.correlationData on my system because I want to implement an API over MQTT. From what I understood, this is already implemented by @Steve-Mcl, is that correct? Can I have a version to play with it?
Hi Semen,
Only on a local clone of my fork at the moment.
Do you have good knowledge of the MQTT5 specification & are you familiar with developing for node-red (or nodejs)?
If I push the MQTT5 branch to my fork, will you test the features and perhaps feedback / contribute?
Regards, Steve.
Yes I am a software developer and have a more or less good knowledge of the MQTT in general, but not really familiar with the internal node-red code base. So I probably would be able to build your fork and test it (at least with the functionality I want to use). Could you share me a link to your fork?
Hi, you will find my branch here
Please feedback here any concerns, or suggestions.
NOTE: I have yet to property discuss with the guys in charge so no PR is in place as of yet.
EDIT - This branch was merged with the upstream/dev branch just before I pushed it so i havent tested it myself yet.
1 Like
Hi Steve, I was able to test your branch, looks awesome so far, I was able to implement a simple request-response workflow I wanted. I will let it run in order to test the stability of the connection and data handling. What can I test beside of this?
anything / everything 
Critique the layout, options etc.
If you are up to speed on MQTT5, are things how you would expect etc. (I had to learn as i went and may not have implemented things correctly / as expected)
Hm I'm not the MQTT expert so far, but what I can confirm is: connection settings I used (actually default settings I use for a stable Node-RED version) for the mqtt-in and mqtt-out blocks work as expected, response topic is set correctly, I'm able to use the responseTopic field to publish back to the client. I took a look also on your source code changes, I didn't found anything suspicious as well.
1 Like
I run that version over the night and can confirm that it runs stable and keep MQTT connection alive. So from my perspective I would like to see it in the next Node-RED release.
2 Likes
Do you @Steve-Mcl think you would create a PR from your fork to the main Node-RED repo so we can have your changes in the regular Node-RED version?
I am waiting on Nick to take a look.
He is a busy man & I don't what to hassle him (too much).
Anyhow, I do have some free time this week so I will try to get it bang up to date with current dev branch in readiness.
@semen-miroshnichenko can I assume all your experiences with my fork were good? Have you been running it continuously?
@Steve-Mcl raise a PR with the code. Until that happens I'm unlikely to look at it.
No bother, will do in a day or two - its amazing how quick you forget stuff if you dont keep using it - gotta do a full re-learn of MQTT5 spec & check it over again.
Against DEV branch Nick?
Yes, dev branch as its new functionality.
1 Like
