Dynamic subscribe MQTT

[Steve-Mcl]

Hi @knolleary @dceejay - is this contribution still wanted?

@knolleary

Where there is a need for another MQTT node is to be able to dynamically subscribe to topics at runtime - where you can't hard code the topic into the node.

We do have a design for such a node and it is sat in the backlog until someone is interested enough to contribute to its implementation.

I had a bit play time today and generated a new node, based fully on the current node-red mqtt nodes (in fact I did it on my node-red fork under branch mqtt-dyn, based off current dev branch today)

Foreword...

To keep things simple, the node does not permit dynamic/runtime choice of broker (ie this new node must have a broker selected in its config). This made implementation relatively easy and very similar in structure to the existing built in MQTT nodes.

For want of a better name, I called this node "mqtt sub"

Implementation

Inputs

The node accepts a msg with 3 topic options...

• "subscribe" - subscribe the item(s) specified in msg.payload
• "unsubscribe" - unsubscribe the item(s) specified in msg.payload
• "list" - this returns the current list of subscriptions

The payload can be either...

info	payload	notes
BASIC/SINGLE	string	The payload is the topic to subscribe to
ADVANCED/SINGLE	object	payload object should contain .topic. Can also include .qos and other MQTT V5 options
BASIC/MULTIPLE	array of strings	an array of topics to permit multiple subs in one go
ADVANCED/MULTIPLE	array of objects	An array of objects to permit multiple subs (with options), in one go

Outputs

The outputs for mqtt subscriptions and status/control messages are kept separate. The reasoning behind having 2 outputs was 2 fold...

1. Sending "list" to list out the topics is all good and well until you actually have a subscription to the topic "list" - you wouldnt know which msg is the sub and which msg is the response to a list command
2. Removes the need for a switch node after the "mqtt sub" node (all outputs from output 1 are subscription messages, all outputs from output 2 are control/status messages)

DEMO...

NOTE: I checked the whiteboard and the issues but could not see the backlog entry for this feature - so my apologies if this is nothing like you wanted.

[knolleary]

Hi @Steve-Mcl

as I mentioned in that old post, we have had a design for such a node for a long time. Had you asked, I'd have happily shared the link before you started implementing something. You can find the details here: designs/dynamic-mqtt-node.md at master · node-red/designs · GitHub

Have a read of that and let me know what you think. Keep in mind it was written prior to v5 so I suspect there's more thought needed in some of the options.

[Steve-Mcl]

I knew i had seen it somewhere - doh.

I'll have a look over and feedback Nick.

[Steve-Mcl]

Hi Nick.

Fortunately what I wrote was not a million miles off and I have now implemented the remaining features connect / disconnect / publish features and the auto connect flag for the broker.

I have also renamed it MQTT control

QUESTION:

• How stuck are you on using msg.mqttControl? It feels a bit superfluous since every "action" is a string - it feels more suited to topic? As it stands, i use topic as the action and payload is the parameters for the action. I have no issue changing this but just wanted to ask first.

NOTES:

• in order to support publish action, I had to refactor the MQTT Out node so as to avoid duplication of code
• regarding "If the broker is already connected, the action will be ignored. ( Q : or should it cause a disconnect and reconnect?)" : currently, if the node is connected, I send a node.error to tell the user to disconnect first. Obviously this can be changed.
• regarding "It will also pass through messages it receives when the action is complete. TODO : identify how to distinguish these from subscription messages" - I handled this by sending control responses to 2nd output That means only subscription messages come out of output 1 (meaning no need to distinguish control messages from subscription messages)

TESTED:

• LWT birth and close sent on actions connect and disconnect - ok
• LWT will sent on broker shutdown - ok
• connect without options + initial good config - connects ok
• connect without options + initial bad config : fails to connect & shows "connecting" - ok
• connect with options + initial bad config - new settings are applied and connects ok
• disconnect - ensure disconnection possible from connected AND connecting state
• subscribe 1 - ok
• subscribe multiple - ok
• subscribe 1/multiple with v5 properties - ok
• unsubscribe 1 - ok
• unsubscribe multiple - ok
• broker retained messages arrive when a connection is made (even when changing broker)

Let me know if you want me to proceed and any thoughts you have.

[knolleary]

Thanks Steve.

I haven't even glanced at this design in any detail since writing it. All of the details are up for discussion, but as it stands, I'll need to remind myself what it says before digesting everything you've written.

@Steve-Mcl

How stuck are you on using msg.mqttControl ? It feels a bit superfluous since every "action" is a string - it feels more suited to topic ?

Given topic has meaning with mqtt, I do not want to overload it as a command property.

Raise a PR in dev and we can take a look. It may well be once we have an implementation to play with, we decide the design needs changing.

I just can't promise to look at it immediately as my plate is full getting 2.0 beta ready this week.

It would be good to get into 2.0, just may not be the first beta.

[Steve-Mcl]

Yeah I was hoping it would make it into 2.0.0 betas as there is a fair bit refactoring.

I understand your point about topic being an mqtt thing and to avoid confusion so I'll spend a bit more time mulling it over and raise a PR tomorrow if that's ok?

[knolleary]

No rush. I'm not likely to look at it in any detail for a few days.

[Steve-Mcl]

Morning Nick.

I know you are super busy but would appreciate 2 mins to quickly comment on the following proposal...

In order to provide visual cues to the user and simplify setting up the action (while avoiding overloading topic) would using a drop down like the HTTP Request node be a good compromise? I propose the user would be offered a list of actions to chose from and a - set by msg.action - option. Using a separate msg variable string like msg.action simplifies things for the user to a simple change node (as opposed to having to create msg.mqttControl object for something simple like a publish or disconnect action)

mock up

notes

I dont want to repeat the whole UI of the connection node / publish node / subscribe node (as I think we should consider this node an advanced user node) but we could provide a tip to direct the user to figuring it out for themselves?

Is this a suitable direction?

[Steve-Mcl]

PR Raised

mqtt-control