I'm writing code for a dashboard to display data and set parameters via MQTT.
The issue I have is that we have multiple instruments that has a bunch of different sensors and pump/valves connected.
There are over hundred different topics i need to relate to, and also many different instruments.
The topic is of the form:
system_serial_number/alfasensor/o2/o2_percent/pv
the system_serial_number is a text string unique to each instrument.
For the mqtt out node it is easy, because here I have full control to set the topic myself in a function preceeding the mqtt out node.
For the mqtt in, I would ideally be able to build the topic by adding a global variable containing the system_serial_number with the remaining part of the topic, like this:
topic = global.get('serialnr')+"/remaining part of topic" Where "/remaining part of topic" is a text string I type in.
The reason for this is when I port the code to a new instrument, I would only need to change the content of the global variable containing the instrument serial_number, and all the rest will be the same. (I do not need to go and change the 100+ topics manually to make it fit the new instrument's serial number).
I have read the post on this forum called "Using subflow environment variable to set mqtt topic". knolleary explains how to use a subflow to enable use of environmental variables as topics. I have tried this, but do not get it to work as I want. I probable have not understood fully how this works.
I would be very thankful if anyone here on the forum can help me how to achieve this.
To recap:
For the MQTT subscribe node, I would like to build up the topic string as a sum of a variable (the serial number) and a text containing rest of the topic. (this is to make it easy to port the code for a new instrument, as only the serial number changes)
For MQTT-In you can use dynamic subscriptions (see the built in help)
Or, you can use topic wildcards: +/alfasensor/o2/o2_percent/pv then ALL matching topics will arrive (you can split the topic to determine which serial number published) Then you can match the split out serial number to a global variable using a switch node.
Thanks for the reply.
Unfortunately none of those options suit my application.
The wildcards will generate too much traffic, cause many of the instruments will reply, which is not desireable.
The dynamic subscription is also not a solution that fit my need.
I was hoping I can put together a topic by using a variable containing the serial number and join that with a text string.
I doubt that is true tbh. How can dynamic subscription not work for you? When a serial number is input you can simply send a control message to the node & subscribe to exactly the item(s) you need
You can, then send it to the mqtt-in node with the appropriate action "subscribe"
But what @Steve-Mcl said doesn't change the traffic happening. All this traffic will still be there if you have the MQTT IN node there and set as suggested or not.
After that node, use a switch node and extract the required topics with something like:
Set the switch node to do topic and not the default payload.
Then if topic contains (device 1) --> output 1 if topic contains (device 2) --> output 2
and so on.
The extra load shouldn't be that much.
And remember: Those message will still be received anyway.
They only become visible when you put the MQTT IN node and start to look at what you are getting.
Not having the node doesn't stop them being received.
I will have a go to see how the system will react to using a wildchar, might be OK.
Dynamic subscription:
The serial number of the instrument never change. I want to subscribe to topics from that unique serial number as soon as the instrument is powered on, and never stop listening to those topics. How would I do that? By using a inject node that triggers once at start-up and injecting the action subscribe with the topic string I want?
That will probably work, it just seems a lot of nodes needed to achieve the goal of setting the topic.
Not really, you can build an array of topics to subscribe to and subscribe them all in one action (2 nodes in total for 1 or 1000 topics)
From the help: Note how the subscription topic can be 1 or more items:
e.g.
Demo Flow
[{"id":"54efeb2f88129762","type":"mqtt in","z":"08c327c2653de72a","name":"","topic":"","qos":"2","datatype":"auto-detect","broker":"57084e057bec8aac","nl":false,"rap":true,"rh":0,"inputs":1,"x":1350,"y":60,"wires":[["b6eacfd466ca7afd"]]},{"id":"938d984d44b95f78","type":"inject","z":"08c327c2653de72a","name":"Startup (serial abc01234)","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":true,"onceDelay":0.1,"topic":"","payload":"abc01234","payloadType":"str","x":950,"y":60,"wires":[["09184efe8ed6a72b"]]},{"id":"09184efe8ed6a72b","type":"function","z":"08c327c2653de72a","name":"Build topics","func":"// get the serial number from \"wherever you keep it\"\nconst serialNo = msg.payload // sent in via msg.payload\n// const serialNo = 'abc123' // hardcoded\n// const serialNo = env.get('SERIAL_NO') // from an Env Variable\n// const serialNo = global.get('serial_number', 'my-persistent-context') // from a persistent context Variable\n\n// ensure serial no is OK\nif (!serialNo) {\n node.error(\"Serial Number is not set\", msg);\n return\n}\n\n// send a msg to unsubscribe all\nnode.send({action:\"unsubscribe\", topic: true})\n\n// build the topics\nconst topics = [\n `${serialNo}/alfasensor/o2/o2_percent/sv`,\n `${serialNo}/alfasensor/o2/o2_percent/pv`,\n `${serialNo}/betasensor/o2/o2_percent/sv`,\n `${serialNo}/betasensor/o2/o2_percent/pv`,\n]\n\n// send a msg with subscribe action and topics\nmsg.action = 'subscribe'\nmsg.topic = topics\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":1200,"y":60,"wires":[["54efeb2f88129762"]]},{"id":"b6eacfd466ca7afd","type":"debug","z":"08c327c2653de72a","name":"debug 151","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":1530,"y":60,"wires":[]},{"id":"2e0315c8a479069b","type":"inject","z":"08c327c2653de72a","name":"Change Serial to xyz56789","props":[{"p":"payload"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"xyz56789","payloadType":"str","x":950,"y":100,"wires":[["09184efe8ed6a72b"]]},{"id":"cf8eec0f113a6b94","type":"mqtt out","z":"08c327c2653de72a","name":"","topic":"","qos":"","retain":"","respTopic":"","contentType":"","userProps":"","correl":"","expiry":"","broker":"57084e057bec8aac","x":1570,"y":160,"wires":[]},{"id":"c8b75e5cf917a2e9","type":"inject","z":"08c327c2653de72a","name":"poke dummy data to abc01234/alfasensor/o2/o2_percent/pv","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"5","crontab":"","once":false,"onceDelay":0.1,"topic":"abc01234/alfasensor/o2/o2_percent/pv","payload":"(\t $minimum := 1;\t $maximum := 10;\t $round(($random() * ($maximum-$minimum)) + $minimum, 0)\t)","payloadType":"jsonata","x":1220,"y":160,"wires":[["cf8eec0f113a6b94"]]},{"id":"0bcafba21a7ed7e0","type":"inject","z":"08c327c2653de72a","name":"poke dummy data to xyz56789/alfasensor/o2/o2_percent/pv","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"5","crontab":"","once":false,"onceDelay":0.1,"topic":"xyz56789/alfasensor/o2/o2_percent/pv","payload":"(\t $minimum := 10;\t $maximum := 20;\t $round(($random() * ($maximum-$minimum)) + $minimum, 0)\t)","payloadType":"jsonata","x":1220,"y":200,"wires":[["cf8eec0f113a6b94"]]},{"id":"57084e057bec8aac","type":"mqtt-broker","name":"","broker":"broker.emqx.io","port":"1883","clientid":"","autoConnect":true,"usetls":false,"protocolVersion":"5","keepalive":"60","cleansession":true,"birthTopic":"","birthQos":"0","birthPayload":"","birthMsg":{},"closeTopic":"","closeQos":"0","closePayload":"","closeMsg":{},"willTopic":"","willQos":"0","willPayload":"","willMsg":{},"userProps":"","sessionExpiry":""}]
Lastly:
If you need to have your topics separated, you could use a switch node or this topic filter subflow node
Using the wildcard topic, all matching MQTT data will be unnecessarily sent to the node-red instance. For example, imagine the topic is set for +/count and there are 1000 different devices publishing a count value. With the wild card you will get 1000 hits in your node red. With dynamic subscription you will get only one hit.
This is nothing to do with dynamic subscription. It is the fact that if you subscribe to a wildcard the broker sends all matching topics to the MQTT node.
If using dynamic allowed one to subscribe only to the topics of interest at that time, rather than subscribing to many more topics using wildcards, then it would reduce the traffic.
Dynamic subscription is a feature of the broker. It allows one to change the topics one is subscribed to without having to disconnect and reconnect. Remember this is about traffic between the broker and node red, not between the publisher and the broker. The broker only sends data to the client (node-red) for topics that are currently subscribed to. With MQTT the publishing devices send data to the broker, and the broker sends the required data to each subscribing client individually. The data does not go direct from the publisher to the subscribers.
This has really enlightened me on how to use the dynamic MQTT subscription.
I have impelmented the dynamic, but with one subscription node per topic. I will have a look and see if it is more benificial to the subscription in one action. I like the code you made in the example, it is very smart way to do it. Thanks a lot for the help
