# Get mqtt topic data on demand?

**URL:** <https://discourse.nodered.org/t/get-mqtt-topic-data-on-demand/6572>\
**Category:** General\
**Created:** [8 January 2019 18:03 UTC](https://discourse.nodered.org/t/get-mqtt-topic-data-on-demand/6572 "2019-01-08T18:03:47Z")\
**Posts on this page:** 20\
**Page:** 3

<div class="post-metadata">

**Author:** ![ukmoose](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/ukmoose/32/13_2.png) [@ukmoose](https://discourse.nodered.org/u/ukmoose)\
**Post date:** [16 January 2019 18:31 UTC](https://discourse.nodered.org/t/get-mqtt-topic-data-on-demand/6572/41 "2019-01-16T18:31:48Z")

</div>

if you google “facebook and MQTT” it should keep you busy!

---

<div class="post-metadata">

**Author:** ![cymplecy](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/cymplecy/32/2773_2.png) [@cymplecy](https://discourse.nodered.org/u/cymplecy)\
**Post date:** [16 January 2019 18:41 UTC](https://discourse.nodered.org/t/get-mqtt-topic-data-on-demand/6572/42 "2019-01-16T18:41:24Z")

</div>

Very interesting - ta 🙂

---

<div class="post-metadata">

**Author:** ![knolleary](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/knolleary/32/3_2.png) [@knolleary](https://discourse.nodered.org/u/knolleary)\
**Post date:** [16 January 2019 20:14 UTC](https://discourse.nodered.org/t/get-mqtt-topic-data-on-demand/6572/43 "2019-01-16T20:14:56Z")

</div>

> [@ukmoose](#):
>
> facebook and MQTT

It's worth saying that is quite an old reference, and they have rewritten their mobile app a few times since. So I'd be wary of rushing to cite them unless a more recent confirmation is found.

---

<div class="post-metadata">

**Author:** ![shellian.systems](https://avatars.discourse-cdn.com/v4/letter/s/b782af/32.png) [@shellian.systems](https://discourse.nodered.org/u/shellian.systems)\
**Post date:** [17 January 2019 13:40 UTC](https://discourse.nodered.org/t/get-mqtt-topic-data-on-demand/6572/44 "2019-01-17T13:40:24Z")

</div>

As Nick noted, the major messaging apps continually update their systems but given Facebook did use mqtt, at least as recently as 5 years ago that I can find cited, I think it's unlikely that they have changed their entire infrastructure to anything different.

From my knowledge, there are really only two practical choices for any large scale messaging apps and that would be XMPP or MQTT. I think in more recent years the shift has been to mqtt from a systems architecture point of view. Certainly, if I was making the decision on which way to go now in a new development it would be mqtt.

Robert Abramczyk over at netguru has an interesting read on implementing a messaging app use case using mqtt...

> **[MQTT - a Real-life Use Case Analysis](https://www.netguru.com/codestories/mqtt-a-real-life-use-case-analysis)**
>
> A real-life application of the MQTT protocol analyzed in detail. Learn how to create a reliable Internet of Things product using all important tricks up MQTT's sleeve.

Brian

---

<div class="post-metadata">

**Author:** ![knolleary](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/knolleary/32/3_2.png) [@knolleary](https://discourse.nodered.org/u/knolleary)\
**Post date:** [17 January 2019 13:53 UTC](https://discourse.nodered.org/t/get-mqtt-topic-data-on-demand/6572/45 "2019-01-17T13:53:14Z")

</div>

> [@shellian.systems](#):
>
> any large scale messaging apps and that would be XMPP or MQTT

Not wanting to derail this topic (although I think this topic has pretty much run its course on the original question), but putting my messaging protocol nerd hat on, AMQP is widely used as well. For example, I believe, most of the Xbox online gaming systems us AMQP at a shockingly large scale.

---

<div class="post-metadata">

**Author:** ![stnwll](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/stnwll/32/5240_2.png) [@stnwll](https://discourse.nodered.org/u/stnwll)\
**Post date:** [17 January 2019 16:07 UTC](https://discourse.nodered.org/t/get-mqtt-topic-data-on-demand/6572/46 "2019-01-17T16:07:18Z")

</div>

Hello gentlemen, I'm in a similar situation with a product we are attempting to introduce, please let me explain.

Remote diagnostics module (RDM) running node red attached via USB to underlying control system.  
On boot the underlying control system sends a serial number to the RDM. The RDM creates a global variable in node red. From that point on all MQTT topics will subscribe and publish using that serial number as the primary high level topic. The RDM will then dynamically add lower level topics to the primary serial number topic based on data sent over USB from the control system.

It is easy to change the publish topic dynamically to the MQTT out node, however there is no method available to change the MQTT in node subscription dynamically. It would be nice if you could allow some code inside the subscription topic window of the "MQTT in" node like this:  
**Topic** {msg.topic=global.get("system\_serial") + "/remote\_command"};

I hope the above scenario makes some sense. At present I'm stuck and searching for a solution. Any help here would be greatly appreciated.

---

<div class="post-metadata">

**Author:** ![zenofmud](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/zenofmud/32/316_2.png) [@zenofmud](https://discourse.nodered.org/u/zenofmud)\
**Post date:** [17 January 2019 16:56 UTC](https://discourse.nodered.org/t/get-mqtt-topic-data-on-demand/6572/47 "2019-01-17T16:56:00Z")

</div>

You were to add a topic to the top level (like "RDM") so you publish would be "RMD/_serial\_number_/other\_topics", then you could subscribe to "RMD/#" and get everything tat was being published. You could parse the topic to get the serial number and do what ever processing you needed.

---

<div class="post-metadata">

**Author:** ![stnwll](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/stnwll/32/5240_2.png) [@stnwll](https://discourse.nodered.org/u/stnwll)\
**Post date:** [17 January 2019 17:09 UTC](https://discourse.nodered.org/t/get-mqtt-topic-data-on-demand/6572/48 "2019-01-17T17:09:29Z")

</div>

Thank you for the reply @ zenofmud, For me doing that would be a problem as there can be thousands of devices deployed. Amazon web services charges on a per transaction basis. Communications to each device will be done using a node red application running on a service PC. I was hoping for something a little more elegant that would reduce transaction traffic through AWS to a minimum.

I tried using the node dynamicMQTT but that node offers no method for SSL/TLS security. I really need a way to dynamically set the MQTT in node. I'm a bit new at this so please excuse ignorance, and thanks again for the reply.

---

<div class="post-metadata">

**Author:** ![knolleary](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/knolleary/32/3_2.png) [@knolleary](https://discourse.nodered.org/u/knolleary)\
**Post date:** [17 January 2019 17:19 UTC](https://discourse.nodered.org/t/get-mqtt-topic-data-on-demand/6572/49 "2019-01-17T17:19:44Z")

</div>

> [@stnwll](#):
>
> I tried using the node dynamicMQTT but that node offers no method for SSL/TLS security. I really need a way to dynamically set the MQTT in node. I'm a bit new at this so please excuse ignorance, and thanks again for the reply.

Hi @stnwll. If you read back to my [earlier reply](https://discourse.nodered.org/t/get-mqtt-topic-data-on-demand/6572/26), there is a design proposal on the table to allow this functionality in the core nodes. The next stage will be for that design to be implemented. That isn't a quick task and is on the backlog along with lots of other development items.

---

<div class="post-metadata">

**Author:** ![stnwll](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/stnwll/32/5240_2.png) [@stnwll](https://discourse.nodered.org/u/stnwll)\
**Post date:** [17 January 2019 17:43 UTC](https://discourse.nodered.org/t/get-mqtt-topic-data-on-demand/6572/50 "2019-01-17T17:43:54Z")

</div>

My apology @knolleary, I scanned quickly through the posts. Am looking forward to this update, unfortunately I'm running out of time for this release and may have to way my options at taking another route. I appreciate your time and efforts, thank you.

---

<div class="post-metadata">

**Author:** ![shellian.systems](https://avatars.discourse-cdn.com/v4/letter/s/b782af/32.png) [@shellian.systems](https://discourse.nodered.org/u/shellian.systems)\
**Post date:** [17 January 2019 18:10 UTC](https://discourse.nodered.org/t/get-mqtt-topic-data-on-demand/6572/51 "2019-01-17T18:10:48Z")

</div>

Indeed, AMQP is a big contender in the messaging arena which I forgot about. My understanding, designed for rich messaging as opposed to telemetry which mqtt was built for, but just happens to be really, really good for messaging as well. True too, this topic has taken some twists and turns away from the original topic, which has been addressed with the road-map to implement the functionality into node-red, but, for me, that's what makes a discourse interesting when interacting with bright, creative, insightful individuals that can do lateral thinking 😉

---

<div class="post-metadata">

**Author:** ![dceejay](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/dceejay/32/38_2.png) [@dceejay](https://discourse.nodered.org/u/dceejay)\
**Post date:** [17 January 2019 19:00 UTC](https://discourse.nodered.org/t/get-mqtt-topic-data-on-demand/6572/52 "2019-01-17T19:00:52Z")

</div>

> On boot the underlying control system sends a serial number to the RDM. The RDM creates a global variable in node red. From that point on all MQTT topics will subscribe and publish using that serial number as the primary high level topic.

As it is set on boot you could possibly use the `$(my_env_var) ` syntax for using an environment variable to set the topic. Not it is only red at start/deploy time , and has to be the complete variable - you can't concatenate text eg you can't do `RMD/$(my_ser_number)`

---

<div class="post-metadata">

**Author:** ![stnwll](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/stnwll/32/5240_2.png) [@stnwll](https://discourse.nodered.org/u/stnwll)\
**Post date:** [17 January 2019 20:09 UTC](https://discourse.nodered.org/t/get-mqtt-topic-data-on-demand/6572/53 "2019-01-17T20:09:04Z")

</div>

Good point @dceejay, I wonder if it would be possible to write py script for serial-port to obtain serial number and insert that serial number into the environment variable before node red starts. hmmm 😕

---

<div class="post-metadata">

**Author:** ![dceejay](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/dceejay/32/38_2.png) [@dceejay](https://discourse.nodered.org/u/dceejay)\
**Post date:** [17 January 2019 20:29 UTC](https://discourse.nodered.org/t/get-mqtt-topic-data-on-demand/6572/54 "2019-01-17T20:29:26Z")

</div>

Depends where the serial number comes from and the OS. If Linux a simple bash script on boot may be enough.

---

<div class="post-metadata">

**Author:** ![cymplecy](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/cymplecy/32/2773_2.png) [@cymplecy](https://discourse.nodered.org/u/cymplecy)\
**Post date:** [17 January 2019 21:49 UTC](https://discourse.nodered.org/t/get-mqtt-topic-data-on-demand/6572/55 "2019-01-17T21:49:06Z")

</div>

Just an off-the-wall idea - run a separate external prog in python using paho lib that handles the dynamic subscription topic based on serial number and re-publishes them as as a fixed topic name that node-red can deal with

---

<div class="post-metadata">

**Author:** ![TotallyInformation](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/totallyinformation/32/31_2.png) [@TotallyInformation](https://discourse.nodered.org/u/TotallyInformation)\
**Post date:** [18 January 2019 09:55 UTC](https://discourse.nodered.org/t/get-mqtt-topic-data-on-demand/6572/56 "2019-01-18T09:55:08Z")

</div>

Not off-the-wall at all though I was going to suggest using a Node.JS app. Each to their own experience of course.

There is a tendency to forget that, because we have such an excellent tool at hand in Node-RED, that it has to be used for everything.

Especially in the professional/commercial world, it is helpful to give some thought to what alternative approaches there may be and whether they may be better. Node.JS apps can be pretty light-weight and focused and so retain easy maintenance.

---

<div class="post-metadata">

**Author:** ![stnwll](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/stnwll/32/5240_2.png) [@stnwll](https://discourse.nodered.org/u/stnwll)\
**Post date:** [18 January 2019 12:52 UTC](https://discourse.nodered.org/t/get-mqtt-topic-data-on-demand/6572/57 "2019-01-18T12:52:49Z")

</div>

@cymplecy excellent suggestion. Pyserial has access to the connected device using pyserial but I'm a little confused as to how the environment variable of node red would set the topic of the MQTT input node. It is my understanding that the change node could access the environment variable but there is no input to the MQTT "in" node, what method would be used to accomplish injecting the new topic from the environment variable?

Sorry didn't mean to hijack thread. Just grasping at straws now.

---

<div class="post-metadata">

**Author:** ![cflurin](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/cflurin/32/29_2.png) [@cflurin](https://discourse.nodered.org/u/cflurin)\
**Post date:** [18 January 2019 13:42 UTC](https://discourse.nodered.org/t/get-mqtt-topic-data-on-demand/6572/58 "2019-01-18T13:42:50Z")

</div>

> [@stnwll](#):
>
> but there is no input to the MQTT "in" node

As an alternative (until the new mqtt node will be available) see the dsm solution above [https://github.com/cflurin/node-red-contrib-dsm/wiki/Mqtt-on-demand](https://github.com/cflurin/node-red-contrib-dsm/wiki/Mqtt-on-demand)

The configuration can be modified/improved to fit your need.

---

<div class="post-metadata">

**Author:** ![dceejay](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/dceejay/32/38_2.png) [@dceejay](https://discourse.nodered.org/u/dceejay)\
**Post date:** [18 January 2019 14:11 UTC](https://discourse.nodered.org/t/get-mqtt-topic-data-on-demand/6572/59 "2019-01-18T14:11:43Z")

</div>

rather than enter a value in a field (eg [MY-TOPIC] in the MQtt topic field you can use the (env\_var) syntax e.g. [(my\_env\_for\_topic)] and then outside of Node-RED set your operating system environment variable - eg for linux `export my_env_for_topic=MY-TOPIC`. When Node-RED starts (or redploys) it will then read that environment variable and use that for the topic.

and on a Pi - you can get the serial number eg by  
cat /proc/cpuinfo | grep Serial | cut -d ' ' -f 2  
so you could do  
export MY-SER="RDM/( cat /proc/cpuinfo | grep Serial | cut -d ' ' -f 2 )" to set the environment variable then in Node-RED set the topic to (MY-SER)

This should work for just about any variable field that accepts text.

---

<div class="post-metadata">

**Author:** ![stnwll](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/stnwll/32/5240_2.png) [@stnwll](https://discourse.nodered.org/u/stnwll)\
**Post date:** [18 January 2019 16:26 UTC](https://discourse.nodered.org/t/get-mqtt-topic-data-on-demand/6572/60 "2019-01-18T16:26:58Z")

</div>

Wow @dceejay, very helpful indeed. **Thank you all for helping me with this**. I now have several options available for the short term.

[Previous page](https://discourse.nodered.org/t/get-mqtt-topic-data-on-demand/6572.md?page=2)

[Next page](https://discourse.nodered.org/t/get-mqtt-topic-data-on-demand/6572.md?page=4)