# Get mqtt topic data on demand?

**URL:** <https://discourse.nodered.org/t/get-mqtt-topic-data-on-demand/6572>\
**Category:** General\
**Created:** [8 January 2019 18:03 UTC](https://discourse.nodered.org/t/get-mqtt-topic-data-on-demand/6572 "2019-01-08T18:03:47Z")\
**Posts on this page:** 20\
**Page:** 2

<div class="post-metadata">

**Author:** ![Colin](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/colin/32/17040_2.png) [@Colin](https://discourse.nodered.org/u/Colin)\
**Post date:** [9 January 2019 20:52 UTC](https://discourse.nodered.org/t/get-mqtt-topic-data-on-demand/6572/21 "2019-01-09T20:52:57Z")

</div>

> [@shellian.systems](#):
>
> It appears that this is boiling down to the overhead of sustaining a persistent connection. MQTT is by design a light weight protocol and it does that very well and efficiently when exchanging data. However, if there is no data to exchange for long periods of time, i.e. days/weeks, then even a keepalive pings every minute, or whatever configured, IS creating wasteful network overhead. If you multiply that by perhaps 100 connections being maintained to a server, which I can think of a very real use case in the logistics industry, and those 100 connections are over a satellite link, or GSM network, or even a RF Lora type network, well now it really matters.

Don't forget that each time it connects there will be the overheads of the connection setup plus the subscription setup plus the download of any subscribed retained topics. Presumably the system will be connecting every so often to check if there is new data. If this were, for example, every hour then that once an hour would have to be offset against the keepalive bandwidth for that hour. I have no idea what the overheads in each case would be, and it would obviously depend on whether it reconnected once an hour, every ten minutes or once a day, but without knowing what those overheads are it is impossible to say what the trade-off is.

---

<div class="post-metadata">

**Author:** ![shellian.systems](https://avatars.discourse-cdn.com/v4/letter/s/b782af/32.png) [@shellian.systems](https://discourse.nodered.org/u/shellian.systems)\
**Post date:** [9 January 2019 21:28 UTC](https://discourse.nodered.org/t/get-mqtt-topic-data-on-demand/6572/22 "2019-01-09T21:28:31Z")

</div>

Indeed @Colin, that would be true. I would suggest though that in a system that that would be needing to retrieve data regularly on a short term basis, i.e. within hours say, then keeping a persistent connection makes sense. Also, I'm not envisioning any sort of polling type scenario where we are looking for new or changed data on a frequent basis.

Imagine a third party company has some piece of data they make available to business partners via an mqtt broker. That company is in a part of the world in which network resources are constrained and expensive. That company keeps that piece of data up to date within their own processes and the changing value of that data does not concern me. The current value of that data, and my application, is only interested in that piece of data on say a once weekly, or bi-weekly basis. Rather than maintaining a persistent connection, and receiving perhaps multiple subscription updates that I don't need, it's now far more efficient for all parties to be able to dynamically make that connection to retrieve the needed data only when it's needed.

That's more along the lines of my use case thinking...

Brian

---

<div class="post-metadata">

**Author:** ![Colin](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/colin/32/17040_2.png) [@Colin](https://discourse.nodered.org/u/Colin)\
**Post date:** [9 January 2019 21:46 UTC](https://discourse.nodered.org/t/get-mqtt-topic-data-on-demand/6572/23 "2019-01-09T21:46:11Z")

</div>

> [@shellian.systems](#):
>
> That's more along the lines of my use case thinking...

That is certainly a valid use case for an MQTT node that only connects on demand.

A viable alternative might be an mqtt bridge. This allows mqtt brokers to be synchronised (on specified topics). One could run a local mqtt broker then when a check for new data was required startup the bridge (via cron or on demand or whatever), let it sync the local one with any new data, and then shut it down.

---

<div class="post-metadata">

**Author:** ![shellian.systems](https://avatars.discourse-cdn.com/v4/letter/s/b782af/32.png) [@shellian.systems](https://discourse.nodered.org/u/shellian.systems)\
**Post date:** [9 January 2019 23:07 UTC](https://discourse.nodered.org/t/get-mqtt-topic-data-on-demand/6572/24 "2019-01-09T23:07:37Z")

</div>

Aye, @Colin, that bridge suggestion, like your fine suggestion to use the Join node, would be a possible workaround. But, again, they are valiant workarounds, as @knolleary pointed out, but requiring undue flow complexity, and potential additional points of failure, if the desired functionality could be incorporated into a single, easy to use node.

I must say, I do enjoy the discourse in the forums here. Banting the ideas around do indeed make you think through a problem and validate the merit of your thinking. A true example of the power of open source community development.

Brian

---

<div class="post-metadata">

**Author:** ![shellian.systems](https://avatars.discourse-cdn.com/v4/letter/s/b782af/32.png) [@shellian.systems](https://discourse.nodered.org/u/shellian.systems)\
**Post date:** [10 January 2019 16:25 UTC](https://discourse.nodered.org/t/get-mqtt-topic-data-on-demand/6572/25 "2019-01-10T16:25:42Z")

</div>

In looking at the 10-mqtt.js code in the repository, perhaps I have a valid suggestion for a simple mod to the core MQTT-IN and MQTT-OUT nodes to achieve the objective of dynamic functionality.

(Please don't beat me up too badly as I'm not a js programmer. My world has been more C++ in embedded systems and other higher level languages.)

My read is that if an MQTT IN/OUT node is configured with a mqtt broker from the configuration node, and there is a valid topic, then the connection and subscription is made when node-red starts/deploys and the appropriate status is displayed under the node.

What if the MQTT-IN node was given an input connection and a new binary configuration option in the node called say "Dynamic Connection Mode". By default the Dynamic Connection Mode is off/false and the node behaves as it does today. But, if the user enables Dynamic Connection Mode, then on node-red startup, the node skips the connection/subscribe and sets the status to something like a black dot with "Dynamic Mode: Idle" or some such thing. When a msg arrives on the input, only then does the node proceed with the connection/subscription as defined, msg.payload is filled with any data available from the broker as usual, then if Dynamic Connection Mode is true, cleanly close the connection again.

The MQTT-OUT node already has an input (obviously), so again just adding the Dynamic Connection Mode flag to the node configuration with a default state of off. When a message arrives on the input, if Dynamic Connection Mode is enabled, then make the defined connection, publish, and cleanly disconnect and return to idle state.

I hope I'm describing the concept clearly enough. I'm suggesting that perhaps new contributed nodes are not necessary, but that the core nodes may be able to have this added without jeopardizing their integrity. Again, I'm not a js programmer but the way the code appears to be structured, it "appears" to me that checking this proposed Dynamic Connection Mode flag prior to making connections, and again to close connections would not be too difficult or risky to the existing code functionality.

Love to hear thoughts or comments about this proposal @knolleary.

Cheers,  
Brian

---

<div class="post-metadata">

**Author:** ![knolleary](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/knolleary/32/3_2.png) [@knolleary](https://discourse.nodered.org/u/knolleary)\
**Post date:** [10 January 2019 16:39 UTC](https://discourse.nodered.org/t/get-mqtt-topic-data-on-demand/6572/26 "2019-01-10T16:39:05Z")

</div>

Hi @shellian.systems

I've put together our own proposal over here: [https://github.com/node-red/node-red/wiki/Design%3A-Dynamic-MQTT-node](https://github.com/node-red/node-red/wiki/Design%3A-Dynamic-MQTT-node)

It tries to address a wider set of requirements than just your scenario.

It introduces a third node - an `mqtt-control` node that can be used to control the behaviour of the associated `mqtt-broker` node. The node has both an input and an output. Passing that node different control messages will cause the broker node to connect, disconnect, subscribe, unsubscribe and publish. The 'connect' action allows you to provide alternative broker details.

The `mqtt-broker` node will have a new option to not automatically connect (similar to what you propose). That will allow the flow to completely control when and how it connects.

You wouldn't have to use it to publish messages - you could still use an MQTT Out node configured to use the same `mqtt-broker` node. Likewise subscriptions (if the scenario allows for a hardcoded subscription as the nodes provide today).

If you need to be able to dynamically subscribe to a topic identified at runtime, the `mqtt-control` node will have an output to send any messages received due to subscriptions it has created. There's an open question on whether it would pass through all of the other flow messages it receives.

To achieve the 'connect, subscribe, get one message, disconnect' behaviour you would need a bit of flow logic around the node to pass it the appropriate control messages needed. That's my preference for version 1 of this node - want to get the basic functionality right first and gather feedback on it before we think about baking in other patterns of behaviour (if at all).

---

<div class="post-metadata">

**Author:** ![shellian.systems](https://avatars.discourse-cdn.com/v4/letter/s/b782af/32.png) [@shellian.systems](https://discourse.nodered.org/u/shellian.systems)\
**Post date:** [10 January 2019 20:31 UTC](https://discourse.nodered.org/t/get-mqtt-topic-data-on-demand/6572/27 "2019-01-10T20:31:18Z")

</div>

That certainly appears to be all encompassing @knolleary!

Obviously, it's far more elaborate and feature rich than what I was envisioning and proposing. Being able to dynamically connect/disconnect and publish/subscribe at will programatically certainly addresses any use case I can currently imagine!

I hope the development moves forward and, of course, if I can be of any assistance with testing, etc please do let me know.

As always, thank you for your time, consideration, and efforts!  
Cheers,  
Brian

---

<div class="post-metadata">

**Author:** ![TotallyInformation](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/totallyinformation/32/31_2.png) [@TotallyInformation](https://discourse.nodered.org/u/TotallyInformation)\
**Post date:** [11 January 2019 20:58 UTC](https://discourse.nodered.org/t/get-mqtt-topic-data-on-demand/6572/28 "2019-01-11T20:58:58Z")

</div>

> [@shellian.systems](#):
>
> all require another persistent MQTT connection to the second broker in question

One other method that I don't think has been mentioned.

MQTT Brokers have the capability of talking to each other. You could at least simplify the problem somewhat - though it doesn't really totally solve it - by creating a bridge from the 2nd broker so that the specific topics are replicated to the 1st. At least you'd only have 1 broker to deal with. You don't need Node-RED for that part.

* * *

Sorry, only just seen:

> [@Colin](#):
>
> A viable alternative might be an mqtt bridge. This allows mqtt brokers to be synchronised (on specified topics). One could run a local mqtt broker then when a check for new data was required startup the bridge (via cron or on demand or whatever), let it sync the local one with any new data, and then shut it down.

---

<div class="post-metadata">

**Author:** ![TotallyInformation](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/totallyinformation/32/31_2.png) [@TotallyInformation](https://discourse.nodered.org/u/TotallyInformation)\
**Post date:** [11 January 2019 21:02 UTC](https://discourse.nodered.org/t/get-mqtt-topic-data-on-demand/6572/29 "2019-01-11T21:02:09Z")

</div>

> [@shellian.systems](#):
>
> Rather than maintaining a persistent connection, and receiving perhaps multiple subscription updates that I don't need, it's now far more efficient for all parties to be able to dynamically make that connection to retrieve the needed data only when it's needed.

You are creating a potential timing issue though and would need to think very carefully about that. If the 2nd broker is on a different network for example, what happens if you get some network noise or latency that delays the connection, it could fail or be delayed long enough to miss the required data.

---

<div class="post-metadata">

**Author:** ![shellian.systems](https://avatars.discourse-cdn.com/v4/letter/s/b782af/32.png) [@shellian.systems](https://discourse.nodered.org/u/shellian.systems)\
**Post date:** [12 January 2019 12:53 UTC](https://discourse.nodered.org/t/get-mqtt-topic-data-on-demand/6572/30 "2019-01-12T12:53:59Z")

</div>

Thanks @TotallyInformation, but I was not advocating for a second broker in the mix. That was a workaround suggestion made to address a more specific use case. I'm advocating for less mqtt connections being made and held persistent for long periods of time, for no good reason, not more. If a node-red flow only connects to a broker when needed, retrieves whatever data from the topic that is made available, and then closes the connection, there is no concern about broker to broker synchronization.

Cheers,  
Brian

---

<div class="post-metadata">

**Author:** ![cflurin](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/cflurin/32/29_2.png) [@cflurin](https://discourse.nodered.org/u/cflurin)\
**Post date:** [12 January 2019 14:19 UTC](https://discourse.nodered.org/t/get-mqtt-topic-data-on-demand/6572/31 "2019-01-12T14:19:26Z")

</div>

> [@shellian.systems](#):
>
> If a node-red flow only connects to a broker when needed, retrieves whatever data from the topic that is made available, and then closes the connection

@shellian.systems

I've already used `mqtt.js` in a couple of node.js packages, so it seems to me that what you need isn't too difficult to implement.

I've a `prototype` based on the dsm node that do the job.

See: [https://github.com/cflurin/node-red-contrib-dsm/wiki/Mqtt-on-demand](https://github.com/cflurin/node-red-contrib-dsm/wiki/Mqtt-on-demand)

Note: This is just a simple solution for testing purpose. The planed core node will be of course better.

---

<div class="post-metadata">

**Author:** ![dynamicdave](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/dynamicdave/32/96_2.png) [@dynamicdave](https://discourse.nodered.org/u/dynamicdave)\
**Post date:** [12 January 2019 18:12 UTC](https://discourse.nodered.org/t/get-mqtt-topic-data-on-demand/6572/32 "2019-01-12T18:12:59Z")

</div>

This might be a bit off topic...but one of the things I've done is...

Send an MQTT message to a Wemos D1 Mini (ESP8266-based microcontroller running ESP-Easy) to trigger an event.  
Running in the Wemos are a set of rules that listen for the event.  
When the event is triggered the rule publishes the data as a json object via MQTT to Node-RED.

I've used this setup in the past to send a trigger to a plant-moisture sensor asking for the current value for soil-moisture content.

---

<div class="post-metadata">

**Author:** ![shellian.systems](https://avatars.discourse-cdn.com/v4/letter/s/b782af/32.png) [@shellian.systems](https://discourse.nodered.org/u/shellian.systems)\
**Post date:** [13 January 2019 12:52 UTC](https://discourse.nodered.org/t/get-mqtt-topic-data-on-demand/6572/33 "2019-01-13T12:52:02Z")

</div>

That's exactly the functionality @cflurin. As you say, if it can be bundled up into a nice tidy node in the palette then users don't need to be concerned with the underlying code. The beauty and power of node-red.

---

<div class="post-metadata">

**Author:** ![leifnel](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/leifnel/32/3053_2.png) [@leifnel](https://discourse.nodered.org/u/leifnel)\
**Post date:** [13 January 2019 19:07 UTC](https://discourse.nodered.org/t/get-mqtt-topic-data-on-demand/6572/34 "2019-01-13T19:07:06Z")

</div>

Not relevant if you are the receiver, but if you are the sender of very unfrequent and unfrequently used data, it ought to be possible to publish the same data on mqtt and a rest interface.

Like the topic sensors/temp/12345 should be accessible on [https://example.org/mqtt/sensors/temp/12345](https://example.org/mqtt/sensors/temp/12345)  
Similar, data could be published over mqtt or put over https:

Is there a broker with both mqtt and rest interface?

---

<div class="post-metadata">

**Author:** ![shellian.systems](https://avatars.discourse-cdn.com/v4/letter/s/b782af/32.png) [@shellian.systems](https://discourse.nodered.org/u/shellian.systems)\
**Post date:** [15 January 2019 19:02 UTC](https://discourse.nodered.org/t/get-mqtt-topic-data-on-demand/6572/35 "2019-01-15T19:02:22Z")

</div>

@leifnel, what were seeing in some industry segments is that the publishers do not want to maintain web services infrastructure, such as needed to facilitate a rest interface. They are wanting to only deploy mqtt servers for ease of setup, management, security/access management, and maintenance.  
They certainly don't want to be maintaining both an mqtt infrastructure and a web services infrastructure. But, of course, they also don't want their mqtt subscribers being persistently subscribed and using up their bandwidth/network resources unnecessarily either. Seems to be the direction things are moving so having the ability on the receiver end applications to dynamically subscribe make sense.

---

<div class="post-metadata">

**Author:** ![cymplecy](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/cymplecy/32/2773_2.png) [@cymplecy](https://discourse.nodered.org/u/cymplecy)\
**Post date:** [16 January 2019 09:47 UTC](https://discourse.nodered.org/t/get-mqtt-topic-data-on-demand/6572/36 "2019-01-16T09:47:29Z")

</div>

> [@shellian.systems](#):
>
> Seems to be the direction things are moving so having the ability on the receiver end applications to dynamically subscribe make sense

Just wondering if you've got an example of a real world application that currently does this? Not a Node-RED one of course, but a python or javascript application that does it?

---

<div class="post-metadata">

**Author:** ![TotallyInformation](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/totallyinformation/32/31_2.png) [@TotallyInformation](https://discourse.nodered.org/u/TotallyInformation)\
**Post date:** [16 January 2019 12:33 UTC](https://discourse.nodered.org/t/get-mqtt-topic-data-on-demand/6572/37 "2019-01-16T12:33:19Z")

</div>

> [@shellian.systems](#):
>
> @leifnel, what were seeing in some industry segments is that the publishers do not want to maintain web services infrastructure, such as needed to facilitate a rest interface. They are wanting to only deploy mqtt servers for ease of setup, management, security/access management, and maintenance.  
> They certainly don't want to be maintaining both an mqtt infrastructure and a web services infrastructure. But, of course, they also don't want their mqtt subscribers being persistently subscribed and using up their bandwidth/network resources unnecessarily either. Seems to be the direction things are moving so having the ability on the receiver end applications to dynamically subscribe make sense.

Sounds like more of a full Enterprise Service Bus requirement than MQTT to be honest. The whole point of MQTT is to keep things light. Though it is certainly hard to argue against the need you've identified.

---

<div class="post-metadata">

**Author:** ![shellian.systems](https://avatars.discourse-cdn.com/v4/letter/s/b782af/32.png) [@shellian.systems](https://discourse.nodered.org/u/shellian.systems)\
**Post date:** [16 January 2019 16:16 UTC](https://discourse.nodered.org/t/get-mqtt-topic-data-on-demand/6572/38 "2019-01-16T16:16:20Z")

</div>

@cymplecy, All the major mqtt libraries have been ported to pretty much every programming language, such as Python or Javascript. Within any of those languages, using simple mqtt library calls, a client app can easily dynamically connect as needed, subscribe or publish, and disconnect as required. It will be great when that can be also be easily accomplished within Node-Red 😉

---

<div class="post-metadata">

**Author:** ![shellian.systems](https://avatars.discourse-cdn.com/v4/letter/s/b782af/32.png) [@shellian.systems](https://discourse.nodered.org/u/shellian.systems)\
**Post date:** [16 January 2019 16:25 UTC](https://discourse.nodered.org/t/get-mqtt-topic-data-on-demand/6572/39 "2019-01-16T16:25:31Z")

</div>

@TotallyInformation, I would agree that if a particular environment was sufficiently large enough with a large number of participant applications, then an ESB infrastructure could be warranted. It appears though that given the high performance characteristics, and relative ease of large scale deployment, of mqtt, it has definitely become the current choice and direction of CTO's. Many of the largest social media messaging services out there are using mqtt and they certainly have scale! lol.

---

<div class="post-metadata">

**Author:** ![cymplecy](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/cymplecy/32/2773_2.png) [@cymplecy](https://discourse.nodered.org/u/cymplecy)\
**Post date:** [16 January 2019 18:15 UTC](https://discourse.nodered.org/t/get-mqtt-topic-data-on-demand/6572/40 "2019-01-16T18:15:44Z")

</div>

> [@shellian.systems](#):
>
> Many of the largest social media messaging services out there are using mqtt and they certainly have scale! lol.

That's very interesting to hear - have you got links to more info on this?

[Previous page](https://discourse.nodered.org/t/get-mqtt-topic-data-on-demand/6572.md?page=1)

[Next page](https://discourse.nodered.org/t/get-mqtt-topic-data-on-demand/6572.md?page=3)