Get mqtt topic data on demand?

General
1

Hi Folks,
I've searched the forums and I'm not seeing an obvious solution to my issue and wondering if anyone can help out...

I have a use case in which I will receive an mqtt event from one topic that I have subscribed to and that triggers a flow just fine. The problem is that when that flow triggers I need to obtain data that is stored retained in another mqtt topic "on demand" to do additional decision making and processing, etc. I can't figure out how to dynamically subscribe to a topic, get the data stored there, then disconnect again. Does the mqtt in node have any sort of enable/disable functionality? Or is there another better method to achieve this?

Thanks in advance.
Brian

2

I would think that you could have a MQTT-in node continuously getting DATA2 and you store it in a Flow or Global variable. That way when topic one comes in, you can immediately look at the Flow/Global variable to make the decision.

3

Thanks @zenofmud but (there is always a but), lol, the second data source publishes very, very infrequently...days or weeks apart even and I'm not in administrative control of that data source. I would have to restart my node-red to initialize the mqtt-in node to do the initial subscribe to get the data to populate a global variable. In a production environment we don't want to be having to restart node-red just to ensure we are synced with the second data source.

Hmmm, maybe adding a enable/disable (connect/disconnect) feature to the mqtt-in node is a feature request to pursue...

Thanks
Brian

4

The core MQTT nodes don't support dynamically setting the topic, but there are contributed nodes available that do. A quick search for MQTT on flows.nodered.org finds https://flows.nodered.org/node/node-red-contrib-mqtt-dynamictopic for example.

5

Hi Nick, @knolleary,

It's like you know a thing or two about Node-Red :wink:

Thanks for popping in on this and your suggestion. I did look at the contributed node that you highlighted but it has several dependency issues right now, doesn't seem to function as described, and the author does not appear to be maintaining it or responding to issues. The same seems to be true for the other couple contributed nodes available that highlight "dynamic" topic functionality.

I'll keep hunting for a workable solution. I just thought that if I have this use case, perhaps others do as well, and given there have been efforts at contributed nodes, perhaps it could be a future feature of the core nodes.

Cheers,
Brian

6

I would do this using a Join node. Feed two MQTT In nodes subscribed to the two topics and configure it like this

image
image.png525Ă—548 26.6 KB

On deploy or restart this will get given the current values of the topics (assuming they are Retained Values) and will pass on a payload object with the two values under the topics as keys. Then any time either of the two gets a new value then another message will be passed with the latest values of both. So all you have to do is pass these on to whatever you want to do when a new value is received.

7

Brian, you can enable file based contest stores (https://nodered.org/docs/user-guide/context#context-stores) so if you hav a MQTT node looking for just that topic, every time it changes you will get the new value which you put into the global. Now when the other data value comes in, you can compare the two.

If you are worried about the scond source changing just when the first comes in, you could use Colins idea of a join, just add a slight delay after receiving the first data packet.

8

These are all valiant workarounds, but it is far from ideal to subscribe to everything just in case you may need it. Kinda defeats the lightweight nature of MQTT, and you don't necessarily know the topic up front to subscribe anyway.

So yes, this has come up enough times we should look at it properly.

1 Like
9

Nick,
He wouldn't have to subscribe to everything( Hopefully). I would hope that the second data would be published with a topic that would isolate it so it would only be recieved when it was published.

But maybe the second topic is not publishing it's data and if that's the case, I'd look into seeing if it could.

10

Thank you gentlemen! Ask and you shall receive :wink:
@Colin, your suggestion works like a treat and does indeed solve the immediate need, so, again, many thanks for that.

As Nick, @knolleary, does highlight, it doesn't make sense to me either to maintain a persistent connection 7/24 to the second broker when it's data is only needed very infrequently. I don't believe my use case is all that unusual the more and more that disparate organizations are integrating their data/control systems with each other for technical and/or business back-end systems purposes. Node-Red is a VERY practical platform to design and carry out those inter-business functions. I personally do see value in having the core MQTT-IN node be able to be used to dynamically connect/subscribe to a broker and topic through a triggering event, get available data, then cleanly close the connection again.

If I had the coding skills necessary myself I would certainly give a go to creating a contribution to that end but, alas, that is not my deep skill set. Good looks and charm are my highest level skills right now :wink:

Thanks again guys!
Brian

11

I'm not sure this needs to be a mode of the existing MQTT-IN node. It's a bit specialised and needs some extra thought re some of the other semantics. I think it may make more sense as a separate node (like we have http-request node as well as in and out)

MQTT is publish/subscribe and not request/response so the pattern of connecting to get data and disconnect while workable is a bit not business as usual. Should it also allow staying connected until the next topic comes along ? or x pieces of data ? or multiple topics ? unsubscribe from specific topics as well ? what about connecting to dynamic brokers as well ?

2 Likes
12

From my understanding of your issue, you have a second MQTT publisher (or broker?) that sends messages rarely that you want to reliably pick up in a flow. There has been some discussion about the need to dynamically set the MQTT topic, but I can't see you've actually articulated this requirement, so I'm going to assume it's not needed.

So my solution would be to subscribe to the second publisher's messages, and when they arrive, republish them (probably with a different topic) and set retain=true. Your flow would then subscribe to the republished messages for your current logic rather than the original ones. If/when your Node Red restarts, it will be resent these republished messages and can act accordingly.

13

Perhaps if I can't contribute much to coding, I could at least suggest some design considerations and better articulate use cases as @michaelblight suggests I may not have done adequately.

One of the premises of node-red, at least from my perspective, is the simplicity of flow designs to achieve the desired results. If/when we have to combine multiple nodes together, into a complex series, to achieve a simple end result function, that suggests to me that there is a need for a new node to encompass that series of functions for simplification. As has been suggested here, there are methods to achieve what I needed, in this case, but all require a series of nodes and configurations, and all require another persistent MQTT connection to the second broker in question, which, as Nick pointed out is indeed wasteful and contrary to the design nature of MQTT, if the connection is only needed very infrequently.

In my professional experience, there are more and more organizations deploying MQTT brokers to make shared, real time data available to business partners, etc. It's much easier and cost effective for business to deploy, secure, and maintain MQTT brokers than web servers with proprietary API's, etc. If not many use cases yet, there will be many more in the future to come as more enterprises turn to MQTT for business-to-business integrations. To be ready for that, I believe that node-red needs a node that, when triggered, allows a flow to dynamically connect to any defined broker (local or cloud), subscribe to a topic, thus retrieving any retained data in that topic, stuff it into the payload, and then disconnect..all within a single, easy to use node.

I have just one or two use cases for this in my situation, but if given a simple to implement capability in node-red, it's easy to foresee designers and integrators suggesting to their business clients that a better way to implement B2B inter-communications is with MQTT, then similar use cases will come fast and furious, in my opinion.

I think I would agree with @dceejay in that perhaps this should be a separate node from the core MQTT-IN to allow for additional functionality as he suggested, and to not "muddy" the clean simple functionality of the existing MQTT-IN node. That said, does it need to be any more feature rich than the existing MQTT-IN but with the added functionality to create the connection/subscription, get the retained data, and disconnect? That seems to me to meet the objective I see but this is the power of discussion...

Cheers,
Brian

14

I think the key from @dceejay is the second paragraph. Already this is sounding like a complex node, which in my view is bad (the trigger node makes me think too hard!). And it would only work for retained messages. And it doesn't save you much - just subscribe and save the payload locally (eg. join node or context variable); then use it when you need it. And I don't like being dependent on a 3rd party marking their message as retained on a broker I don't control. FWIW.

And on another note, does the join solution work for you in all cases? If a new message is sent by the rare publisher, it will trigger your flow, using the previous message from your main subscribed topic which has already been 'processed'.

15

I think this is the key statement

What you are wanting to do is not what MQTT was/is designed for. Subscribing is not just for Christmas :slight_smile:

Having said that, there has been a requirement for a while to have another type of MQTT in node that is triggerable and it would be great to have it

16

Why? The overheads are trivial. I believe that when nothing relevant is happening on the connection then all that happens is an occasional keep alive message. This will have no measurable effects on anything.

17

I don't believe that is what @knolleary meant. I believe that he thought you intended to subscribe to all topics on the broker, which is not the case.
In addition I think that if you imagine that there were an mqtt node that could be enabled and disabled by a trigger I think that you might well find the solution more complex than that using a permanent connection and the Join node, as you would need some sort or triggering system to enable it and later disable it, and you would still need the Join node or some similar mechanism to combine the value with that from the other topic.

18

@colin - trivial maybe - but do cost something either in terms of memory, power, or actual $$$ (if say over a mobile phone link)

19

True, but if saving any of those was important then I suspect that closing the mqtt connection would be unlikely be the best way of saving something.

20

It appears that this is boiling down to the overhead of sustaining a persistent connection. MQTT is by design a light weight protocol and it does that very well and efficiently when exchanging data. However, if there is no data to exchange for long periods of time, i.e. days/weeks, then even a keepalive pings every minute, or whatever configured, IS creating wasteful network overhead. If you multiply that by perhaps 100 connections being maintained to a server, which I can think of a very real use case in the logistics industry, and those 100 connections are over a satellite link, or GSM network, or even a RF Lora type network, well now it really matters. In the global B2B domain, there remains significant network infrastructure in which ALL network traffic has a accumulating cost, and those costs can be significant at scale.

Indeed, MQTT is a publish/subscribe model. That doesn't necessarily mean that the publishing and subscribing functions must, or will, occur in real time. Hence why the retain flag exists at all. MQTT has been designed from the ground up with the idea of unreliable network connections. Again, this is why we have retain, qos, and lwt abilities.

In considering more use case scenarios, I think now there is also the need for not only an on demand, dynamic MQTT-IN type node that does the connect/subscribe/disconnect, but also an MQTT-OUT node to allow for event driven connect/publish/disconnect. Perhaps that can be accomplished within a single node structure (which I think has merit), but, again, I'm not a heavy duty coder so I can't really comment on the software engineering here.

I think node-red is a very, very important platform towards pushing the adoption of MQTT across multiple types of business domains. But, as MQTT is adopted, if node-red makes it too challenging or awkward or convoluted to perform a simple task, or doesn't provide enough flexibility to adopt to wide, unpredictable use cases, then designers will find other solutions.

It already has a beautifully, simplistic method to drop in a subscribe or publish node that seems to work flawlessly, but those create persistent connections and only connect on a reload/restart which is a no-go in a production, high availability deployment. Both of those are problematic in many use cases so, again, I'm just suggesting there is merit and value in having these nodes be able to be dynamically controlled via events.

Cheers,
Brian