Don't forget that each time it connects there will be the overheads of the connection setup plus the subscription setup plus the download of any subscribed retained topics. Presumably the system will be connecting every so often to check if there is new data. If this were, for example, every hour then that once an hour would have to be offset against the keepalive bandwidth for that hour. I have no idea what the overheads in each case would be, and it would obviously depend on whether it reconnected once an hour, every ten minutes or once a day, but without knowing what those overheads are it is impossible to say what the trade-off is.
Indeed @Colin, that would be true. I would suggest though that in a system that that would be needing to retrieve data regularly on a short term basis, i.e. within hours say, then keeping a persistent connection makes sense. Also, I'm not envisioning any sort of polling type scenario where we are looking for new or changed data on a frequent basis.
Imagine a third party company has some piece of data they make available to business partners via an mqtt broker. That company is in a part of the world in which network resources are constrained and expensive. That company keeps that piece of data up to date within their own processes and the changing value of that data does not concern me. The current value of that data, and my application, is only interested in that piece of data on say a once weekly, or bi-weekly basis. Rather than maintaining a persistent connection, and receiving perhaps multiple subscription updates that I don't need, it's now far more efficient for all parties to be able to dynamically make that connection to retrieve the needed data only when it's needed.
That's more along the lines of my use case thinking...
Brian
That is certainly a valid use case for an MQTT node that only connects on demand.
A viable alternative might be an mqtt bridge. This allows mqtt brokers to be synchronised (on specified topics). One could run a local mqtt broker then when a check for new data was required startup the bridge (via cron or on demand or whatever), let it sync the local one with any new data, and then shut it down.
Aye, @Colin, that bridge suggestion, like your fine suggestion to use the Join node, would be a possible workaround. But, again, they are valiant workarounds, as @knolleary pointed out, but requiring undue flow complexity, and potential additional points of failure, if the desired functionality could be incorporated into a single, easy to use node.
I must say, I do enjoy the discourse in the forums here. Banting the ideas around do indeed make you think through a problem and validate the merit of your thinking. A true example of the power of open source community development.
Brian
In looking at the 10-mqtt.js code in the repository, perhaps I have a valid suggestion for a simple mod to the core MQTT-IN and MQTT-OUT nodes to achieve the objective of dynamic functionality.
(Please don't beat me up too badly as I'm not a js programmer. My world has been more C++ in embedded systems and other higher level languages.)
My read is that if an MQTT IN/OUT node is configured with a mqtt broker from the configuration node, and there is a valid topic, then the connection and subscription is made when node-red starts/deploys and the appropriate status is displayed under the node.
What if the MQTT-IN node was given an input connection and a new binary configuration option in the node called say "Dynamic Connection Mode". By default the Dynamic Connection Mode is off/false and the node behaves as it does today. But, if the user enables Dynamic Connection Mode, then on node-red startup, the node skips the connection/subscribe and sets the status to something like a black dot with "Dynamic Mode: Idle" or some such thing. When a msg arrives on the input, only then does the node proceed with the connection/subscription as defined, msg.payload is filled with any data available from the broker as usual, then if Dynamic Connection Mode is true, cleanly close the connection again.
The MQTT-OUT node already has an input (obviously), so again just adding the Dynamic Connection Mode flag to the node configuration with a default state of off. When a message arrives on the input, if Dynamic Connection Mode is enabled, then make the defined connection, publish, and cleanly disconnect and return to idle state.
I hope I'm describing the concept clearly enough. I'm suggesting that perhaps new contributed nodes are not necessary, but that the core nodes may be able to have this added without jeopardizing their integrity. Again, I'm not a js programmer but the way the code appears to be structured, it "appears" to me that checking this proposed Dynamic Connection Mode flag prior to making connections, and again to close connections would not be too difficult or risky to the existing code functionality.
Love to hear thoughts or comments about this proposal @knolleary.
Cheers,
Brian
I've put together our own proposal over here: https://github.com/node-red/node-red/wiki/Design%3A-Dynamic-MQTT-node
It tries to address a wider set of requirements than just your scenario.
It introduces a third node - an mqtt-control node that can be used to control the behaviour of the associated mqtt-broker node. The node has both an input and an output. Passing that node different control messages will cause the broker node to connect, disconnect, subscribe, unsubscribe and publish. The 'connect' action allows you to provide alternative broker details.
The mqtt-broker node will have a new option to not automatically connect (similar to what you propose). That will allow the flow to completely control when and how it connects.
You wouldn't have to use it to publish messages - you could still use an MQTT Out node configured to use the same mqtt-broker node. Likewise subscriptions (if the scenario allows for a hardcoded subscription as the nodes provide today).
If you need to be able to dynamically subscribe to a topic identified at runtime, the mqtt-control node will have an output to send any messages received due to subscriptions it has created. There's an open question on whether it would pass through all of the other flow messages it receives.
To achieve the 'connect, subscribe, get one message, disconnect' behaviour you would need a bit of flow logic around the node to pass it the appropriate control messages needed. That's my preference for version 1 of this node - want to get the basic functionality right first and gather feedback on it before we think about baking in other patterns of behaviour (if at all).
2 Likes
That certainly appears to be all encompassing @knolleary!
Obviously, it's far more elaborate and feature rich than what I was envisioning and proposing. Being able to dynamically connect/disconnect and publish/subscribe at will programatically certainly addresses any use case I can currently imagine!
I hope the development moves forward and, of course, if I can be of any assistance with testing, etc please do let me know.
As always, thank you for your time, consideration, and efforts!
Cheers,
Brian
One other method that I don't think has been mentioned.
MQTT Brokers have the capability of talking to each other. You could at least simplify the problem somewhat - though it doesn't really totally solve it - by creating a bridge from the 2nd broker so that the specific topics are replicated to the 1st. At least you'd only have 1 broker to deal with. You don't need Node-RED for that part.
Sorry, only just seen:
You are creating a potential timing issue though and would need to think very carefully about that. If the 2nd broker is on a different network for example, what happens if you get some network noise or latency that delays the connection, it could fail or be delayed long enough to miss the required data.
Thanks @TotallyInformation, but I was not advocating for a second broker in the mix. That was a workaround suggestion made to address a more specific use case. I'm advocating for less mqtt connections being made and held persistent for long periods of time, for no good reason, not more. If a node-red flow only connects to a broker when needed, retrieves whatever data from the topic that is made available, and then closes the connection, there is no concern about broker to broker synchronization.
Cheers,
Brian
I've already used mqtt.js in a couple of node.js packages, so it seems to me that what you need isn't too difficult to implement.
I've a prototype based on the dsm node that do the job.
See: https://github.com/cflurin/node-red-contrib-dsm/wiki/Mqtt-on-demand
Note: This is just a simple solution for testing purpose. The planed core node will be of course better.
This might be a bit off topic...but one of the things I've done is...
Send an MQTT message to a Wemos D1 Mini (ESP8266-based microcontroller running ESP-Easy) to trigger an event.
Running in the Wemos are a set of rules that listen for the event.
When the event is triggered the rule publishes the data as a json object via MQTT to Node-RED.
I've used this setup in the past to send a trigger to a plant-moisture sensor asking for the current value for soil-moisture content.
1 Like
That's exactly the functionality @cflurin. As you say, if it can be bundled up into a nice tidy node in the palette then users don't need to be concerned with the underlying code. The beauty and power of node-red.
Not relevant if you are the receiver, but if you are the sender of very unfrequent and unfrequently used data, it ought to be possible to publish the same data on mqtt and a rest interface.
Like the topic sensors/temp/12345 should be accessible on https://example.org/mqtt/sensors/temp/12345
Similar, data could be published over mqtt or put over https:
Is there a broker with both mqtt and rest interface?
@leifnel, what were seeing in some industry segments is that the publishers do not want to maintain web services infrastructure, such as needed to facilitate a rest interface. They are wanting to only deploy mqtt servers for ease of setup, management, security/access management, and maintenance.
They certainly don't want to be maintaining both an mqtt infrastructure and a web services infrastructure. But, of course, they also don't want their mqtt subscribers being persistently subscribed and using up their bandwidth/network resources unnecessarily either. Seems to be the direction things are moving so having the ability on the receiver end applications to dynamically subscribe make sense.
Just wondering if you've got an example of a real world application that currently does this? Not a Node-RED one of course, but a python or javascript application that does it?
Sounds like more of a full Enterprise Service Bus requirement than MQTT to be honest. The whole point of MQTT is to keep things light. Though it is certainly hard to argue against the need you've identified.
@cymplecy, All the major mqtt libraries have been ported to pretty much every programming language, such as Python or Javascript. Within any of those languages, using simple mqtt library calls, a client app can easily dynamically connect as needed, subscribe or publish, and disconnect as required. It will be great when that can be also be easily accomplished within Node-Red 
@TotallyInformation, I would agree that if a particular environment was sufficiently large enough with a large number of participant applications, then an ESB infrastructure could be warranted. It appears though that given the high performance characteristics, and relative ease of large scale deployment, of mqtt, it has definitely become the current choice and direction of CTO's. Many of the largest social media messaging services out there are using mqtt and they certainly have scale! lol.
That's very interesting to hear - have you got links to more info on this?
1 Like
