I currently develop a cusom node for mqtt communication according to the homie convention.
Devices using this "protocol" propagate there capabilities by a bunch of retained messages (in my setup over 450). It is the nature of mqtt as far as I know that you can't determine the sequence when the messages arrive.
I`m looking for a way to determine when all retained messages arrived after subscribing to the base topic to do a validation if there are any errors in these messages. I currently do a validation for every message arriving but this only gives me positive conformation (when i.e. a branch is configured completely). Negative confirmation is difficult as the next message (of the remaining retained messages) can be the one which is missing.
My questions:
Is there an elegant way to find out if all retained messages are received form a mqtt broker (I use msquitto)?
When you publish you have to add a timestamp yourself I believe, mqtt does not have timestamps afaik
Or use MQTT explorer, that one keeps track of timestamps
I timestamp can be a solution but the homie convention do not provide this. I read somewhere that after subscribing first ALL retained messages should be sent before a "live" message is published.
Therefore perhaps I have to subscribe to the $sys topic. But I do not know which topic is best to chose. Should be one ONLY sent after all retained messages are issued and do not have the retained flag set.
I use MQTT explorer too, great tool! But here the topics are nicely sorted even if one (the $format in example) arrived (and will be published) last of all other 450+ messages.
All is OK. But my problem is the startup process. So after "restart flow" the node subscribes to the homie topic and collects all configuration. When this is finished messages are (should be) issued to configure dashboard nodes (in this case setting the min max calues for numeric inputs like a slider) automatically according to the $format value "0:100" to ui_control.max=0 and ui_control.min=100.
But how to determine and trigger "this is finshed" ?
I am sure you have considered this possibility and rejected it, but rather than use many different topics for the data could you instead build a single JavaScript object containing all the data and publish that to a single topic?
I had the same thoughts when I first stumbled across the "homie convetion" when openHAB 2.4 was released (and the MQTT binding using it as there future standard). But digging a little bit deeper I found it so compelling that I wrote a controller & receiver plugin for the ESPEasy firmware.
This works great making my ESP8266 and ESP32 DIY devices are auto-detected in openHAB. But after other considerations I decided to switch over to Node-RED mostly because I was not willing to learn Java and digging deeper into the openHAB scripting language.
Agreeing on an open standard is always a good thing even if there are perhaps a few drawbacks. But as I used it for a long time now I more and more found that the homie convention is a well throughout standard which has the potential to grow as more and more firmware and controller systems supporting it.
I will go for the "positive validation" method collecting and validating data until the a certain dataset is complete and issue the message then. This will take more time but only happens once after restart. Errors can be detected later if the user opens the node configuration dialog like here ($type not specified, a bug in the ESPEasy firmware not sending the requited data):
I had a similar problem when I started using retained messages in my broker. My automations run when somebody restarts, which isn't always ideal! I now check the .retained flag in NodeRed and ignore messages with that flag set. Now only unretained messages cause automations to run.
if (msg.retain != 1)
{
/// Do something
}
Unless I misunderstood, it seems that would solve your problem too, no?
By the way, I do all of my automations from within Node Red. I don't currently use OpenHAB, Hassio, etc.
If I interpret that correctly then it means that you have mqtt messages which cause actions and they have been defined as Retained. A better concept is to retain the state of the system rather than actions. Then on restart the flow will be fed what the current state should be and can decide whether it needs to do anything about it, rather being fed an instruction to so something.
So there are multiple situations to account for... In one case an actual change has occurred (someone opened a door) for example. In another case, a system restarted and caused actions to be re-reported (ie. retained MQTT messages) for which other receivers of that information are unaware of the restart and recognize them instead as new actions. But how do information receivers know the difference in order to determine the actual state of the system? MQTT retained message help solve part of that issue, but (if retained improperly) can create the other problem. So we must be able to determine retained messages versus real-time messages in our automation system.
I would have thought that the answer there is to have a notification when someone "when someone restarts".
That way, you have a positive indication of state rather than trying to derive a negative indication which is very likely to fall foul of other edge-cases that you are not yet aware of.
This is why, for example, uibuilder has a control interface separate to the data output and the client tells Node-RED when it connects (actually the Node can also do that so you don't have to rely on the client doing something if you don't want to). In uibuilder's case, the typical action is to trigger the release of cached information so that the new client immediately receives it on connection.
But of course, this could also be used in a much wider context and you could control everything via MQTT messages if you wanted to.
Thank you for all your comments. The homie convention has a quite good way to solve the problems mentioned above. A parameter is published only by the device. If the parameter is „settable“ the controller(s) send new states to the parameter/set topic. The device then has or should acknowledge the new state by updating the parameter topic.
So the controller will be updated by the retained messages after restart even when changes happens during downtime. That is why I don’t save the state in Node-RED because it could be outdated.
The original question was not to filter out retained messages. The question was how to detect when all retained messages are received to perform certain tasks because there are dependencies between certain topics and the order of the messages are not guaranteed.
I found out that receiving the first message without the retained flag indicates that all retained messages arrived. But I have to wait until that happens ... so I forget detection of errors and went to positive validation triggers only valid messages, which works fine.
