Using MQTT retained messages to store state between reboots

Hi all
I am using ESPs and pubsubclient to publish to and subscribe from node-red MQTT.
I have a fairly extensive device network now running that listens for the published heartbeat and then responds by publishing local sensor data as a consequence of the callback.
These devices save their config in eeprom generally. But after too many writes, eeprom dies. So I thought I could use MQTT to publish a devices last state in a retained message sent to MQTT.
When the device wakes up again, it re-subscribes and reads back the last retained message under the state topic.
Having read around the topic, I see I need to
1, publish with a QoS >=1 and the retain flag set.
2, maybe connect with a different client id each time, that way there is little chance the broker thinks it has already published to this subscriber the retained value.

However, when the device wakes up, it doesnt receive the retained value from the last published data from this device when it subscribes, instead the subscription returns the next value I publish to the topic on receiving the first heartbeat.. So at least the subscription is working.
Any ideas if I am doing something plain wrong here ?
I am using MQTT lens in Chrome apps to view whats being published to the topic and using it to seed the topic with a state value.

What MQTT broker are you running and on what OS? Not all brokers support retention. If you are using mosquitto check the config file (on linux that is usually /etc/mosquitto/mosquitto.conf) and make sure it has in it something like

persistence true
persistence_location /var/lib/mosquitto/

I am using the built-in broker in node-red, hence why asking here.
The host OS for node.js is Windows 10.
i have tried setting the storage context to local file through settings.js but that didnt work for me. In the meantime, the broker should be persistent in memory as long as the server is running, which barring power issues, is 24x7

Neither Windows 10 nor node-red have a built-in mqtt broker

eh? Node-red has a MQTT broker that is installed as part of the default installation and available direct from the dashboard node set. I have configured the broker and a heartbeat node as one of the very first things I did on setup.

"MQTT broker. Because this MQTT broker is implemented by Node.js, you can use MQTT-in and MQTT-out nodes without an external broker like Mosquitto."
Is this not native to node-red ?

Honestly - no MQTT broker is installed as part of the default installation - are you running docker or something?

Where are you quoting from?

Not at all running docker or any containerisation - the quote is taken from the dashboard node : node-red-contrib-mqtt-broker which is mosca v0.2.5. I see that it needs to be replaced with Aedes but that slightly beside the point. The quote is taken from the node description field.
Node-red is version 0.20.7
Node js is version 12.14.1

Could you provide link to url please? :slight_smile:

FYI That is a quite an old version of Node-RED - when did you install it?

I think that does not support retained messages. Use mosquitto instead.

If I do that do I also need to find a different set of dashboard nodes to publish and subscribe ?

@skybadger Just to clarify - although the problem is quite likely to be that you are using the deprecated mosca broker provided by the contrib node node-red-contrib-broker, and that broker doesn't support retained messages as @colin has said (without a lot of other things being installed/configured)....

... that node ISN'T installed by any default installation that I/we know of

So it would be great to get a handle on how you've got to this point with Node-RED

No you do not.

I'm thinking that what @skybadger is calling "dashboard nodes" isn't what we call a dashboard ?

Th worst case would be that I have installed it from the node-red catalogue using the node-red-contrib-mqtt-broker node set. The broker installed is mosca, which is compliant with node.js up to current versions but seemingly recently not well supported, hence the recommendation to move to Aedes. I'd have to dig to discover whether retain is or isn't supported. I'd be surprised, its fairly fundamental.
My node-red has been running for the better part of two years now, running my observatory, so my belief that it came with the npm install could easily be wrong.

@cymplecy I agree, I think I am mis-labelling it here when I call them dashboard-nodes - I mean palette nodes.

It isn't confgured out-of-the-box and requires quite a bit of setting up/configuring - its great for quick and dirty stuff - not good for a proper MQTT install

This is why mosquitto is being recommended to you as that is what 99% of people use on their own networks

Also, we are generally big fans of using MQTT Explorer as our MQTT message tester/inspector

Is the recommendation then to use Mosquito as a stand-alone broker rather than Aedes as a hosted node.js broker ? Is there a guide to setting up a quality MQTT broker to interact with from node-red and third-party IOT devices ?

The mosquitto site should have full instructions on setting up - if you have any difficulties - come back here and we should be able to help you out :slight_smile:

Mosquitto is the gold standard of MQTT brokers :slight_smile:

I'll get on that and give it a go. Is there a recommended palette node for working with mosquitto from node-red ?