Hi
I have an use case which requires persisting the MQTT messages in Broker whenever a subscriber goes offline. When the subscriber reconnects all the messages published when it was offline should be delivered.
To achieve this I am using the Node Red MQTT Out and In nodes for publish subscribe. My Node Red Version is 0.17.5 . I am using Mosquitto Broker Version 1.4.9.
For persisting messages,
At the subscriber side:
I have provided valid entries for Topic , Qos(1) ,Server, Port ,Keep Alive(60s),Client Id.Following entries are kept unchecked
1.Use Clean Session 2. Enable secure SSL/TLS Connection.
The option 'Use Legacy MQTT 3.1' support is kept checked.
At the Publisher side:
Similar settings are replicated. Client Id is automatically set to the client id of subscriber .Retain option is not chosen.
I triggered timestamp values at publisher side by keeping subscriber connected to broker in a separate flow .The subscriber was able to receive messages. Then I disabled the flow with subscriber. Did a redeploy. Triggered the time stamp messages again from publisher side. Reenabled the flow with subscriber and redeployed. The subscriber was not able to retrieve the messages published when it was disabled.
I was able to reproduce persistence using Mosquitto Pub and Sub clients with the following command line options but I am not able to achieve the same using Node Red. I could achieve the message retrieval/subscription in Node Red only when the subscriber is online at the time of publishing.
As @cymplecy MQTT does not itself support queueing of messages when the client cannot connect to the server. You would have to implement something yourself to accomplish this. You could for example push the messages into an array or sqlite db on the client as they become available and then send them to MQTT with an incrementing id or timestamp when the server becomes available. Then in the receiving client as each one is received send back a handshake saying that one has been received so the sender can drop that one off the list waiting to be sent. Obviously there are all sorts of edge cases to consider when designing this. A bit of googling might reveal that someone has designed something like this already.
Sorry Chaps, you're wrong here. MQTT does support queuing up QoS 1 and 2 messages whilst a client is offline.
connect client with Clean Session set to False
subscribe to a topic at qos 1 or 2
disconnect
(messages continue to arrive on the subscribed topic)
Reconnect with the same ID - Clean Session still set to false
subscribe to a topic
You will then receive any qos 1 or 2 messages that were published whilst offline. QoS 0 messages are not queued.
@suryas I can recreate what you describe. Having added some debug in the node's code, I can see the node is receiving the messages as expected, but it happens so quickly on startup, that the later nodes in the flow may not be ready to receive them so they don't get handled. Please can you raise an issue on github.
@suryas actually, I think the issue is a bit more subtle and related to the enable/disable of flows.
In my testing, I was starting/stopping Node-RED and I definitely see the messages arrive when I restart NR (that were published from mos_pub). When I enable/disable the tab with the subscriber on, I don't see the messages arrive. So definitely an issue, just not sure what it is yet...
@suryas I have found the root cause. When an MQTT In node is 'closed' - ie when the flows are stopped - the node is explicitly unsubscribing from the topic. This means no messages will be stored in the broker for it.
After the node was originally written, we have added the ability for the node to know if it is being closed because it has been removed, or closed because the flow is stopping. If the node is removed, then it should still unsubscribe as it won't be coming back. If the node is just being stopped, then it should not unsubscribe - that's a fix we'll apply for the next release.
However, the method you're using to test the scenario will not benefit from the fix. Disabling the flow appears to the runtime as if the node is being removed. In that case the node will still unsubscribe to tidy up the resources it used. That is the right behaviour and not one we'll change.
I am not seeing the queueing. I configured a node-red subscriber on PC B subscribed to a broker on PC A with the publisher on PC A. After checking everything was working I pulled the ethernet cable on B and waited till the LWT Offline message was sent by the broker (monitored on A). I then injected some data to the broker on A, then plugged the cable back in again. Nothing was received by B.
It is correct that if the cable is only unplugged for a short time, so that B is disconnected but still Online then the broker does queue the messages, but once B is Offline then the broker does not queue messages for it.
Hmm, no I hadn't, and now I am astonished. How long is the broker going to hang onto the messages in case the client reconnects? 10 minutes or 10 years? Is it going to hang onto hundreds, possibly thousands, of messages if it needs to?
Mosquitto, for example, will queue up 100 messages by default per client and will discard the oldest messages as new ones are queued. It also has an option to queue qos 0 (turned off by default). See the max_queued_messages, max_queued_bytes and queue_qos0_messages options in the docs - mosquitto.conf man page | Eclipse Mosquitto
Well I have been RTFM'd with full justification. Thanks, I don't remember ever noticing that.
Messages are not queued if it is the publisher that is disconnected from the broker, only if it is the subscriber that is disconnected. That doesn't surprise me.
For such requirements I write the data to an sqlite db. Then I have a separate flow that monitors the db and if there is anything then it attempts to send it. If it succeeds then it removes it from the db and does the same for any remaining records. If it can't send it then it leaves them in the db and has another go a bit later.
But in this way shipping is not guaranteed in order. I have a FIFO queue where I write 1 message. In the case that there is connection, it sent the entire queue in order. Thus, if while I am sending a new message arrives (or several) it is put at the end of the queue and sent in the correct order.
The order is guaranteed provided you take them out in the same order you put them in.
[Edit]
The database is a FIFO when used like this. For me it is important that I do not lose messages over a power down or restart which is guaranteed with a database. You could do it with persistent context storage the same way.
