MQTT Caching in the event of a WAN failure

[TotallyInformation]

@joearkay

I could dig deeper into the TCP mechanism behind MQTT, however I believe that the 'subscribe to the same topic as publishing and implement queue around this' is an elegant solution and one to look into.

I find it somewhat amusing that you are twisting the simple MQTT protocol into a full-fat message queue system (that isn't a dig at you by the way, I understand your need).

It might be worth you looking at your cloud broker to see if it is really MQTT or whether (as on Azure for example) it exposes MQTT but has greater capabilities under the skin.

This is because you are really asking for a level of reliability that MQTT isn't designed to provide as far as I know. Whereas if you were to look at a more complete message queue or Enterprise Service Bus, you would find that they define guaranteed delivery and proper acknowledgement messages. Any clients for those services should therefore be able to handle lack of acks gracefully.

[dceejay]

the second T in MQTT stands for Transport... it is a Transport Protocol just like HTTP (ok so not like HTTP but...)

[joearkay]

Understand your thoughts and no offence taken! It started as a thought so will likely evolve! Any other protocols you have in mind which may have such capabilities included?

[dceejay]

Of course these nodes may be worth looking at https://flows.nodered.org/node/node-red-contrib-mqttdb

[machadotiago]

we had the same problem, a few years ago, and the verdict was: MQTT node is not 100% reliable if you have critical data.
Even using QoS 2 and maintaining a fixed ClientID, it is possible to lose data.
Because of that we developed a Node-RED node, called safe-queue that persists and queue the messages in disk and only delete from the queue after receiving an acknowledge from the server.
The inconvenient here is that you also need Node-RED (or a specific aplication) at the server side, to generate the acknowledge message.

[joearkay]

Interesting - what protocol does it rely on?

[machadotiago]

maybe I was not so clear... safe-queue was designed for queuing, whatever the protocol is.
In this case you can use MQTT together with safe-queue, for example:

you can find the complete example here:

[{"id":"7c2a9215.d659cc","type":"tab","label":"Device","disabled":false,"info":""},{"id":"e3a3c424.d271f8","type":"tab","label":"Server","disabled":false,"info":""},{"id":"739c47b8.b5c2a8","type":"queue config","z":"","name":"","storage":"fs","path":"queue","timeoutAck":"1000","startJob":true,"typeTimeout":"retry-times","typeError":"move-error","retryTimeout":"1000","retryError":"","maxInMemory":""},{"id":"f4797408.aeb728","type":"mqtt-broker","z":"","name":"","broker":"ipa","port":"1883","clientid":"","usetls":false,"compatmode":false,"keepalive":"60","cleansession":true,"birthTopic":"","birthQos":"0","birthPayload":"","closeTopic":"","closeQos":"0","closePayload":"","willTopic":"","willQos":"0","willPayload":""},{"id":"4d5f4ff6.2a73c","type":"ui_base","theme":{"name":"theme-light","lightTheme":{"default":"#0094CE","baseColor":"#0094CE","baseFont":"-apple-system,BlinkMacSystemFont,Segoe UI,Roboto,Oxygen-Sans,Ubuntu,Cantarell,Helvetica Neue,sans-serif","edited":true,"reset":false},"darkTheme":{"default":"#097479","baseColor":"#097479","baseFont":"-apple-system,BlinkMacSystemFont,Segoe UI,Roboto,Oxygen-Sans,Ubuntu,Cantarell,Helvetica Neue,sans-serif","edited":false},"customTheme":{"name":"Untitled Theme 1","default":"#4B7930","baseColor":"#4B7930","baseFont":"-apple-system,BlinkMacSystemFont,Segoe UI,Roboto,Oxygen-Sans,Ubuntu,Cantarell,Helvetica Neue,sans-serif"},"themeState":{"base-color":{"default":"#0094CE","value":"#0094CE","edited":false},"page-titlebar-backgroundColor":{"value":"#0094CE","edited":false},"page-backgroundColor":{"value":"#fafafa","edited":false},"page-sidebar-backgroundColor":{"value":"#ffffff","edited":false},"group-textColor":{"value":"#1bbfff","edited":false},"group-borderColor":{"value":"#ffffff","edited":false},"group-backgroundColor":{"value":"#ffffff","edited":false},"widget-textColor":{"value":"#111111","edited":false},"widget-backgroundColor":{"value":"#0094ce","edited":false},"widget-borderColor":{"value":"#ffffff","edited":false},"base-font":{"value":"-apple-system,BlinkMacSystemFont,Segoe UI,Roboto,Oxygen-Sans,Ubuntu,Cantarell,Helvetica Neue,sans-serif"}},"angularTheme":{"primary":"indigo","accents":"blue","warn":"red","background":"grey"}},"site":{"name":"Node-RED Dashboard","hideToolbar":"false","allowSwipe":"false","lockMenu":"false","allowTempTheme":"true","dateFormat":"DD/MM/YYYY","sizes":{"sx":48,"sy":48,"gx":6,"gy":6,"cx":6,"cy":6,"px":0,"py":0}}},{"id":"e2307a4f.0cc9a8","type":"inject","z":"7c2a9215.d659cc","name":"Dummy data","topic":"","payload":"{\"data\" : 12345}","payloadType":"json","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":230,"y":180,"wires":[["6179e922.bd7688"]]},{"id":"695ba1db.a955c","type":"mqtt out","z":"7c2a9215.d659cc","name":"","topic":"topic/data","qos":"2","retain":"false","broker":"f4797408.aeb728","x":660,"y":280,"wires":[]},{"id":"6179e922.bd7688","type":"queue in","z":"7c2a9215.d659cc","name":"","config":"739c47b8.b5c2a8","sendError":true,"x":640,"y":180,"wires":[[]]},{"id":"4d17c3dc.caef0c","type":"mqtt in","z":"e3a3c424.d271f8","name":"","topic":"topic/data","qos":"2","datatype":"json","broker":"f4797408.aeb728","x":140,"y":160,"wires":[["38adfccb.8777a4","b3338a0c.7e2e98"]]},{"id":"b3338a0c.7e2e98","type":"debug","z":"e3a3c424.d271f8","name":"Received message","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","x":630,"y":160,"wires":[]},{"id":"177c0028.c7ced","type":"change","z":"7c2a9215.d659cc","name":"pack message","rules":[{"t":"set","p":"payload.uuid","pt":"msg","to":"uuid","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":440,"y":280,"wires":[["695ba1db.a955c"]]},{"id":"f9ba4b66.d0e328","type":"mqtt out","z":"e3a3c424.d271f8","name":"","topic":"topic/data/ack","qos":"","retain":"","broker":"f4797408.aeb728","x":620,"y":220,"wires":[]},{"id":"38adfccb.8777a4","type":"change","z":"e3a3c424.d271f8","name":"prepare ack","rules":[{"t":"set","p":"payload","pt":"msg","to":"payload.uuid","tot":"msg"},{"t":"delete","p":"topic","pt":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":370,"y":220,"wires":[["f9ba4b66.d0e328"]]},{"id":"82735aed.e5b808","type":"inject","z":"7c2a9215.d659cc","name":"","topic":"","payload":"","payloadType":"str","repeat":"2","crontab":"","once":true,"onceDelay":0.1,"x":210,"y":660,"wires":[["751ea03e.17c7f"]]},{"id":"751ea03e.17c7f","type":"queue control","z":"7c2a9215.d659cc","name":"","config":"739c47b8.b5c2a8","operation":"queue-size","days":0,"x":500,"y":660,"wires":[[]]},{"id":"aebf7867.ddaa38","type":"queue out","z":"7c2a9215.d659cc","name":"","config":"739c47b8.b5c2a8","x":220,"y":280,"wires":[["177c0028.c7ced"]]},{"id":"9e25b10a.966c9","type":"comment","z":"7c2a9215.d659cc","name":"Queue the messages","info":"","x":220,"y":120,"wires":[]},{"id":"2a76b91.5a02846","type":"comment","z":"7c2a9215.d659cc","name":"Send message through MQTT","info":"","x":240,"y":240,"wires":[]},{"id":"ca8272d0.d78c3","type":"mqtt in","z":"7c2a9215.d659cc","name":"","topic":"topic/data/ack","qos":"2","datatype":"auto","broker":"f4797408.aeb728","x":210,"y":400,"wires":[["103ad9a4.775a36"]]},{"id":"103ad9a4.775a36","type":"change","z":"7c2a9215.d659cc","name":"unpack message","rules":[{"t":"set","p":"uuid","pt":"msg","to":"payload","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":430,"y":400,"wires":[["3e248dd2.48a752"]]},{"id":"3e248dd2.48a752","type":"queue ack","z":"7c2a9215.d659cc","name":"","config":"739c47b8.b5c2a8","x":660,"y":400,"wires":[]},{"id":"9c3e9932.6ed628","type":"comment","z":"7c2a9215.d659cc","name":"Receives the acknoledges","info":"","x":230,"y":340,"wires":[]},{"id":"891fd98b.bab878","type":"comment","z":"7c2a9215.d659cc","name":"Check queue size","info":"","x":210,"y":620,"wires":[]}]

[TotallyInformation]

@joearkay

Any other protocols you have in mind which may have such capabilities included?

You would need to look at what your cloud platform supports.

Some years ago, I worked on one of the largest enterprise directory and message bus projects, the NHS SPINE which at the time used a proprietary system from British Telecom to provide the bus and used HL7 messaging standards. That supported millions of transactions per second. It was eventually replaced by a more open service based I think on Mulesoft? Probably a bit overkill for most projects though