MQTT Caching in the event of a WAN failure

General
21

I find it somewhat amusing that you are twisting the simple MQTT protocol into a full-fat message queue system :slight_smile: (that isn't a dig at you by the way, I understand your need).

It might be worth you looking at your cloud broker to see if it is really MQTT or whether (as on Azure for example) it exposes MQTT but has greater capabilities under the skin.

This is because you are really asking for a level of reliability that MQTT isn't designed to provide as far as I know. Whereas if you were to look at a more complete message queue or Enterprise Service Bus, you would find that they define guaranteed delivery and proper acknowledgement messages. Any clients for those services should therefore be able to handle lack of acks gracefully.

22

the second T in MQTT stands for Transport... it is a Transport Protocol just like HTTP (ok so not like HTTP but...)

23

Understand your thoughts and no offence taken! It started as a thought so will likely evolve! Any other protocols you have in mind which may have such capabilities included?

24

Of course these nodes may be worth looking at https://flows.nodered.org/node/node-red-contrib-mqttdb

2 Likes
25

we had the same problem, a few years ago, and the verdict was: MQTT node is not 100% reliable if you have critical data.
Even using QoS 2 and maintaining a fixed ClientID, it is possible to lose data.
Because of that we developed a Node-RED node, called safe-queue that persists and queue the messages in disk and only delete from the queue after receiving an acknowledge from the server.
The inconvenient here is that you also need Node-RED (or a specific aplication) at the server side, to generate the acknowledge message.

4 Likes
26

Interesting - what protocol does it rely on?

27

maybe I was not so clear... safe-queue was designed for queuing, whatever the protocol is.
In this case you can use MQTT together with safe-queue, for example:

image
image925×869 27.7 KB

you can find the complete example here:

[{"id":"7c2a9215.d659cc","type":"tab","label":"Device","disabled":false,"info":""},{"id":"e3a3c424.d271f8","type":"tab","label":"Server","disabled":false,"info":""},{"id":"739c47b8.b5c2a8","type":"queue config","z":"","name":"","storage":"fs","path":"queue","timeoutAck":"1000","startJob":true,"typeTimeout":"retry-times","typeError":"move-error","retryTimeout":"1000","retryError":"","maxInMemory":""},{"id":"f4797408.aeb728","type":"mqtt-broker","z":"","name":"","broker":"ipa","port":"1883","clientid":"","usetls":false,"compatmode":false,"keepalive":"60","cleansession":true,"birthTopic":"","birthQos":"0","birthPayload":"","closeTopic":"","closeQos":"0","closePayload":"","willTopic":"","willQos":"0","willPayload":""},{"id":"4d5f4ff6.2a73c","type":"ui_base","theme":{"name":"theme-light","lightTheme":{"default":"#0094CE","baseColor":"#0094CE","baseFont":"-apple-system,BlinkMacSystemFont,Segoe UI,Roboto,Oxygen-Sans,Ubuntu,Cantarell,Helvetica Neue,sans-serif","edited":true,"reset":false},"darkTheme":{"default":"#097479","baseColor":"#097479","baseFont":"-apple-system,BlinkMacSystemFont,Segoe UI,Roboto,Oxygen-Sans,Ubuntu,Cantarell,Helvetica Neue,sans-serif","edited":false},"customTheme":{"name":"Untitled Theme 1","default":"#4B7930","baseColor":"#4B7930","baseFont":"-apple-system,BlinkMacSystemFont,Segoe UI,Roboto,Oxygen-Sans,Ubuntu,Cantarell,Helvetica Neue,sans-serif"},"themeState":{"base-color":{"default":"#0094CE","value":"#0094CE","edited":false},"page-titlebar-backgroundColor":{"value":"#0094CE","edited":false},"page-backgroundColor":{"value":"#fafafa","edited":false},"page-sidebar-backgroundColor":{"value":"#ffffff","edited":false},"group-textColor":{"value":"#1bbfff","edited":false},"group-borderColor":{"value":"#ffffff","edited":false},"group-backgroundColor":{"value":"#ffffff","edited":false},"widget-textColor":{"value":"#111111","edited":false},"widget-backgroundColor":{"value":"#0094ce","edited":false},"widget-borderColor":{"value":"#ffffff","edited":false},"base-font":{"value":"-apple-system,BlinkMacSystemFont,Segoe UI,Roboto,Oxygen-Sans,Ubuntu,Cantarell,Helvetica Neue,sans-serif"}},"angularTheme":{"primary":"indigo","accents":"blue","warn":"red","background":"grey"}},"site":{"name":"Node-RED Dashboard","hideToolbar":"false","allowSwipe":"false","lockMenu":"false","allowTempTheme":"true","dateFormat":"DD/MM/YYYY","sizes":{"sx":48,"sy":48,"gx":6,"gy":6,"cx":6,"cy":6,"px":0,"py":0}}},{"id":"e2307a4f.0cc9a8","type":"inject","z":"7c2a9215.d659cc","name":"Dummy data","topic":"","payload":"{\"data\" : 12345}","payloadType":"json","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":230,"y":180,"wires":[["6179e922.bd7688"]]},{"id":"695ba1db.a955c","type":"mqtt out","z":"7c2a9215.d659cc","name":"","topic":"topic/data","qos":"2","retain":"false","broker":"f4797408.aeb728","x":660,"y":280,"wires":[]},{"id":"6179e922.bd7688","type":"queue in","z":"7c2a9215.d659cc","name":"","config":"739c47b8.b5c2a8","sendError":true,"x":640,"y":180,"wires":[[]]},{"id":"4d17c3dc.caef0c","type":"mqtt in","z":"e3a3c424.d271f8","name":"","topic":"topic/data","qos":"2","datatype":"json","broker":"f4797408.aeb728","x":140,"y":160,"wires":[["38adfccb.8777a4","b3338a0c.7e2e98"]]},{"id":"b3338a0c.7e2e98","type":"debug","z":"e3a3c424.d271f8","name":"Received message","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","x":630,"y":160,"wires":[]},{"id":"177c0028.c7ced","type":"change","z":"7c2a9215.d659cc","name":"pack message","rules":[{"t":"set","p":"payload.uuid","pt":"msg","to":"uuid","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":440,"y":280,"wires":[["695ba1db.a955c"]]},{"id":"f9ba4b66.d0e328","type":"mqtt out","z":"e3a3c424.d271f8","name":"","topic":"topic/data/ack","qos":"","retain":"","broker":"f4797408.aeb728","x":620,"y":220,"wires":[]},{"id":"38adfccb.8777a4","type":"change","z":"e3a3c424.d271f8","name":"prepare ack","rules":[{"t":"set","p":"payload","pt":"msg","to":"payload.uuid","tot":"msg"},{"t":"delete","p":"topic","pt":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":370,"y":220,"wires":[["f9ba4b66.d0e328"]]},{"id":"82735aed.e5b808","type":"inject","z":"7c2a9215.d659cc","name":"","topic":"","payload":"","payloadType":"str","repeat":"2","crontab":"","once":true,"onceDelay":0.1,"x":210,"y":660,"wires":[["751ea03e.17c7f"]]},{"id":"751ea03e.17c7f","type":"queue control","z":"7c2a9215.d659cc","name":"","config":"739c47b8.b5c2a8","operation":"queue-size","days":0,"x":500,"y":660,"wires":[[]]},{"id":"aebf7867.ddaa38","type":"queue out","z":"7c2a9215.d659cc","name":"","config":"739c47b8.b5c2a8","x":220,"y":280,"wires":[["177c0028.c7ced"]]},{"id":"9e25b10a.966c9","type":"comment","z":"7c2a9215.d659cc","name":"Queue the messages","info":"","x":220,"y":120,"wires":[]},{"id":"2a76b91.5a02846","type":"comment","z":"7c2a9215.d659cc","name":"Send message through MQTT","info":"","x":240,"y":240,"wires":[]},{"id":"ca8272d0.d78c3","type":"mqtt in","z":"7c2a9215.d659cc","name":"","topic":"topic/data/ack","qos":"2","datatype":"auto","broker":"f4797408.aeb728","x":210,"y":400,"wires":[["103ad9a4.775a36"]]},{"id":"103ad9a4.775a36","type":"change","z":"7c2a9215.d659cc","name":"unpack message","rules":[{"t":"set","p":"uuid","pt":"msg","to":"payload","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":430,"y":400,"wires":[["3e248dd2.48a752"]]},{"id":"3e248dd2.48a752","type":"queue ack","z":"7c2a9215.d659cc","name":"","config":"739c47b8.b5c2a8","x":660,"y":400,"wires":[]},{"id":"9c3e9932.6ed628","type":"comment","z":"7c2a9215.d659cc","name":"Receives the acknoledges","info":"","x":230,"y":340,"wires":[]},{"id":"891fd98b.bab878","type":"comment","z":"7c2a9215.d659cc","name":"Check queue size","info":"","x":210,"y":620,"wires":[]}]
2 Likes
28

You would need to look at what your cloud platform supports.

Some years ago, I worked on one of the largest enterprise directory and message bus projects, the NHS SPINE which at the time used a proprietary system from British Telecom to provide the bus and used HL7 messaging standards. That supported millions of transactions per second. It was eventually replaced by a more open service based I think on Mulesoft? Probably a bit overkill for most projects though :grinning:

29

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.