MQTT Large Buffering on Raspberry Pi 4

Jobs
1

We are in search of a skilled developer to create a Node-RED flow that buffers MQTT messages on a Raspberry Pi 4 when the MQTT-OUT node loses its internet connection.

Project Context:

  • The flow sends out MQTT messages every 10 seconds.
  • The MQTT messages contain electrical measurement data.

Project Requirements:

  1. Buffering: When the MQTT-OUT node is disconnected due to a lack of internet connectivity, all outgoing messages must be buffered to disk.
  2. Controlled Resending: Upon re-establishing the internet connection, the MQTT-OUT node should resume sending messages at a regulated pace to prevent connection overload.
  3. Data Volume: The system must be capable of buffering a large volume of messages, potentially reaching sizes up to 300MB.

Note: It maybe necessary to review the actual flow sending MQTT to consolidate the solution.

1 Like
2

I have seen a few discussions about guaranteed delivery of messages.

@Colin has a sample guaranteed delivery flow https://flows.nodered.org/flow/05e6d61f14ef6af763ec4cfd1049ab61

I am trying to document for my own use exactly how much is offered by MQTT v3 vs v5, QOS, retained connection and persistent context in Node-red.
We do seem to be in need of a best/safest practise guide.

3

The flow pointed to by @jbudd has an example of exactly what you want to do (if I understand the requirement), except for the limited publish rate on recovery. That can easily be coped with by inserting a Delay node in Rate Limit mode in front of the MQTT node to limit the rate that data can be sent.

You will have to cope with the possibility of repeated messages being received at the subscriber end, but I think you will have to do that whatever strategy you employ in node-red.

4

Thanks for the hint. Where is the node buffering?
In memory or disk?

5

It is in memory, but if you enable persistent context then it will be backed up on disc, so that the queue will survive a node-red restart or power fail.

6

That was the topic of a recent discussion: how often to flush in-memory context to disc. And relating to SD card based systems, whether the OS and the card itself actually writes the file when requested. That's more relevant to power outages than network down time though.

7

I am a freelancer and I can help you on this.
I am an IBMer and worked in Node-red for many years and even now.
Please message me directly at josekavunkal@gmail.com