Archiving and synchronization of data along with time stamp with MQTT broker after failure of internet connection in IOT gateway

General
1

Dear Team,
I have a following requirement for water project.
We are getting water pump station data in IOT gateway . We want send this data to MQTT broker which is working fine. But we need to implement functionality for data archiving and synchronization with MQTT broker in MQTT client (IOT Gateway). If there is having internet connection failure/ Power loss at gateway then pump data should get logged locally in IOT gateway along with timestamp and once Internet connection is available in IOT gateway should synchronize this with broker again. IoT gateway is working on node red. Some one having standard flow could be help full.

2

What have you tried so far, how is your current flow looking and at what part did you get stuck?

3

A good first step is always to search this forum to see if the question has come up before.

Which while not giving an answer, will give you lots of things to think about...

4

One of the first things you need to think about is when you say

Will the gateway still be able to receive data if the power is down ? Is all the local network from the pump to the gateway running on battery/UPS ? etc...

1 Like
5

Hi afelix,
I can able to get PLC (S71500-Siemens) data (variableS)in to SIMATIC IOT 2040 and same variables i am able to pass to customer MQTT broker over MQTT communication.
Now i am unable to understand how to log data locally when there is no internet connection and after getting internet connection how to synchronize the logged data to MQTT Broker .

6

Hello dceejay,
Some of the cases, there is UPS supply for gateway. But main focus on if there is no internet connection to IOT then how log the data locally on gateway and once resume the internet connection then how to synchronize the logged data again MQTT broker.

7

So to recap. The IoT gateway is running Node-RED, it may be on a UPS and so it and the sending device will continue to work though the network or MQTT might be un-contactable.

So you will need to do several things, all of which I think Node-RED can do.

  • Monitor the availability of the MQTT broker - set a flag using a global or flow variable.
  • When receiving data for logging, check the flag, if it is false, start recording the data to a local store (perhaps again a variable with the filestore). You will also need to check for size since you will not want this to get too big.
  • When the flag goes back to true, you will need to empty the store as well as continue to pass current date. This is where things get difficult because you need to take into account as to whether you have to restrict your outputs to be in strict date/time order or whether that matters (depends on where that data ends up probably).

If you need everything to happen in strict date/time order then your local store needs to be a FIFO queue which is more complex to code but I expect there are examples around.

An alternative approach is to use a local database as an intermediary & then to synchronise the database rather than using MQTT as the transport. A database such as CouchDB for example is guaranteed to be "eventually" in sync across linked stores.

1 Like
8

Hello TotallyInformation,
Exactly the same i need to do it . Storing of the data base will be along with time stamp.
If you have sample readily available for the same then i will try to develop on the same basis.

9

I don't have anything ready-made I'm afraid.

If you are using a db to sync, you will need to start by looking at some database engines that will run on your gateway and that support syncing.