# Archiving and synchronization of data along with time stamp with MQTT broker after failure of internet connection in IOT gateway

**URL:** <https://discourse.nodered.org/t/archiving-and-synchronization-of-data-along-with-time-stamp-with-mqtt-broker-after-failure-of-internet-connection-in-iot-gateway/12770>\
**Category:** General\
**Created:** [1 July 2019 06:22 UTC](https://discourse.nodered.org/t/archiving-and-synchronization-of-data-along-with-time-stamp-with-mqtt-broker-after-failure-of-internet-connection-in-iot-gateway/12770 "2019-07-01T06:22:43Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![DJIOT](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/djiot/32/9778_2.png) [@DJIOT](https://discourse.nodered.org/u/DJIOT)\
**Post date:** [1 July 2019 06:22 UTC](https://discourse.nodered.org/t/archiving-and-synchronization-of-data-along-with-time-stamp-with-mqtt-broker-after-failure-of-internet-connection-in-iot-gateway/12770/1 "2019-07-01T06:22:43Z")

</div>

Dear Team,  
I have a following requirement for water project.  
We are getting water pump station data in IOT gateway . We want send this data to MQTT broker which is working fine. But we need to implement functionality for data archiving and synchronization with MQTT broker in MQTT client (IOT Gateway). If there is having internet connection failure/ Power loss at gateway then pump data should get logged locally in IOT gateway along with timestamp and once Internet connection is available in IOT gateway should synchronize this with broker again. IoT gateway is working on node red. Some one having standard flow could be help full.

---

<div class="post-metadata">

**Author:** ![afelix](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/afelix/32/9743_2.png) [@afelix](https://discourse.nodered.org/u/afelix)\
**Post date:** [1 July 2019 06:31 UTC](https://discourse.nodered.org/t/archiving-and-synchronization-of-data-along-with-time-stamp-with-mqtt-broker-after-failure-of-internet-connection-in-iot-gateway/12770/2 "2019-07-01T06:31:10Z")

</div>

What have you tried so far, how is your current flow looking and at what part did you get stuck?

---

<div class="post-metadata">

**Author:** ![ukmoose](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/ukmoose/32/13_2.png) [@ukmoose](https://discourse.nodered.org/u/ukmoose)\
**Post date:** [1 July 2019 07:22 UTC](https://discourse.nodered.org/t/archiving-and-synchronization-of-data-along-with-time-stamp-with-mqtt-broker-after-failure-of-internet-connection-in-iot-gateway/12770/3 "2019-07-01T07:22:02Z")

</div>

A good first step is always to search this forum to see if the question has come up before.

> [@MQTT - Storing data when disconnected](https://discourse.nodered.org/t/mqtt-storing-data-when-disconnected/9306/9):
>
> So the Node-RED mqtt node has made qos 1 and 2 redundant by having a connection check around publish. The underlying mqtt client lib already manages messages correctly based on qos. In fact the mqtt client can be configured with a storage policy. The default policy is es6-map. If Node-RED is running in a constrained environment where qos\>0 can not be supported. Then I would expect qos to be set to 0 in that environment.

Which while not giving an answer, will give you lots of things to think about...

---

<div class="post-metadata">

**Author:** ![dceejay](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/dceejay/32/38_2.png) [@dceejay](https://discourse.nodered.org/u/dceejay)\
**Post date:** [1 July 2019 07:24 UTC](https://discourse.nodered.org/t/archiving-and-synchronization-of-data-along-with-time-stamp-with-mqtt-broker-after-failure-of-internet-connection-in-iot-gateway/12770/4 "2019-07-01T07:24:22Z")

</div>

One of the first things you need to think about is when you say

> [@DJIOT](#):
>
> If there is having internet connection failure/ Power loss at gateway then pump data should get logged locally in IOT gateway

Will the gateway still be able to receive data if the power is down ? Is all the local network from the pump to the gateway running on battery/UPS ? etc...

---

<div class="post-metadata">

**Author:** ![DJIOT](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/djiot/32/9778_2.png) [@DJIOT](https://discourse.nodered.org/u/DJIOT)\
**Post date:** [1 July 2019 07:58 UTC](https://discourse.nodered.org/t/archiving-and-synchronization-of-data-along-with-time-stamp-with-mqtt-broker-after-failure-of-internet-connection-in-iot-gateway/12770/5 "2019-07-01T07:58:20Z")

</div>

Hi afelix,  
I can able to get PLC (S71500-Siemens) data (variableS)in to SIMATIC IOT 2040 and same variables i am able to pass to customer MQTT broker over MQTT communication.  
Now i am unable to understand how to log data locally when there is no internet connection and after getting internet connection how to synchronize the logged data to MQTT Broker .

---

<div class="post-metadata">

**Author:** ![DJIOT](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/djiot/32/9778_2.png) [@DJIOT](https://discourse.nodered.org/u/DJIOT)\
**Post date:** [1 July 2019 08:00 UTC](https://discourse.nodered.org/t/archiving-and-synchronization-of-data-along-with-time-stamp-with-mqtt-broker-after-failure-of-internet-connection-in-iot-gateway/12770/6 "2019-07-01T08:00:44Z")

</div>

Hello dceejay,  
Some of the cases, there is UPS supply for gateway. But main focus on if there is no internet connection to IOT then how log the data locally on gateway and once resume the internet connection then how to synchronize the logged data again MQTT broker.

---

<div class="post-metadata">

**Author:** ![TotallyInformation](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/totallyinformation/32/31_2.png) [@TotallyInformation](https://discourse.nodered.org/u/TotallyInformation)\
**Post date:** [1 July 2019 08:32 UTC](https://discourse.nodered.org/t/archiving-and-synchronization-of-data-along-with-time-stamp-with-mqtt-broker-after-failure-of-internet-connection-in-iot-gateway/12770/7 "2019-07-01T08:32:03Z")

</div>

So to recap. The IoT gateway is running Node-RED, it may be on a UPS and so it and the sending device will continue to work though the network or MQTT might be un-contactable.

So you will need to do several things, all of which I think Node-RED can do.

- Monitor the availability of the MQTT broker - set a flag using a global or flow variable.
- When receiving data for logging, check the flag, if it is false, start recording the data to a local store (perhaps again a variable with the filestore). You will also need to check for size since you will not want this to get too big.
- When the flag goes back to true, you will need to empty the store as well as continue to pass current date. This is where things get difficult because you need to take into account as to whether you have to restrict your outputs to be in strict date/time order or whether that matters (depends on where that data ends up probably).

If you need everything to happen in strict date/time order then your local store needs to be a FIFO queue which is more complex to code but I expect there are examples around.

An alternative approach is to use a local database as an intermediary & then to synchronise the database rather than using MQTT as the transport. A database such as CouchDB for example is guaranteed to be "eventually" in sync across linked stores.

---

<div class="post-metadata">

**Author:** ![DJIOT](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/djiot/32/9778_2.png) [@DJIOT](https://discourse.nodered.org/u/DJIOT)\
**Post date:** [1 July 2019 09:11 UTC](https://discourse.nodered.org/t/archiving-and-synchronization-of-data-along-with-time-stamp-with-mqtt-broker-after-failure-of-internet-connection-in-iot-gateway/12770/8 "2019-07-01T09:11:34Z")

</div>

Hello TotallyInformation,  
Exactly the same i need to do it . Storing of the data base will be along with time stamp.  
If you have sample readily available for the same then i will try to develop on the same basis.

---

<div class="post-metadata">

**Author:** ![TotallyInformation](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/totallyinformation/32/31_2.png) [@TotallyInformation](https://discourse.nodered.org/u/TotallyInformation)\
**Post date:** [1 July 2019 09:25 UTC](https://discourse.nodered.org/t/archiving-and-synchronization-of-data-along-with-time-stamp-with-mqtt-broker-after-failure-of-internet-connection-in-iot-gateway/12770/9 "2019-07-01T09:25:27Z")

</div>

I don't have anything ready-made I'm afraid.

If you are using a db to sync, you will need to start by looking at some database engines that will run on your gateway and that support syncing.