# MQTT to Influxdb data missing

**URL:** <https://discourse.nodered.org/t/mqtt-to-influxdb-data-missing/1618>\
**Category:** General\
**Created:** [16 July 2018 18:56 UTC](https://discourse.nodered.org/t/mqtt-to-influxdb-data-missing/1618 "2018-07-16T18:56:34Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![hsadeghilam](https://avatars.discourse-cdn.com/v4/letter/h/90ced4/32.png) [@hsadeghilam](https://discourse.nodered.org/u/hsadeghilam)\
**Post date:** [16 July 2018 18:56 UTC](https://discourse.nodered.org/t/mqtt-to-influxdb-data-missing/1618/1 "2018-07-16T18:56:34Z")

</div>

I'm using Node-Red to save some data from MQTT to Influxdb. The problem is when I look at my saved data in Influxdb I see some data points are missing. I have more data missing when I save data at high sampling rate (1 kHz) compared to low frequency (1 Hz). Any idea if this is a Node-Red issue or Influxdb issue?

---

<div class="post-metadata">

**Author:** ![shrickus](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/shrickus/32/517_2.png) [@shrickus](https://discourse.nodered.org/u/shrickus)\
**Post date:** [16 July 2018 19:01 UTC](https://discourse.nodered.org/t/mqtt-to-influxdb-data-missing/1618/2 "2018-07-16T19:01:42Z")

</div>

When you set up the influx db, what is the smallest time unit supported? If I remember correctly, it defaults to milliseconds -- so two measurements generated with the same ms timestamp will only create one record (well, the second overwrites the first).

---

<div class="post-metadata">

**Author:** ![hsadeghilam](https://avatars.discourse-cdn.com/v4/letter/h/90ced4/32.png) [@hsadeghilam](https://discourse.nodered.org/u/hsadeghilam)\
**Post date:** [16 July 2018 20:11 UTC](https://discourse.nodered.org/t/mqtt-to-influxdb-data-missing/1618/3 "2018-07-16T20:11:06Z")

</div>

The default is millisecond but I've already changed it to microsecond.

---

<div class="post-metadata">

**Author:** ![ukmoose](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/ukmoose/32/13_2.png) [@ukmoose](https://discourse.nodered.org/u/ukmoose)\
**Post date:** [16 July 2018 20:53 UTC](https://discourse.nodered.org/t/mqtt-to-influxdb-data-missing/1618/4 "2018-07-16T20:53:02Z")

</div>

What is Node-RED and Influxdb running on?

---

<div class="post-metadata">

**Author:** ![hsadeghilam](https://avatars.discourse-cdn.com/v4/letter/h/90ced4/32.png) [@hsadeghilam](https://discourse.nodered.org/u/hsadeghilam)\
**Post date:** [16 July 2018 21:13 UTC](https://discourse.nodered.org/t/mqtt-to-influxdb-data-missing/1618/5 "2018-07-16T21:13:34Z")

</div>

They are both running on my local Windows 10.

---

<div class="post-metadata">

**Author:** ![ukmoose](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/ukmoose/32/13_2.png) [@ukmoose](https://discourse.nodered.org/u/ukmoose)\
**Post date:** [16 July 2018 21:32 UTC](https://discourse.nodered.org/t/mqtt-to-influxdb-data-missing/1618/6 "2018-07-16T21:32:41Z")

</div>

So are you trying to do 1000 db writes a second in Node-RED on a desktop of laptop machine?

---

<div class="post-metadata">

**Author:** ![hsadeghilam](https://avatars.discourse-cdn.com/v4/letter/h/90ced4/32.png) [@hsadeghilam](https://discourse.nodered.org/u/hsadeghilam)\
**Post date:** [16 July 2018 21:36 UTC](https://discourse.nodered.org/t/mqtt-to-influxdb-data-missing/1618/7 "2018-07-16T21:36:53Z")

</div>

I have a sensor that publishes a reading every 1 millisecond and I'd like to save it onto Influxdb and I'm using Node-RED to set this up. Does not a normal laptop have the resources to do this? I've done the same measurement at 1 Hz frequency and I still loose some data. Although there is less data gone missing at 1 Hz!

---

<div class="post-metadata">

**Author:** ![ghayne](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/ghayne/32/39_2.png) [@ghayne](https://discourse.nodered.org/u/ghayne)\
**Post date:** [16 July 2018 21:57 UTC](https://discourse.nodered.org/t/mqtt-to-influxdb-data-missing/1618/8 "2018-07-16T21:57:45Z")

</div>

I think you will need a dedicated processor like an ESP8266 or ESP32 for this sort of sampling rate.

---

<div class="post-metadata">

**Author:** ![hsadeghilam](https://avatars.discourse-cdn.com/v4/letter/h/90ced4/32.png) [@hsadeghilam](https://discourse.nodered.org/u/hsadeghilam)\
**Post date:** [16 July 2018 22:01 UTC](https://discourse.nodered.org/t/mqtt-to-influxdb-data-missing/1618/9 "2018-07-16T22:01:33Z")

</div>

I'm actually sending my sensor data to my laptop through ESP8266 and then send the data to Node-Red. Here is my data flow: Sensor \> ESP8266 \> Node-red \> Influxdb

---

<div class="post-metadata">

**Author:** ![ghayne](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/ghayne/32/39_2.png) [@ghayne](https://discourse.nodered.org/u/ghayne)\
**Post date:** [16 July 2018 22:07 UTC](https://discourse.nodered.org/t/mqtt-to-influxdb-data-missing/1618/10 "2018-07-16T22:07:16Z")

</div>

Are you certain that the MQTT is coping with the data rate? Qos can play a big part.  
Maybe ESP8266 \> InfluxDB.  
Just some thoughts.

---

<div class="post-metadata">

**Author:** ![hsadeghilam](https://avatars.discourse-cdn.com/v4/letter/h/90ced4/32.png) [@hsadeghilam](https://discourse.nodered.org/u/hsadeghilam)\
**Post date:** [16 July 2018 22:17 UTC](https://discourse.nodered.org/t/mqtt-to-influxdb-data-missing/1618/11 "2018-07-16T22:17:29Z")

</div>

I've confirmed with two different ways that all the mqtt data come in to Node-Red. One way was by just creating a debug icon in node-red and looking at the data. So the data go missing while being transferred to influxdb.

---

<div class="post-metadata">

**Author:** ![ghayne](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/ghayne/32/39_2.png) [@ghayne](https://discourse.nodered.org/u/ghayne)\
**Post date:** [16 July 2018 22:23 UTC](https://discourse.nodered.org/t/mqtt-to-influxdb-data-missing/1618/12 "2018-07-16T22:23:26Z")

</div>

At 1kHz a lot of debug messages!! I seriously doubt that node red can cope with the data rate.

If you are simply channeling the data to InfluxDB through node red I would try sending the data directly from the ESP8266:  
[https://github.com/tobiasschuerg/ESP8266\_Influx\_DB](https://github.com/tobiasschuerg/ESP8266_Influx_DB).

I have not yet used this library so you would have to test it.

---

<div class="post-metadata">

**Author:** ![ukmoose](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/ukmoose/32/13_2.png) [@ukmoose](https://discourse.nodered.org/u/ukmoose)\
**Post date:** [17 July 2018 06:48 UTC](https://discourse.nodered.org/t/mqtt-to-influxdb-data-missing/1618/13 "2018-07-17T06:48:09Z")

</div>

you need to remember that Node-RED is single threaded

---

<div class="post-metadata">

**Author:** ![hsadeghilam](https://avatars.discourse-cdn.com/v4/letter/h/90ced4/32.png) [@hsadeghilam](https://discourse.nodered.org/u/hsadeghilam)\
**Post date:** [17 July 2018 15:31 UTC](https://discourse.nodered.org/t/mqtt-to-influxdb-data-missing/1618/14 "2018-07-17T15:31:55Z")

</div>

@ghayne ghayne: I will give the library a shot and let you know how it goes.

@ukmoose Is there an specific rate limitation that single threaded platforms can do? Does Node-red have a config file that I can change the default to multi thread?

---

<div class="post-metadata">

**Author:** ![ukmoose](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/ukmoose/32/13_2.png) [@ukmoose](https://discourse.nodered.org/u/ukmoose)\
**Post date:** [17 July 2018 15:41 UTC](https://discourse.nodered.org/t/mqtt-to-influxdb-data-missing/1618/15 "2018-07-17T15:41:28Z")

</div>

No and No as the underlying Nodejs is by design a single threaded app. If you google it you will find all the info if you wish

---

<div class="post-metadata">

**Author:** ![hsadeghilam](https://avatars.discourse-cdn.com/v4/letter/h/90ced4/32.png) [@hsadeghilam](https://discourse.nodered.org/u/hsadeghilam)\
**Post date:** [17 July 2018 15:49 UTC](https://discourse.nodered.org/t/mqtt-to-influxdb-data-missing/1618/16 "2018-07-17T15:49:08Z")

</div>

Its interesting because as I mentioned I created a debug icon in node-red and by looking at the incoming data I confirmed nothing is missing. The data go missing while being sent influxdb. Does it mean Node-red rate limitation is not in receiving data, but in sending out?

---

<div class="post-metadata">

**Author:** ![ghayne](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/ghayne/32/39_2.png) [@ghayne](https://discourse.nodered.org/u/ghayne)\
**Post date:** [17 July 2018 16:23 UTC](https://discourse.nodered.org/t/mqtt-to-influxdb-data-missing/1618/17 "2018-07-17T16:23:53Z")

</div>

Out of interest, why do you need such a high sampling rate?

---

<div class="post-metadata">

**Author:** ![Colin](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/colin/32/17040_2.png) [@Colin](https://discourse.nodered.org/u/Colin)\
**Post date:** [17 July 2018 16:29 UTC](https://discourse.nodered.org/t/mqtt-to-influxdb-data-missing/1618/18 "2018-07-17T16:29:36Z")

</div>

> [@hsadeghilam](#):
>
> The data go missing while being sent influxdb

If you are convinced that the messages are being sent to the influxdb node then it must be a problem in that area. If you examine the timestamps on the samples that _are_ getting to the db how accurate are they? Particularly look at the ones around a missing sample.

---

<div class="post-metadata">

**Author:** ![TotallyInformation](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/totallyinformation/32/31_2.png) [@TotallyInformation](https://discourse.nodered.org/u/TotallyInformation)\
**Post date:** [17 July 2018 16:32 UTC](https://discourse.nodered.org/t/mqtt-to-influxdb-data-missing/1618/19 "2018-07-17T16:32:26Z")

</div>

This is a really high data rate for high-level systems to cope with and I'm not convinced that InfluxDB can cope though I've certainly not tested it. The frequency of data is such that you may be hitting packet loss issues when delivering to InfluxDB's REST interface. I recommend checking on the InfluxDB forum or support to see if that is an issue.

As Garry says though - do you REALLY need such a crazy rate. Also how long are you trying to keep that data in InfluxDB? What output rate - in terms of reporting/charting are you really going to be using. What is it about the data that would indicate that a khz range rate would be of any use at all to a human being?

---

<div class="post-metadata">

**Author:** ![hsadeghilam](https://avatars.discourse-cdn.com/v4/letter/h/90ced4/32.png) [@hsadeghilam](https://discourse.nodered.org/u/hsadeghilam)\
**Post date:** [17 July 2018 22:46 UTC](https://discourse.nodered.org/t/mqtt-to-influxdb-data-missing/1618/20 "2018-07-17T22:46:06Z")

</div>

ghayne: I'm using a vibration sensor. Up to 10 kHz sampling rate is pretty normal while dealing with vibration. I have used sampling rate of up to 1 MHz in the past with normal Oscilloscopes.

Colin: I'm convinced the data come into node-red with no issue, I'm not sure if they are all sent out to influxdb. My debug icon is connected to incoming mqtt. The time step between missing data changes and it could be between 1 ms up to 10 ms.

TotallyInformation: For now I'm saving very small amount of data and keeping them for only a couple of hours. I'm reading the data once and I'm not making queries and writing to database simultaneously.

[Next page](https://discourse.nodered.org/t/mqtt-to-influxdb-data-missing/1618.md?page=2)