I'm using Node-Red to save some data from MQTT to Influxdb. The problem is when I look at my saved data in Influxdb I see some data points are missing. I have more data missing when I save data at high sampling rate (1 kHz) compared to low frequency (1 Hz). Any idea if this is a Node-Red issue or Influxdb issue?
When you set up the influx db, what is the smallest time unit supported? If I remember correctly, it defaults to milliseconds -- so two measurements generated with the same ms timestamp will only create one record (well, the second overwrites the first).
The default is millisecond but I've already changed it to microsecond.
What is Node-RED and Influxdb running on?
They are both running on my local Windows 10.
So are you trying to do 1000 db writes a second in Node-RED on a desktop of laptop machine?
I have a sensor that publishes a reading every 1 millisecond and I'd like to save it onto Influxdb and I'm using Node-RED to set this up. Does not a normal laptop have the resources to do this? I've done the same measurement at 1 Hz frequency and I still loose some data. Although there is less data gone missing at 1 Hz!
I think you will need a dedicated processor like an ESP8266 or ESP32 for this sort of sampling rate.
I'm actually sending my sensor data to my laptop through ESP8266 and then send the data to Node-Red. Here is my data flow: Sensor > ESP8266 > Node-red > Influxdb
Are you certain that the MQTT is coping with the data rate? Qos can play a big part.
Maybe ESP8266 > InfluxDB.
Just some thoughts.
I've confirmed with two different ways that all the mqtt data come in to Node-Red. One way was by just creating a debug icon in node-red and looking at the data. So the data go missing while being transferred to influxdb.
At 1kHz a lot of debug messages!! I seriously doubt that node red can cope with the data rate.
If you are simply channeling the data to InfluxDB through node red I would try sending the data directly from the ESP8266:
https://github.com/tobiasschuerg/ESP8266_Influx_DB.
I have not yet used this library so you would have to test it.
you need to remember that Node-RED is single threaded
@ghayne ghayne: I will give the library a shot and let you know how it goes.
@ukmoose Is there an specific rate limitation that single threaded platforms can do? Does Node-red have a config file that I can change the default to multi thread?
No and No as the underlying Nodejs is by design a single threaded app. If you google it you will find all the info if you wish
Its interesting because as I mentioned I created a debug icon in node-red and by looking at the incoming data I confirmed nothing is missing. The data go missing while being sent influxdb. Does it mean Node-red rate limitation is not in receiving data, but in sending out?
Out of interest, why do you need such a high sampling rate?
If you are convinced that the messages are being sent to the influxdb node then it must be a problem in that area. If you examine the timestamps on the samples that are getting to the db how accurate are they? Particularly look at the ones around a missing sample.
This is a really high data rate for high-level systems to cope with and I'm not convinced that InfluxDB can cope though I've certainly not tested it. The frequency of data is such that you may be hitting packet loss issues when delivering to InfluxDB's REST interface. I recommend checking on the InfluxDB forum or support to see if that is an issue.
As Garry says though - do you REALLY need such a crazy rate. Also how long are you trying to keep that data in InfluxDB? What output rate - in terms of reporting/charting are you really going to be using. What is it about the data that would indicate that a khz range rate would be of any use at all to a human being?
1 Like
ghayne: I'm using a vibration sensor. Up to 10 kHz sampling rate is pretty normal while dealing with vibration. I have used sampling rate of up to 1 MHz in the past with normal Oscilloscopes.
Colin: I'm convinced the data come into node-red with no issue, I'm not sure if they are all sent out to influxdb. My debug icon is connected to incoming mqtt. The time step between missing data changes and it could be between 1 ms up to 10 ms.
TotallyInformation: For now I'm saving very small amount of data and keeping them for only a couple of hours. I'm reading the data once and I'm not making queries and writing to database simultaneously.
