MQTT to Influxdb data missing

[dceejay]

One possible test may be to replace the database node with a simple file out node (for a short while which would let yo see if its a specific node issue or something wider.

Of course we don't know if its the implementation in that node, or the underlying library, or the connection , or back end yet..

[TotallyInformation]

@hsadeghilam

TotallyInformation: For now I'm saving very small amount of data and keeping them for only a couple of hours. I'm reading the data once and I'm not making queries and writing to database simultaneously.

OK, so it sounds like you aren't really making use of the power of InfluxDB and its features? If not, as Dave say's, I'd first try an alternative, SQLite could probably handle it, especially if your PC has sufficient memory to keep it all in RAM without paging. Without knowing the data and how you want to use it, it's hard to say in detail.

[hsadeghilam]

@ghayne I used ESP8266_Influx library and I was able to write data into influxdb without any loss! I will have to dig into it a bit deeper though! This shows Influxdb can support high sampling rate so the issue is probably Node-red.

dceejay: I will try your suggestion and let you know.

TotallyInformation: I'm not familiar with SQLite. Is this something that I can replace with Node-red in my data flow? I'm currently using mqtt.

[TotallyInformation]

@hsadeghilam

TotallyInformation: I'm not familiar with SQLite. Is this something that I can replace with Node-red in my data flow? I'm currently using mqtt.

It is a mainly in-memory SQL database library with write to disk. You use it as you would MySQL or any other SQL database. Typically, SQL databases are much better at fast writes than nosql type databases. InfluxDB should be an exception to that rule according to the vendors but then SQLite is so small and well written, I think it would be really hard to beat for fast sequential writes.

There is a SQLlite node. You would need to know some SQL but it isn't too hard to learn and is fairly similar to the QL used in InfluxDB.

@ghayne thanks for the reference to the ESP libary for InfluxDB, I wasn't aware of that. Though, in hindsight, it should be fairly simple since IDB uses REST for all queries and updates.

[clickworkorange]

If you need to store data in InfluxDB with that kind of resolution, why not batch the values, say 1000 messages at a time, to a list object (with timestamps if you want) and push that to InfluxDB either as a batch insert or an array of some sort?

(Sorry for the necro; I just felt this needed saying.)