Where/how does "Delay" store data?

General
1

I have a system where data arrives in waves. There are a potentially large number of sensors that report data every few minutes.

My node-red flow grabs this data and writes it to influxdb. I'm using a delay node as a rate limiter to even out the spikes.

It's working fine at the moment.

However, I was wondering about the details of how Delay works.

Specifically, how and where does Delay store the delayed data. Is it in memory, on the disk, or ?

I'm asking because I'm running this on a Raspberry Pi 4, and I'd like to understand what might break the system as the number of sensors increases.

All feedback welcome.

2

The rate limited messages are held in memory (in a queue so to speak) until its their time to be released.

1 Like
3

If throughput is really an issue (do you know that it is?) you might be better to use the influxdb-batch node, which allows you to add a number of samples in one call, which is more efficient that many individual writes.

1 Like
4

Thank you. Very helpful.

5

Hi Colin,

Good points. I don't know for a fact that it will be an issue, I'm just trying to plan for potential issues.

I'm using the Pi with an SSD, so it is quite fast. However, when loading the same type of data via CSV, I found that pushing more than 200 writes per second caused some data loss. That's really but a complaint -- I had no idea I could get that performance from a Pi.

I will definitely look at batch mode. Any idea what the limitations of batch mode are (broadly speaking)?

I'm blown away at how robust node-red (and this whole stack) seems to be. It's come a very long way in just a couple of years.

I also very much appreciate the quick and helpful responses from you and @Steve-Mcl. Thank you both!

6

Not necessarily relevant when using influx as the db writes happen in a different process. However with that many to handle then the batch node is probably a good idea anyway.

It is impossible to answer that really. Are you talking about how fast influx can write the data or how fast you can prepare it and send it to influx? It is probably impossible to predict which one of those might be the limiting factor never mind say what the limitation might be. The only way is to try it with your hardware and see how it goes.

1 Like
7

Got it.

I was just checking for any major gotchas that might not be apparent to a noob such as myself... Just the classic "here be dragons" stuff like "this works perfectly for everything except ." I hadn't seen any documented problems with batch, other than the minor issue with timestamps that I saw discussed elsewhere. I figured I'd ask just in case.

Thanks again!

8

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.