How to set the reading speed of the CSV or File Read node?

[hkayan]

I am trying to simulate an experiment. The setup I have is like below:

The issue is that file read node works too fast for me. I need to set up this in a way that data are forwarded to a database at 20Hz.

That function 2 is just forwarding messages right now. I am brainstorming over if I can do this via function node.

I appreciate any advice.

[jbudd]

Can you overwrite the file at 20Hz?
A Watch-file node to detect the change wired to a file-in node to read the entire file (one line each time)

[Colin]

Or do you mean that the individual lines come too quickly? In which case insert a Delay node set to Rate Limit mode, with the rate set at 20 per second.
It would, however, be much more efficient to write them all at once by passing the influx node a complete array of records.

[hkayan]

Yes, that is what I mean. If I insert a delay node, it just outputs all at once with given intervals.

What I mean is that, I inserted one and set it to 50ms, then each 50ms, it outputs not 1 but X number of lines that csv node forwarded.

[hkayan]

Or do you mean that the individual lines come too quickly? In which case insert a Delay node set to Rate Limit mode, with the rate set at 20 per second.

Oh, I see now, I haven't realized there was such an option.

It would, however, be much more efficient to write them all at once by passing the influx node a complete array of records.

Could you elaborate this please?

Edit: When I use Rate Limit, Node-RED dies for some reason.

node-red.service: Main process exited, code=killed, status=9/KILL
node-red.service: Failed with result 'signal'.

[cameo69]

Hi @hkayan, I am not sure why your node-red died, but maybe your file is too large?
If you find that he delay node is not precise enough I suggest my custom node @cameo69/node-red-ratelimit. In my testing it works fairly precise even for short intervals and in my testing it handled even large queues of 1mio msg without issues.

[hkayan]

Many thanks, I will give it a try.

In the meantime, my workaround is just using a Python script with exec node.