Just wondering if there is a way of rate limiting message processing within a node. I know it is possible to rate limit with a delay node. This works perfectly. However it would be nice to be able to control the processing of messages from within the node that makes the API requests.
The API has a rate limit on requests per second that I am trying to avoid hitting.
For example I am spitting a JSON file into 1500 individual messages and then processing each one in a node that makes an API request via Fetch.
The problem seems to be that without a rate limit applied via a delay node the speed of message input is that great that it overwhelms the server that requests are being made to.
Requests are throttled, some false positives are thrown and sometimes the service becomes unavailable. Rather than offsetting the messages in a flow I have looked at a way of adding a sleep mechanism For example asking fetch to sleep after each request with the sleep promises library or applying a sleep via some middleware for example:
However, if rolling your own, you should simply set the custom node to drop all inbound messages unless a flag is set. Use a setInterval to control turning on the flag and turn it off after each message received. That way, all messages are dropped until the flag is turned on which allows through a single message. Set the interval to something sensible like 1000ms (on msg per second) or whatever makes sense. Expose the interval in the node's settings if you need to be able to tune it.
A lot of work though to achieve the same as the delay node.
Does your custom node have a config node that it uses? If so, that would be an ideal place to hold runtime stats (e.g. a total and/or rolling average of requests in the last hour). That way, it does not matter how many of your custom nodes are deployed across your flows -- they all use the same config node.
Of course, your custom node would have to check with the config node's stats to determine how long to wait before doing it's own work (like making an api call). Essentially you are building a feedback loop in your custom code, and including some logic similar to what the delay node already does... so I agree with the others that it's better to use the available nodes to build a custom "flow" to do all of this.
If you are concerned about limiting the number of api requests that are "in-flight" at the same time, take a look at the node-red-contrib-semaphore nodes -- start your flow with a semaphore-take to get a ticket from the pool (with configurable size), and when the api processing is done, use the semaphore-leave node to return the ticket for the next msg to use.
I've use this technique to process millions of database records/files... see this thread and the "ETL processing" discussions for more ideas.
__
Steve
That node is over 6y old and has a few short comings. Our very own @marcus-j-davies has written a newer & more feature complete version node-red-semaphore-plus. Marcus is very active here and very responsive so I would recommend anyone using semaphore nodes switch to these.
Ho hum, thanks for the responses. To me the delay node is ideal, however I am but one developer in a team that feels that there is a need for a more bespoke solution. To me why reinvent the wheel. Node-RED is designed for flow creation and the delay node handles things perfectly well.
It also encourages people to think about the sustainability of their software use.
If the core delay node works for you, then that's perfect. Stop there. The above were suggested for for stronger interlocking (a semaphore provides a means of flow control with certainty whereas delays often fall foul of some future "slowdown" or "glitch")
But if your solution works perfectly, ignore the above.
I found that async-sema resolved the issue with Fetch and being throttled by the M365 graph API. It appears if 429 codes are not respected the API just closes the door i.e. too many requests in parallel
