Rate limiting based on exec node completion

The "problem" is that yes the msg.flush is added to the properties of whatever msg you have - if that is the ONLY property then the delay node will not send it on... so you need to delete the existing msg.payload and maybe msg.topic and any other properties using your change node. (Or use a function node to create a "clean" msg with only msg.flush=1 .

eg

[{"id":"31721b1737c1207d","type":"delay","z":"257090c402849288","name":"","pauseType":"rate","timeout":"5","timeoutUnits":"seconds","rate":"1","nbRateUnits":"20","rateUnits":"second","randomFirst":"1","randomLast":"5","randomUnits":"seconds","drop":false,"allowrate":false,"outputs":1,"x":430,"y":520,"wires":[["055e2529428f571a"]]},{"id":"055e2529428f571a","type":"exec","z":"257090c402849288","command":"sleep","addpay":"","append":"5","useSpawn":"false","timer":"","winHide":false,"oldrc":false,"name":"","x":630,"y":520,"wires":[["dde23da0b5477d33"],[],["d5e88b9341d02686"]]},{"id":"dde23da0b5477d33","type":"debug","z":"257090c402849288","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":810,"y":500,"wires":[]},{"id":"d5e88b9341d02686","type":"change","z":"257090c402849288","name":"","rules":[{"t":"set","p":"flush","pt":"msg","to":"1","tot":"num"},{"t":"delete","p":"payload","pt":"msg"},{"t":"delete","p":"topic","pt":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":450,"y":620,"wires":[["31721b1737c1207d","481eafc6e4f48447"]]},{"id":"481eafc6e4f48447","type":"debug","z":"257090c402849288","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":670,"y":620,"wires":[]},{"id":"7e0141d6ac754791","type":"inject","z":"257090c402849288","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payloadType":"date","x":180,"y":520,"wires":[["31721b1737c1207d"]]}]

Aah - I'll eventually get to know everything about the delay node :slight_smile: :slight_smile:
I'll give that a go and see where I get with it

Another approach that does work in the simple case of 'only one message at a time in the flow' is to use a pair of node-red-contrib-semaphore nodes around the flow. Arguably that is simpler, particularly if you don't need to retain messages in the queue over a restart.

1 Like

Maybe we could have a delete all exisiting properties option in the change node instead of having to do this?

I'm always wanting to do stuff without resorting to function nodes :slight_smile:

1 Like

Well in normal flush mode you may just want to add the msg.flush = true to flush out all the messages including the one currently being sent. (with all it's properties)

Deleting all msg properties and just sending a msg.flush=1 seems to do the job :slight_smile:

All we need for a core-node only solution is something to delete all message properties.....

I think an option to add a create blank object or remove all properties or ( insert better name here) option on the change node is worth a discussion

"clear object"

If you get a working flow using core nodes (or the q-gate node) could you post it here so I can try it please. For testing I suggest removing the exec node and the pico node and just using a fixed delay as the process to run, then we can all try it.

See @dceejay example flow for that

JFI This is my current working flow
image

The change node has 5 rules since I'm processing incoming MQTT messages, so it needs to delete quite a few msg properties and then set msg.flush to 1 before feeding it back into the delay node

It seems to be working fine for the past 30 mins

1 Like

Yes, ok, that principle does work, if you can specify a maximum time that the message processing will take. Which likely you can in this case.

well you can use the maximum time as the safety valve/timeout - in this case you could set it to some really large number as you should always get a feedback message to release the next message.

It is important to make sure that the delay time is noticeably greater than the message processing time can ever be, otherwise the flow can fail. Consider the case where there are two queued messages, with a rate of 1 per 20 seconds specified. If the process takes exactly 20 seconds then the nodes that will send the flush message get started. However, before the flush gets back to the delay node it will release the next message, then when the flush arrives it will send another message immediately, so two messages will have been sent together.

err no they won't - the flush (being blank (see other discussions)) - doesn't send it's own message - so if there is a message waiting it releases it. if not it doesn't send anything (until next message arrives)

I was not clear enough. I meant one being processed and two in the queue. Then the sequence is as follows:
Just before 20 seconds the message being processed completes and sends it message to (in @cymplecy's flow) the 'Change 5 rules' node
At 20 seconds (the flush has not got through the Change node yet) the Delay node releases its next message.
A fraction of a second later the flush gets emitted by the Change node and into the Delay node, which releases the next message.

ah right - yes - hence why I said you need to set the timeout to a really large number (that would never happen in normal circumstance - eg minutes or hours)

I don't think that flow extends to the case where the message needs to be re-tried after a timeout in the case of a failure does it?

I think we are mis-using the delay node (in rate limiter mode) for this application but in practice its very good with an easily user definable timeout if things go wrong upstream for any reason.

@colin - just noticed that you posted while I'm typing this but will press reply on this one anyway

There isn't a requirement for a retry - it's a QOS 0 type philosophy just designed not to overrun processes that take some time to complete

Your flow/node does that very well when certainty is required