it's very hard for me to ask this question because I understand that's a really complex scenario.
But... let's try.
I've got a split & delay with rate limit that sends every 10sec. a message.
For each message I need (and I can't do that in a different way) to call an API, then do a little process and so on.
My problem is that, obviously, API call is asyncronous and so I've got a different amount of time at response.. Sometimes this thime is bigger tha 10 seconds, so the the next message is passed.
That's not a problem, if not I'll need to be sure tha every single message is processed.
So i checked msg.parts to check when the loop arrives to the end but, for the previous reason, it could happen that a message is not processed "in time" and the flow pass over to next parts, without doing the correct elaboration.
So my question is: how to check at every iteration if ALL MESSAGES are be processed?
Do you mean that, ideally, you want to not release the next message until the previous one has completed? If so then the current beta version of node red (v3.0 beta 4) has a fix to the delay node that allows you to feedback when the flow has completed to release the next one from the delay node. See Wait for payload to finish flow before passing another payload - #33 by Colin for a flow that works in the beta version (it didn't at the time of the post). Set the delay node to rate limit with a large timeout so it only releases messages if a message gets lost. Otherwise the next one will be released as soon as the previous one says it is complete. Note that the message sent back must be an empty message except for msg.flush.
It's complicated to explain, but I need to call an API to read some ata related to the message and then call others two APIs that wrote foundamentally write changes on a database (I can't change the backend logic, if not it was okay).
So I need that every message calls the 3 APIs before "exit" the loop and continue with the rest of the flow.
So what is wrong with using the delay node as I suggested, or semaphore nodes to achieve the same thing? If it is that you want to run the three APIs in parallel to increase the throughput then put a delay node loop around each one, which will stop messages overtaking each other as they go through.
Unfortunately it is not simple (maybe not possible) to use that node for this purpose, even though it was intended to be used in that way. I was unable to construct a simple flow using it that did not suffer from problems due to race conditions, when messages arrived at the front at inopportune moments.
There isn't a problem with the node, in the sense that it does do what the docs say it should do.
The problem using it for the situation here relates, if I remember correctly, to handling the situation where a new message comes in the front just after the previous one has been has been completed. At that stage I couldn't see an obvious enhancement to the node that would solve the problem.
Aah - from my quick reading it seemed there was queue management available within the node - i assumed this would allow you to tell the queue when processing had finished and to release the next queued message.
@drmibell - could you comment on this as Colin has a lot more knowledge than me and this appears to be your baby ?