Out of order http responses

Hello

I have a problem of possible out-of-order processing in my flow
The flow goes through an http-request, which I imagine is processed asynchronously. This introduce an uncertainty on the execution order.
On this example, we can see that the order of the messages at the output are not the same as the input. They can eventually be different at each execution.

[{"id":"9a86d59e88442f2c","type":"tab","label":"Flow 1","disabled":false,"info":"","env":[]},{"id":"9342a516f1bafda0","type":"inject","z":"9a86d59e88442f2c","name":"","props":[{"p":"payload"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"[{\"title\":\"ITEM A\"},{\"title\":\"ITEM B\"},{\"title\":\"ITEM C\"},{\"title\":\"ITEM D\"},{\"title\":\"ITEM E\"},{\"title\":\"ITEM F\"},{\"title\":\"ITEM G\"},{\"title\":\"ITEM H\"}]","payloadType":"json","x":430,"y":220,"wires":[["377fb634af59c3fa"]]},{"id":"fc962674bf19f1e9","type":"debug","z":"9a86d59e88442f2c","name":"Debug 1","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"\"Debug 1: title:\" & msg.save_details.title","targetType":"jsonata","statusVal":"","statusType":"auto","x":1020,"y":160,"wires":[]},{"id":"377fb634af59c3fa","type":"split","z":"9a86d59e88442f2c","name":"","splt":"\\n","spltType":"str","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"","x":550,"y":220,"wires":[["b7a7fc71bc9eb7cc"]]},{"id":"9e139c96772f9582","type":"http request","z":"9a86d59e88442f2c","name":"","method":"GET","ret":"txt","paytoqs":"ignore","url":"https://google.com","tls":"","persist":false,"proxy":"","authType":"","senderr":false,"x":1050,"y":220,"wires":[["c651b52621a1c3d4"]]},{"id":"b7a7fc71bc9eb7cc","type":"change","z":"9a86d59e88442f2c","name":"save details","rules":[{"t":"set","p":"save_details","pt":"msg","to":"payload","tot":"msg","dc":true}],"action":"","property":"","from":"","to":"","reg":false,"x":870,"y":220,"wires":[["fc962674bf19f1e9","9e139c96772f9582"]]},{"id":"c651b52621a1c3d4","type":"debug","z":"9a86d59e88442f2c","name":"Debug 2","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"\"Debug 2: title:\" & msg.save_details.title","targetType":"jsonata","statusVal":"","statusType":"auto","x":1220,"y":220,"wires":[]}] 

Unfortuntately in that flow, I need to be sure that the http-request are executed at the server side in the right order.

So far the only workaround I found is to add a rate limiter before the http-request not to insure the flow is completed before the next request is performed.

[{"id":"9a86d59e88442f2c","type":"tab","label":"Flow 1","disabled":false,"info":"","env":[]},{"id":"9342a516f1bafda0","type":"inject","z":"9a86d59e88442f2c","name":"","props":[{"p":"payload"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"[{\"title\":\"ITEM A\"},{\"title\":\"ITEM B\"},{\"title\":\"ITEM C\"},{\"title\":\"ITEM D\"},{\"title\":\"ITEM E\"},{\"title\":\"ITEM F\"},{\"title\":\"ITEM G\"},{\"title\":\"ITEM H\"}]","payloadType":"json","x":430,"y":220,"wires":[["377fb634af59c3fa"]]},{"id":"fc962674bf19f1e9","type":"debug","z":"9a86d59e88442f2c","name":"Debug 1","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"\"Debug 1: title:\" & msg.save_details.title","targetType":"jsonata","statusVal":"","statusType":"auto","x":1020,"y":160,"wires":[]},{"id":"377fb634af59c3fa","type":"split","z":"9a86d59e88442f2c","name":"","splt":"\\n","spltType":"str","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"","x":550,"y":220,"wires":[["7b21bb9c815f61ef"]]},{"id":"9e139c96772f9582","type":"http request","z":"9a86d59e88442f2c","name":"","method":"GET","ret":"txt","paytoqs":"ignore","url":"https://google.com","tls":"","persist":false,"proxy":"","authType":"","senderr":false,"x":1050,"y":220,"wires":[["c651b52621a1c3d4"]]},{"id":"b7a7fc71bc9eb7cc","type":"change","z":"9a86d59e88442f2c","name":"save details","rules":[{"t":"set","p":"save_details","pt":"msg","to":"payload","tot":"msg","dc":true}],"action":"","property":"","from":"","to":"","reg":false,"x":870,"y":220,"wires":[["fc962674bf19f1e9","9e139c96772f9582"]]},{"id":"c651b52621a1c3d4","type":"debug","z":"9a86d59e88442f2c","name":"Debug 2","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"\"Debug 2: title:\" & msg.save_details.title","targetType":"jsonata","statusVal":"","statusType":"auto","x":1220,"y":220,"wires":[]},{"id":"7b21bb9c815f61ef","type":"delay","z":"9a86d59e88442f2c","name":"","pauseType":"rate","timeout":"5","timeoutUnits":"seconds","rate":"1","nbRateUnits":"2","rateUnits":"second","randomFirst":"1","randomLast":"5","randomUnits":"seconds","drop":false,"allowrate":false,"outputs":1,"x":700,"y":220,"wires":[["b7a7fc71bc9eb7cc"]]}]

This has problems as if I want it to be relliable, I need a sufficient delay which is drastically slowing down my flow.
Is there any better solution to insure that the http-request is performed in an "atomic" way ?

Thanks

Yes - you can use the delay node in rate limit mode - set the rate to max you will allow, and let them queue up.... when you get a successful response create a feedback with ONLY a msg.flush set to 1 to release the next message.

Do they need to be executed in the correct order, or do you just need them returned in correct order?

@dceejay
Thanks, working like a charm
I did as below with a function node with only return {'flush': 1}; and it works fine
I first tried with a change node to set msg.flush but it started an endless loop probably because the payload and other fields were still there.
Is there another way than a function node to build a message with only msg.flush = 1 ?

@E1cid
I need the request to be executed in the same order on the server.
As I don't want to assume anything about the quieing on the server, I prefer to get the response before sending the next.
It may be possible that the server queue and execute in order, and that only the responses are out of order but I need to be 100% sure so the above solution fullfill my need.
But happy to hear about another one if you have. Always good to learn a trick :smiley:

1 Like

I have a solution to have the returns in correct order, but it would only be of use to you if you did not care about what order they are executed. If the order they are executed does not matter, you could then send all request and not have any delay, other than the time it takes for longest request response.

Ok
So not a solution for my today need :smile:
I'll keep it in mind
Thanks