Batch HTTP requests and possible race condition

[sindre.olsen.hk]

Hi folks,

I have a flow where I query the chripstack rest api for all devices through a http in node. To do this I have to query /applications to get the total applications, then query /applications?limit= then for each application I send a new http request asking for its number of applications and then list all of them.

My problem begins when I loop through all applications to send a new http request for each. I can not guarantee that they are in order or that all requests have succeeded. Thus, if they all succeed everything is fine, but if some fail I can't join them with my join node relying on the parts property.

I want to respond to the original http request that something went wrong, but I need to make sure I do not let the other requests progress through the flow if one of the parts fail. One option is to do all the fetchin inside the function node and use async/await, but I do not really want that.

Below is a picture of the flow and the code for the requestPerApplication function node that creates the requests that causes my headache.

// Generate UUID
const randId = crypto.randomUUID();

// The number of applications
const count = parseInt(msg.numberOfApplications);

// Storing the result
const result = msg.payload.result;

// Request all the devices for each application
result.forEach((item, index) => {
    const newMsg = {
        "payload":{},
        "url": `${baseUrl}&applicationID=${item.id}`,
        "method": "GET",
        "req": msg.req,
        "res": msg.res,
        "_msgid": msg._msgid,
        "parts": { "id": randId, "index": index + 1, "count": count, "type": "array", "key": "msg.payload.result"}
    };

    // Send the request
    node.send(newMsg);
});

// All requests are sent
node.done();

[E1cid]

You do not describe your join, but if you join as object key using the the unique apllication id or name as the key you will not need to worry about order.

As to failed request you would be best to use the split node to send individual request, or add valid msg.parts in the function, you can use a switch node to check that msg.statusCode === 200, then select recreate message sequence, this will enable the join to complete. With valid msg.parts you can also rejoin automatically.

here is a simple example

[{"id":"ec923ebb20b39aef","type":"inject","z":"b9860b4b9de8c8da","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"[{\"id\":1,\"value\":2},{\"id\":2,\"value\":1},{\"id\":3,\"value\":2},{\"id\":4,\"value\":2}]","payloadType":"json","x":170,"y":2360,"wires":[["9235500d4dc9a8b7"]]},{"id":"9235500d4dc9a8b7","type":"split","z":"b9860b4b9de8c8da","name":"","splt":"\\n","spltType":"str","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"","x":290,"y":2360,"wires":[["4dba22ea179aa45d"]]},{"id":"4dba22ea179aa45d","type":"delay","z":"b9860b4b9de8c8da","name":"","pauseType":"random","timeout":"5","timeoutUnits":"seconds","rate":"1","nbRateUnits":"1","rateUnits":"second","randomFirst":"1","randomLast":"10","randomUnits":"seconds","drop":false,"allowrate":false,"outputs":1,"x":380,"y":2300,"wires":[["99880f9d1c80e917"]]},{"id":"99880f9d1c80e917","type":"switch","z":"b9860b4b9de8c8da","name":"","property":"payload.value","propertyType":"msg","rules":[{"t":"eq","v":"2","vt":"num"}],"checkall":"true","repair":true,"outputs":1,"x":450,"y":2360,"wires":[["76318943d19c7aee"]]},{"id":"76318943d19c7aee","type":"change","z":"b9860b4b9de8c8da","name":"","rules":[{"t":"set","p":"parts.key","pt":"msg","to":"payload.id","tot":"msg"},{"t":"set","p":"parts.type","pt":"msg","to":"object","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":620,"y":2360,"wires":[["19e2b4474a4be302","00afe66a756702b4"]]},{"id":"19e2b4474a4be302","type":"join","z":"b9860b4b9de8c8da","name":"","mode":"auto","build":"object","property":"payload","propertyType":"msg","key":"topic","joiner":"\\n","joinerType":"str","accumulate":"false","timeout":"","count":"","reduceRight":false,"x":490,"y":2440,"wires":[["a177262b6f990cdf"]]},{"id":"00afe66a756702b4","type":"debug","z":"b9860b4b9de8c8da","name":"debug 294","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":630,"y":2240,"wires":[]},{"id":"a177262b6f990cdf","type":"debug","z":"b9860b4b9de8c8da","name":"debug 293","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":630,"y":2440,"wires":[]}]

[sindre.olsen.hk]

I corrected the msg.parts.key to the unique id (thanks ), and it still joins the way I want. The result is that statuscode 200 on all requests yields no issues, but if one or more results in something other than 200 I need to send a http error response. However, I can not send the response before I know all requests have been proccessed, and since they are sent asynchronous I cannot rely on the index out of the total count (in msg.parts). I have tried using flow variables, but they can not be deleted and tried using a sqlite database. These solutions still don't seem to fully fix my problem. I was wondering if anyone else has experienced the same and have a better way of solving this in Node-RED, without using fetch with async/await directly in the function node.