Aggregate multiple async requests into a single message

General
1

Hi all,

Assuming we have a Node-RED http-in endpoint that, once called, fires multiple (eg: 3) async HTTP requests to 3rd party servers, how to I aggregate the HTTP 3 responses into a single message and therefore provide an http-out with the result?

I looked up and aggregating by time or buffer is not an option, as we can have multiple users requesting the initial HTTP endpoint. Is the AWAIT smart to do such?

Best,

2

Welcome to the forum @djsb

The easiest way is probably to run them one at a time in series. After the first one run via (presumably) http request then move the data you want to keep into a property such as msg.httpRequest1, then pass to the second http request. That node should not touch msg.httpRequest1 so then you can save the data from the second in the same way, and so on. At the end you will have all three.
If you need to do them in parallel then use three http request nodes, starting them all at the same time. Then you can use a Join node in key/value mode to combine them into one message. That can be tricky though if another request comes in to the http in node, which will trigger the three http request nodes again and it can be tricky making sure you join the messages correctly.

3

Hi Colin, thanks!

The main question here is:
- Assuming the Node-RED http-in endpoint will have multiple users concurrently accessing it
- How can we join async requests that derivate from that http-in and join them together in a way that is not time-based or store references to later join everything.

Maybe the easiest way is to actually do it inside the function in async promise or await?

4

If you do then in series as I suggested then there is no problem.

5

Yeah, makes sense. Sequential would work, but just to make it even more complex, each http-in request would generate an array of objects. For each object I need to make two async lookups. Even though I can make them sequentially, would you suggest an specific way to joining all results for a single http-out response?

http-in -> array of objects -> split -> lookup 1 -> lookup 2 -> http-out -> array of objects containing lookup 1 + lookup 2

6

Do you mean in the case of doing them in parallel?
If so it can get seriously complex. If you imagine the situation where another request comes in at the front while the lookup requests are running then that will send additional messages to the lookup nodes. If, for example, lookup 1 takes a lot longer than lookup 2 then the result from the second run of lookup 2 could appear at the end before the first from lookup 1 is completed. I think you would have to put logic around the outside so that only one transaction is in operation at a time, then provided you know the number of 'lookup' results expected you can configure a Join node to wait for that many inputs before passing them on. You could do that with node-red-contrib-semaphore for example. There can still be complications, how to handle a network issue for example that prevents the lookups from working. It would be easy to end up with a lockup situation of some sort.
Seriously, unless you cannot tolerate the lookups running one after another then do them in series. Remember that though they will run in series for any one transaction, if a new request comes in to the http in then the lookup 2 for the first request can run at the same time as the lookup 1 for the second so it should not affect the total system throughput. In fact it might give you better system throughput.

7

I think I would just do this with a function node. http-in -> function -> http-out

[{"id":"f6f2187d.f17ca8","type":"tab","label":"Flow 1","disabled":false,"info":""},{"id":"7ed6665d712a3ac0","type":"function","z":"f6f2187d.f17ca8","name":"","func":"function mockApiRequest(item) {\n  // Don't `await` here\n  // just return the Promise which will be resolved in Promise.all()\n  return Promise.resolve(item * 2);\n}\n\nconst items = [1, 2];\n\nconst promises = items.map(mockApiRequest);\nconst values = await Promise.all(promises);\n\n// Do any extra stuff here like parsing the request to get the necessary data\n\nmsg.payload = values;\nreturn msg;\n","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":320,"y":140,"wires":[["5c65d35def1b8673"]]},{"id":"09fa8885926cd707","type":"inject","z":"f6f2187d.f17ca8","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":160,"y":140,"wires":[["7ed6665d712a3ac0"]]},{"id":"5c65d35def1b8673","type":"debug","z":"f6f2187d.f17ca8","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":490,"y":140,"wires":[]}]

function mockApiRequest(item) {
  // Don't `await` here
  // just return the Promise which will be resolved in Promise.all()
  return Promise.resolve(item * 2);
}

const items = [1, 2];

const promises = items.map(mockApiRequest);
const responses = await Promise.all(promises);

// Do any extra stuff here like parsing the responses to get the data in the format you need for http-out

msg.payload = responses;
return msg;
8

Certainly it can be done in a function. In fact you could include the http-in and out too and do the whole thing in one function.

9

...feeding the responses into a DSM node, like with this example: Collecting multiple values · cflurin/node-red-contrib-dsm Wiki · GitHub may work??

10

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.