# Aggregate multiple async requests into a single message

**URL:** <https://discourse.nodered.org/t/aggregate-multiple-async-requests-into-a-single-message/54126>\
**Category:** General\
**Created:** [22 November 2021 17:06 UTC](https://discourse.nodered.org/t/aggregate-multiple-async-requests-into-a-single-message/54126 "2021-11-22T17:06:44Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![djsb](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/djsb/32/51445_2.png) [@djsb](https://discourse.nodered.org/u/djsb)\
**Post date:** [22 November 2021 17:06 UTC](https://discourse.nodered.org/t/aggregate-multiple-async-requests-into-a-single-message/54126/1 "2021-11-22T17:06:44Z")

</div>

Hi all,

Assuming we have a Node-RED http-in endpoint that, once called, fires multiple (eg: 3) async HTTP requests to 3rd party servers, how to I aggregate the HTTP 3 responses into a single message and therefore provide an http-out with the result?

I looked up and aggregating by time or buffer is not an option, as we can have multiple users requesting the initial HTTP endpoint. Is the AWAIT smart to do such?

Best,

---

<div class="post-metadata">

**Author:** ![Colin](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/colin/32/17040_2.png) [@Colin](https://discourse.nodered.org/u/Colin)\
**Post date:** [22 November 2021 17:29 UTC](https://discourse.nodered.org/t/aggregate-multiple-async-requests-into-a-single-message/54126/2 "2021-11-22T17:29:11Z")

</div>

Welcome to the forum @djsb

The easiest way is probably to run them one at a time in series. After the first one run via (presumably) http request then move the data you want to keep into a property such as msg.httpRequest1, then pass to the second http request. That node should not touch msg.httpRequest1 so then you can save the data from the second in the same way, and so on. At the end you will have all three.  
If you need to do them in parallel then use three http request nodes, starting them all at the same time. Then you can use a Join node in key/value mode to combine them into one message. That can be tricky though if another request comes in to the http in node, which will trigger the three http request nodes again and it can be tricky making sure you join the messages correctly.

---

<div class="post-metadata">

**Author:** ![djsb](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/djsb/32/51445_2.png) [@djsb](https://discourse.nodered.org/u/djsb)\
**Post date:** [22 November 2021 17:43 UTC](https://discourse.nodered.org/t/aggregate-multiple-async-requests-into-a-single-message/54126/3 "2021-11-22T17:43:27Z")

</div>

Hi Colin, thanks!

The main question here is:  
- Assuming the Node-RED http-in endpoint will have multiple users concurrently accessing it  
- How can we join async requests that derivate from that http-in and join them together in a way that is not time-based or store references to later join everything.

Maybe the easiest way is to actually do it inside the function in async promise or await?

---

<div class="post-metadata">

**Author:** ![Colin](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/colin/32/17040_2.png) [@Colin](https://discourse.nodered.org/u/Colin)\
**Post date:** [22 November 2021 18:05 UTC](https://discourse.nodered.org/t/aggregate-multiple-async-requests-into-a-single-message/54126/4 "2021-11-22T18:05:54Z")

</div>

If you do then in series as I suggested then there is no problem.

---

<div class="post-metadata">

**Author:** ![djsb](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/djsb/32/51445_2.png) [@djsb](https://discourse.nodered.org/u/djsb)\
**Post date:** [22 November 2021 18:22 UTC](https://discourse.nodered.org/t/aggregate-multiple-async-requests-into-a-single-message/54126/5 "2021-11-22T18:22:10Z")

</div>

Yeah, makes sense. Sequential would work, but just to make it even more complex, each http-in request would generate an array of objects. For each object I need to make two async lookups. Even though I can make them sequentially, would you suggest an specific way to joining all results for a single http-out response?

http-in -\> array of objects -\> split -\> lookup 1 -\> lookup 2 -\> http-out -\> array of objects containing lookup 1 + lookup 2

---

<div class="post-metadata">

**Author:** ![Colin](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/colin/32/17040_2.png) [@Colin](https://discourse.nodered.org/u/Colin)\
**Post date:** [22 November 2021 20:43 UTC](https://discourse.nodered.org/t/aggregate-multiple-async-requests-into-a-single-message/54126/6 "2021-11-22T20:43:15Z")

</div>

> [@djsb](#):
>
> suggest an specific way to joining all results for a single http-out response

Do you mean in the case of doing them in parallel?  
If so it can get seriously complex. If you imagine the situation where another request comes in at the front while the lookup requests are running then that will send additional messages to the lookup nodes. If, for example, lookup 1 takes a lot longer than lookup 2 then the result from the second run of lookup 2 could appear at the end before the first from lookup 1 is completed. I think you would have to put logic around the outside so that only one transaction is in operation at a time, then provided you know the number of 'lookup' results expected you can configure a Join node to wait for that many inputs before passing them on. You could do that with node-red-contrib-semaphore for example. There can still be complications, how to handle a network issue for example that prevents the lookups from working. It would be easy to end up with a lockup situation of some sort.  
Seriously, unless you cannot tolerate the lookups running one after another then do them in series. Remember that though they will run in series for any one transaction, if a new request comes in to the http in then the lookup 2 for the first request can run at the same time as the lookup 1 for the second so it should not affect the total system throughput. In fact it might give you better system throughput.

---

<div class="post-metadata">

**Author:** ![dczysz](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/dczysz/32/46158_2.png) [@dczysz](https://discourse.nodered.org/u/dczysz)\
**Post date:** [22 November 2021 21:24 UTC](https://discourse.nodered.org/t/aggregate-multiple-async-requests-into-a-single-message/54126/7 "2021-11-22T21:24:47Z")

</div>

I think I would just do this with a function node. http-in -\> function -\> http-out

```auto
[{"id":"f6f2187d.f17ca8","type":"tab","label":"Flow 1","disabled":false,"info":""},{"id":"7ed6665d712a3ac0","type":"function","z":"f6f2187d.f17ca8","name":"","func":"function mockApiRequest(item) {\n // Don't `await` here\n // just return the Promise which will be resolved in Promise.all()\n return Promise.resolve(item * 2);\n}\n\nconst items = [1, 2];\n\nconst promises = items.map(mockApiRequest);\nconst values = await Promise.all(promises);\n\n// Do any extra stuff here like parsing the request to get the necessary data\n\nmsg.payload = values;\nreturn msg;\n","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":320,"y":140,"wires":[["5c65d35def1b8673"]]},{"id":"09fa8885926cd707","type":"inject","z":"f6f2187d.f17ca8","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":160,"y":140,"wires":[["7ed6665d712a3ac0"]]},{"id":"5c65d35def1b8673","type":"debug","z":"f6f2187d.f17ca8","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":490,"y":140,"wires":[]}]

```

```auto
function mockApiRequest(item) {
  // Don't `await` here
  // just return the Promise which will be resolved in Promise.all()
  return Promise.resolve(item * 2);
}

const items = [1, 2];

const promises = items.map(mockApiRequest);
const responses = await Promise.all(promises);

// Do any extra stuff here like parsing the responses to get the data in the format you need for http-out

msg.payload = responses;
return msg;

```

---

<div class="post-metadata">

**Author:** ![Colin](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/colin/32/17040_2.png) [@Colin](https://discourse.nodered.org/u/Colin)\
**Post date:** [22 November 2021 21:38 UTC](https://discourse.nodered.org/t/aggregate-multiple-async-requests-into-a-single-message/54126/8 "2021-11-22T21:38:06Z")

</div>

Certainly it can be done in a function. In fact you could include the http-in and out too and do the whole thing in one function.

---

<div class="post-metadata">

**Author:** ![hominidae](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/hominidae/32/4174_2.png) [@hominidae](https://discourse.nodered.org/u/hominidae)\
**Post date:** [22 November 2021 22:06 UTC](https://discourse.nodered.org/t/aggregate-multiple-async-requests-into-a-single-message/54126/9 "2021-11-22T22:06:27Z")

</div>

...feeding the responses into a DSM node, like with this example: [Collecting multiple values · cflurin/node-red-contrib-dsm Wiki · GitHub](https://github.com/cflurin/node-red-contrib-dsm/wiki/Collecting-multiple-values) may work??

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/1X/d073cd938eafa2e558d7c2cd59003b3ef4963033.png) [@system](https://discourse.nodered.org/u/system)\
**Post date:** [21 January 2022 22:06 UTC](https://discourse.nodered.org/t/aggregate-multiple-async-requests-into-a-single-message/54126/10 "2022-01-21T22:06:44Z")

</div>

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.