# Can node-red process a large http (json) response as a stream?

**URL:** <https://discourse.nodered.org/t/can-node-red-process-a-large-http-json-response-as-a-stream/73787>\
**Category:** General\
**Tags:** http-request\
**Created:** [17 January 2023 02:01 UTC](https://discourse.nodered.org/t/can-node-red-process-a-large-http-json-response-as-a-stream/73787 "2023-01-17T02:01:27Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![darrencruse](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/darrencruse/32/9568_2.png) [@darrencruse](https://discourse.nodered.org/u/darrencruse)\
**Post date:** [17 January 2023 02:01 UTC](https://discourse.nodered.org/t/can-node-red-process-a-large-http-json-response-as-a-stream/73787/1 "2023-01-17T02:01:27Z")

</div>

For using node-red for combining/transforming multiple json microservice apis, some of whose responses can be very large in the 10s or occasionally even 100s of megabytes:

Is there any way to not have the http request node have to read the entire response before sending it on to the next node?

i.e. is there some way it can submit the request and then start streaming in the response and (here I'm assuming the response is a json array) split the array elements sending those individually to the next node, _while the response is still streaming in_?

I'm not sure if it helps but googling in the node.js world I often see references to the "[JSONStream](https://www.npmjs.com/package/JSONStream)" package as a popular one for this kind of thing...

Thanks,

Darren

---

<div class="post-metadata">

**Author:** ![bakman2](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/bakman2/32/6207_2.png) [@bakman2](https://discourse.nodered.org/u/bakman2)\
**Post date:** [17 January 2023 06:35 UTC](https://discourse.nodered.org/t/can-node-red-process-a-large-http-json-response-as-a-stream/73787/2 "2023-01-17T06:35:40Z")

</div>

> [@darrencruse](#):
>
> Is there any way to not have the http request node have to read the entire response before sending it on to the next node?

That is not possible. Do the microservices you are "talking" to have other API connection options (other than REST), like websockets, kafka, MQ etc ? These are made for streaming responses. Or does the REST API have options/parameters to offset the data ?

---

<div class="post-metadata">

**Author:** ![TotallyInformation](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/totallyinformation/32/31_2.png) [@TotallyInformation](https://discourse.nodered.org/u/TotallyInformation)\
**Post date:** [17 January 2023 14:23 UTC](https://discourse.nodered.org/t/can-node-red-process-a-large-http-json-response-as-a-stream/73787/3 "2023-01-17T14:23:22Z")

</div>

As bakman2 says, that isn't possible with the core http-request node. However, it should be possible if you wanted to get your hands dirty.

You would need to import some suitable libraries that would allow the request along with a streamed response with the stream creating chunked output messages.

Have you waded through the 6 pages of things that mention "stream" on Flows? [Library - Node-RED](https://flows.nodered.org/search?term=stream)

Alternatively, a roll-your-own approach might use something like [node-fetch - npm (npmjs.com)](https://www.npmjs.com/package/node-fetch) imported to Node-RED.

---

<div class="post-metadata">

**Author:** ![darrencruse](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/darrencruse/32/9568_2.png) [@darrencruse](https://discourse.nodered.org/u/darrencruse)\
**Post date:** [17 January 2023 14:33 UTC](https://discourse.nodered.org/t/can-node-red-process-a-large-http-json-response-as-a-stream/73787/4 "2023-01-17T14:33:33Z")

</div>

Thanks @bakman2

And no these are just normal REST apis though yeah some (not all) do have pagination options for asking for a specified number of records at an offset.

For those supporting pagination I guess a flow could hit the api repeatedly and split the array that comes back for each page to achieve something similar to what I was thinking?

Though after processing the array elements I will need to join the results back together in the end for my final result.

I should have said I've played a bit with node-red a couple years ago I'm not a total beginner but no expert btw. But I do node.js programming for my day job so I could write function tasks or even a custom node if need be.

We process these json apis I'm referring to using node.js code already but we are having performance challenges there too. That code doesn't use a streaming/dataflow style if I could achieve what I'd described in node-red I'd hoped to see how it's performance compared to our code (though I wish node-red also had some kind of web worker/multi-threaded support it still doesn't right).

fwiw here's one of the articles that made me think theoretically what I described should be possible even in node-red (though I haven't tried maybe I'm missing something):

> **[Parsing And Serializing Large Objects Using JSONStream In Node.js](https://www.bennadel.com/blog/3232-parsing-and-serializing-large-objects-using-jsonstream-in-node-js.htm)**
>
> Ben Nadel experiments with JSONStream - an npm module that allows for massive objects to be serialized and deserialized as JavaScript Object Notation (JSON) by using Node.js streams to incrementally t

Anyway thanks again.

---

<div class="post-metadata">

**Author:** ![bakman2](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/bakman2/32/6207_2.png) [@bakman2](https://discourse.nodered.org/u/bakman2)\
**Post date:** [17 January 2023 14:43 UTC](https://discourse.nodered.org/t/can-node-red-process-a-large-http-json-response-as-a-stream/73787/5 "2023-01-17T14:43:06Z")

</div>

> [@darrencruse](#):
>
> For those supporting pagination I guess a flow could hit the api repeatedly and split the array that comes back for each page to achieve something similar to what I was thinking?

Indeed, it is the best and fastest option.

> fwiw here's one of the articles that made me think theoretically what I described should be possible even in node-red (though I haven't tried maybe I'm missing something)

This is just a dirty workaround, writing/reading while the data is coming in. It is kind of weird to have a RESTful API that delivers such huge loads of data, it hurts the performance of the server too i can imagine.

---

<div class="post-metadata">

**Author:** ![darrencruse](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/darrencruse/32/9568_2.png) [@darrencruse](https://discourse.nodered.org/u/darrencruse)\
**Post date:** [17 January 2023 14:46 UTC](https://discourse.nodered.org/t/can-node-red-process-a-large-http-json-response-as-a-stream/73787/6 "2023-01-17T14:46:07Z")

</div>

Thanks @TotallyInformation I was writing my reply to @bakman2 then saw yours

I'm pretty new/rusty on node-red (I really only played with it a couple years ago) and no I didn't know to look at those "stream" matches in [flows](https://flows.nodered.org/search?term=stream)

But I will that's a gr8 idea thanks

---

<div class="post-metadata">

**Author:** ![TotallyInformation](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/totallyinformation/32/31_2.png) [@TotallyInformation](https://discourse.nodered.org/u/TotallyInformation)\
**Post date:** [17 January 2023 17:31 UTC](https://discourse.nodered.org/t/can-node-red-process-a-large-http-json-response-as-a-stream/73787/7 "2023-01-17T17:31:05Z")

</div>

No problem. For those with pagination - which is typical for things like Graph and OData API's, the presence of the paging link is usually sufficient and you can simply put in a link back to the input of your request with the new URL substituted.

> [@bakman2](#):
>
> > fwiw here's one of the articles that made me think theoretically what I described should be possible even in node-red (though I haven't tried maybe I'm missing something)
> 
> This is just a dirty workaround, writing/reading while the data is coming in. It is kind of weird to have a RESTful API that delivers such huge loads of data, it hurts the performance of the server too i can imagine.

There are now many API's that are capable of returning huge datasets. I was playing with the Microsoft Graph API just a few days ago, our Azure Active Directory is "only" less than a 100,000 entries but obviously that is still a whacking great set of data. But not that unusual to need to fetch all of the user entries to do analysis. The paging is the only thing that allows this to be feasible since the back-end processing is also paged so that the servers aren't overwhelmed.

Just remember that you might need to put in a delay node to prevent you from spamming the server otherwise you might get banned. A few seconds is normally more than enough.

---

<div class="post-metadata">

**Author:** ![bakman2](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/bakman2/32/6207_2.png) [@bakman2](https://discourse.nodered.org/u/bakman2)\
**Post date:** [17 January 2023 18:22 UTC](https://discourse.nodered.org/t/can-node-red-process-a-large-http-json-response-as-a-stream/73787/8 "2023-01-17T18:22:53Z")

</div>

That is what I was referring to. An API request is: server work, client (a)waits for complete response. A single request returning 100s of megabytes is not normal, nor common.

If you look at the Microsoft Graph documentation for example, rate limits, throttling, paging are important topics when accessing the/an API. If paging is available: use it.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/1X/d073cd938eafa2e558d7c2cd59003b3ef4963033.png) [@system](https://discourse.nodered.org/u/system)\
**Post date:** [18 March 2023 18:23 UTC](https://discourse.nodered.org/t/can-node-red-process-a-large-http-json-response-as-a-stream/73787/9 "2023-03-18T18:23:02Z")

</div>

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.