# Max-http-header-size how to set

**URL:** <https://discourse.nodered.org/t/max-http-header-size-how-to-set/61907>\
**Category:** General\
**Created:** [28 April 2022 06:42 UTC](https://discourse.nodered.org/t/max-http-header-size-how-to-set/61907 "2022-04-28T06:42:54Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![ketpa](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/ketpa/32/42496_2.png) [@ketpa](https://discourse.nodered.org/u/ketpa)\
**Post date:** [28 April 2022 06:42 UTC](https://discourse.nodered.org/t/max-http-header-size-how-to-set/61907/1 "2022-04-28T06:42:54Z")

</div>

I have error "HPE\_HEADER\_OVERFLOW" from http request.  
How to set max http header size in node-red?

---

<div class="post-metadata">

**Author:** ![Steve-Mcl](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/steve-mcl/32/4826_2.png) [@Steve-Mcl](https://discourse.nodered.org/u/Steve-Mcl)\
**Post date:** [28 April 2022 07:24 UTC](https://discourse.nodered.org/t/max-http-header-size-how-to-set/61907/2 "2022-04-28T07:24:07Z")

</div>

What on earth are you sending in the headers???

From node-js docs...

> `HPE_UNEXPECTED_CONTENT_LENGTH` [^](https://nodejs.org/api/errors.html#hpe_unexpected_content_length)
> 
> Server is sending both a `Content-Length` header and `Transfer-Encoding: chunked` .
> 
> `Transfer-Encoding: chunked` allows the server to maintain an HTTP persistent connection for dynamically generated content. In this case, the `Content-Length` HTTP header cannot be used.
> 
> Use `Content-Length` or `Transfer-Encoding: chunked` .

And (IF YOU MUST) a work around...  
Set the node-js CLI option [`--max-http-header-size`](https://nodejs.org/api/cli.html#--max-http-header-sizesize) when you start node-red

---

<div class="post-metadata">

**Author:** ![ketpa](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/ketpa/32/42496_2.png) [@ketpa](https://discourse.nodered.org/u/ketpa)\
**Post date:** [28 April 2022 07:43 UTC](https://discourse.nodered.org/t/max-http-header-size-how-to-set/61907/3 "2022-04-28T07:43:18Z")

</div>

> What on earth are you sending in the headers???

I want o read web site 🙂

 ![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/c/b/cbf8de82debbe6bcf29f4010df872b62c682fe17.png)

---

<div class="post-metadata">

**Author:** ![Steve-Mcl](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/steve-mcl/32/4826_2.png) [@Steve-Mcl](https://discourse.nodered.org/u/Steve-Mcl)\
**Post date:** [28 April 2022 07:55 UTC](https://discourse.nodered.org/t/max-http-header-size-how-to-set/61907/4 "2022-04-28T07:55:49Z")

</div>

What do you hope to grab from that URL?

You know that it is just a page outline and any prices will probably requested via ajax and populated client side AFTER the http request? In other words, there is not really much on that URL you could actually want.

---

<div class="post-metadata">

**Author:** ![ketpa](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/ketpa/32/42496_2.png) [@ketpa](https://discourse.nodered.org/u/ketpa)\
**Post date:** [28 April 2022 08:25 UTC](https://discourse.nodered.org/t/max-http-header-size-how-to-set/61907/5 "2022-04-28T08:25:38Z")

</div>

> [@Steve-Mcl](#):
>
> You know that it is just a page outline and any prices will probably requested via ajax and populated client side AFTER the http request? In other words, there is not really much on that URL you could actually want.

I want scrap prices from this site [Promocje](https://www.lidl.pl/q/query/wyprzedaze?offset=96&sort=FFCheckoutCount-desc&legacy_ff=Zm9sbG93U2VhcmNoPTk5MDA)

---

<div class="post-metadata">

**Author:** ![TotallyInformation](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/totallyinformation/32/31_2.png) [@TotallyInformation](https://discourse.nodered.org/u/TotallyInformation)\
**Post date:** [28 April 2022 12:30 UTC](https://discourse.nodered.org/t/max-http-header-size-how-to-set/61907/6 "2022-04-28T12:30:29Z")

</div>

As Steve says, that site is generated dynamically using a LOT of JavaScript. As a cross-Europe shopping platform, it will have all sorts of checks and security built in as well.

So realistically, the only way to scrape the data is to use a headless browser like puppet. I think that there are 1 or 2 nodes that will do that but it is still likely to be complex.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/1X/d073cd938eafa2e558d7c2cd59003b3ef4963033.png) [@system](https://discourse.nodered.org/u/system)\
**Post date:** [27 June 2022 12:31 UTC](https://discourse.nodered.org/t/max-http-header-size-how-to-set/61907/7 "2022-06-27T12:31:19Z")

</div>

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.