Max-http-header-size how to set

I have error "HPE_HEADER_OVERFLOW" from http request.
How to set max http header size in node-red?

What on earth are you sending in the headers???

From node-js docs...


Server is sending both a Content-Length header and Transfer-Encoding: chunked .

Transfer-Encoding: chunked allows the server to maintain an HTTP persistent connection for dynamically generated content. In this case, the Content-Length HTTP header cannot be used.

Use Content-Length or Transfer-Encoding: chunked .

And (IF YOU MUST) a work around...
Set the node-js CLI option --max-http-header-size when you start node-red

What on earth are you sending in the headers???

I want o read web site :slight_smile:

What do you hope to grab from that URL?

You know that it is just a page outline and any prices will probably requested via ajax and populated client side AFTER the http request? In other words, there is not really much on that URL you could actually want.

I want scrap prices from this site Wyprzedaże

As Steve says, that site is generated dynamically using a LOT of JavaScript. As a cross-Europe shopping platform, it will have all sorts of checks and security built in as well.

So realistically, the only way to scrape the data is to use a headless browser like puppet. I think that there are 1 or 2 nodes that will do that but it is still likely to be complex.

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.