Max-http-header-size how to set

General
1

I have error "HPE_HEADER_OVERFLOW" from http request.
How to set max http header size in node-red?

2

What on earth are you sending in the headers???

From node-js docs...

HPE_UNEXPECTED_CONTENT_LENGTH ^

Server is sending both a Content-Length header and Transfer-Encoding: chunked .

Transfer-Encoding: chunked allows the server to maintain an HTTP persistent connection for dynamically generated content. In this case, the Content-Length HTTP header cannot be used.

Use Content-Length or Transfer-Encoding: chunked .

And (IF YOU MUST) a work around...
Set the node-js CLI option --max-http-header-size when you start node-red

3

What on earth are you sending in the headers???

I want o read web site :slight_smile:

image
image1671×326 30 KB

4

What do you hope to grab from that URL?

You know that it is just a page outline and any prices will probably requested via ajax and populated client side AFTER the http request? In other words, there is not really much on that URL you could actually want.

5

I want scrap prices from this site Promocje

6

As Steve says, that site is generated dynamically using a LOT of JavaScript. As a cross-Europe shopping platform, it will have all sorts of checks and security built in as well.

So realistically, the only way to scrape the data is to use a headless browser like puppet. I think that there are 1 or 2 nodes that will do that but it is still likely to be complex.

7

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.