HTTP request results in "access denied (edgesuite)" error

[pazkaw]

I'm trying to use Node Red to scrape the ingredients lists of various products from a supermarket's website. When I send an http request to the supermarket's website I receive the response "access denied" followed by a link to the edgesuite website.

People who'd reported similar issues usually find that a missing User-Agent is the root cause of the problem. Indeed, when I explicitly set the User-Agent in a curl request to the supermarket's website:

curl -v "https://www.ah.nl" -H "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:132.0) Gecko/20100101 Firefox/132.0"

then my error message disappears. However, adding the same User-Agent info to the http request in Node Red still returns the same error. I don't have any issues resolving other websites with Node Red. Any suggestions? Cheers

[
{
"id": "8ef4745c1aaeba27",
"type": "http request",
"z": "57a70889b0fabe0a",
"name": "",
"method": "GET",
"ret": "txt",
"paytoqs": "ignore",
"url": "https://www.ah.nl",
"tls": "",
"persist": false,
"proxy": "",
"insecureHTTPParser": false,
"authType": "",
"senderr": false,
"headers": [
{
"keyType": "User-Agent",
"keyValue": "",
"valueType": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:132.0) Gecko/20100101 Firefox/132.0",
"valueValue": ""
}
],
"x": 650,
"y": 120,
"wires": [
[
"4a864bfb9a60c86e"
]
]
}
]

[bakman2]

when searching google, there were issues for mealie (a recipe aggregate app) as well, someone provided a comment:

ah.nl uses TLS fingerprinting detection to do bot detection

you could use a tls proxy, or perhaps use puppeteer instead ?

[E1cid]

Your http request node URL is not correct and gives error, clears when i remove the 1.
The user agent field was also blank , when i add it it seems to work fine for me. e.g.

[{"id":"ea11a4b36247ec49","type":"inject","z":"d1395164b4eec73e","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":340,"y":7600,"wires":[["8ef4745c1aaeba27"]]},{"id":"8ef4745c1aaeba27","type":"http request","z":"d1395164b4eec73e","name":"","method":"GET","ret":"txt","paytoqs":"ignore","url":"https://www.ah.nl","tls":"","persist":false,"proxy":"","insecureHTTPParser":false,"authType":"","senderr":false,"headers":[{"keyType":"User-Agent","keyValue":"","valueType":"other","valueValue":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:132.0) Gecko/20100101 Firefox/132.0"}],"x":510,"y":7600,"wires":[["b43e306f48972352"]]},{"id":"b43e306f48972352","type":"debug","z":"d1395164b4eec73e","name":"debug 2575","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":690,"y":7600,"wires":[]}]

Without user agent I get access denied.

[pazkaw]

Thanks a million! A TLS proxy indeed seems to do the trick. The ja3proxy suggested in that comment did not work for me, but the one from GitHub - rosahaj/tlsproxy: HTTP proxy with per-request uTLS fingerprint mimicry and upstream proxy tunneling. Currently WIP. runs smoothly.

[bakman2]

I get the same error with your flow.

[E1cid]

Strange as these two flows i get to different results 200 and 403

[{"id":"ea11a4b36247ec49","type":"inject","z":"d1395164b4eec73e","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":950.0000762939453,"y":7650.000608444214,"wires":[["8ef4745c1aaeba27"]]},{"id":"8ef4745c1aaeba27","type":"http request","z":"d1395164b4eec73e","name":"","method":"GET","ret":"txt","paytoqs":"ignore","url":"https://www.ah.nl","tls":"","persist":false,"proxy":"","insecureHTTPParser":false,"authType":"","senderr":false,"headers":[{"keyType":"User-Agent","keyValue":"","valueType":"other","valueValue":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:132.0) Gecko/20100101 Firefox/132.0"}],"x":1120.0000762939453,"y":7650.000608444214,"wires":[["b43e306f48972352"]]},{"id":"b43e306f48972352","type":"debug","z":"d1395164b4eec73e","name":"debug 2575","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":1300.0000762939453,"y":7650.000608444214,"wires":[]},{"id":"589255d66e3666a5","type":"inject","z":"d1395164b4eec73e","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":956.0000762939453,"y":7723.000609397888,"wires":[["23cd9c72d027747c"]]},{"id":"23cd9c72d027747c","type":"http request","z":"d1395164b4eec73e","name":"","method":"GET","ret":"txt","paytoqs":"ignore","url":"https://www.ah.nl","tls":"","persist":false,"proxy":"","insecureHTTPParser":false,"authType":"","senderr":false,"headers":[],"x":1126.0000762939453,"y":7723.000609397888,"wires":[["1eadec1ae34eddb4"]]},{"id":"1eadec1ae34eddb4","type":"debug","z":"d1395164b4eec73e","name":"debug 2575","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":1306.0000762939453,"y":7723.000609397888,"wires":[]}]

Only thing i can think is DNS is not my ISP's I use cloud flare in router, and all ISP safety and protection is turned off.