Have been running NodeRed for quite a few years and up until recently was running 3.1.9 using the official amd64 Docker image. Last Friday I updated the container to 4.0, with no other changes. Since then I have been having a weird issue and wondering if anyone else has seen similar.
One of my flows sends notification messages (maybe 50 or 100+ messages per day) by using the http-request node to send commands to the Discord bot API. After the upgrade to 4.0, these messages stopped abruptly.
When I view the output from the http-request node, it shows HTTP error 403, and an error text returned from the remote server "internal network error, code 40333"
A bit of googling suggests this error is coming from the Cloudflare frontend deployed by Discord and usually occurs when a HTTP request sent to the API has been malformed in some way.
The timing of the issue coincides exactly with my upgrade to 4.0 so I wonder if something in NodeRed, or in node version 20, might have changed the handling of http requests.
If I send the same HTTP request (with same url, same payload and same headers) using cURL on a command-line, the request is successful.
I also deployed a new container of NodeRed 3.1.9 on a different machine and copied just that part of my flow which sends the message to Discord. This flow still works fine on 3.1.9
My current workaround in my original flow under 4.0 is to use an exec node which calls cURL with the same arguments, and this is successful.
Bit stumped here and wondering if anyone else has encountered anything similar.
I now wonder if the addition of user-agent (albeit recommended by the RFC) is the problem.
Could you try adding an empty (or alterative) user-agent header to your requests?
Essentially, set
msg.headers['user-agent'] = ''
or
msg.headers['user-agent'] = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.79 Safari/537.36'
Yes I did also stumble on that link while googling my issue I do suspect that this is somehow related to a User-Agent header.
In my flow I normally send a valid User-Agent, but I tried just now to send a null User-Agent as a test, but the Cloudflare 403 error still occurs.
I suspect the http-request node itself might be overwriting the User-Agent with some internal value of its own, but I don't have a way to confirm this.
Another issue I noticed: while trying to inspect the traffic generated by the http-request node, I set up an intercepting proxy ("Charles" -- https://www.charlesproxy.com/) . I configured the http-request node to send requests via this proxy. However in my NodeRed 4.0 instance, the proxy setting appears to be ignored and the connection goes out direct, so I can't inspect the headers.
In my NodeRed 3.1.9 instance, the proxy setting in http-request does work and I can see the connection going via Charles as expected
Correct, i just discovered the http node cannot sent an empty string (will be raising an issue shortly) HOWEVER, i have confirmed that if you set a user agent in msg.headers or in the UI it will be sent.
Could you share (private DM if required) further details? Perhaps share an example flow and proxy details so I can test and debug this? There were a lot of changes around proxy handling in NR 4. There may be a bug here. also, please include any *_proxy ENV VARs you set (including uppercase/lowercase variants)
Then launch the "charles" application and it starts listening on port 8888. By default there is no authentication.
Inside NodeRed I configured the http-request node with the IP of my Ubuntu machine and port 8888.
To enable SSL decryption in the proxy there is an additional step ... to grab the SSL certificate which Charles generates, and configure that in the http-request node TLS config:
By the way , when I noticed that the proxy settings I had defined in the http-request node did not appear to take effect on NR 4.0 ... I restarted my NodeRed container with the charles proxy defined in the variable "http_proxy" instead. But this also didn't have any effect that I could see, and the connection still went out direct.
Are you able to share details of this endpoint so that I can debug the issue? It would make this much quicker to have a known problematic endpoint to poke at. Share details via DM if you can. If you cannot, then no worries.
PS, i have discovered we are not able to set a header value empty (issue raised here) so instead, try setting an alternative user-agent is the only option currently.
Hi, that's strange about the User-Agent header, because I have been setting a value for this (tested both by setting msg.headers, and explicitly in the http-request node itself). But I am still getting the Cloudflare error.
I will send a DM with details of a curl command and flow which you can try. The curl command works, but the flow (which I believe should send the exact same http request) does not.
I do suspect that this is somehow related to a User-Agent header.
