HTTPS request fails on 2.0.5 but worked on 1.3.4

General
1

I tried up upgrade to 2.0.5 last night and it went fairly well but not perfectly. I'm still a bit of a NodeRed newbie (though not a programming newbie) and I was hoping you could help me figure out what's wrong.

My most important flow is used to screen scrape a website. It uses my credentials to login then uses the auth cookies to extract info on my behalf. That flow fails under 2.0.5 with a 404 for the POST to the login page.

I run NodeRed in a docker container on my Synology array so I'm a little more limited in watching what is happening, however, in the terminal I was able to watch the NodeRed container start up and I found it was complaining about the https.request code being patched by old code in a dependency. I hunted down what was using that old code, turned out it was speedtest-updated (or something like that) so I removed that node from a flow that used it, removed it from my NodeRed instance, restarted the instance, and verified that I no longer get the warning on startup. That POST still returned a 404.

I then closed that container down and switched back to the container that's running 1.3.4 and I was able to immediately use that flow. I verified that the POST returns a 302 redirect as expected.

What would be the best way to proceed in debugging this?

I grabbed the msg object input and output from the successful POST under 1.3.4 so I was thinking tonight I would start up 2.0.5 again and grab the msg object input and output for the failed POST then see if I can find any difference. I suspect there will not be any difference in the input and the output likely will not be helpful since it'll just be the content of the 404 page.

I tried to install the flow debugger but I couldn't figure out how to do it, or maybe it's already installed and I just don't know how to use it. I suspect in this case it won't help because I know the input to that node is correct, it feels like something is going wrong during the request to the site and I don't know how to dig in that deep to find out nor do I know if there's a way to directly see what the node actually sends to the site to see how it differs between 1.3.4 and 2.0.5.

I could create a very simple repro flow to give to you guys but since it requires credentials to use it I'm not sure how best to package those for sharing (since I obviously won't share a flow with credentials). Clearly I need to learn if there's a way to supply credentials to a flow without including them directly in the flow. Evidence of being a newbie.

2

The flow debugger doesn't debug the internal code of a node, so won't help here.

We'd need more specific details of the URL you are trying to access and exactly what errors you're getting - what's the full error? Plus any relevant details for trying to recreate it.

3

The error is just a straight 404 NOT FOUND so there's no error message returned. 1.3.4 is able to POST to login but for 2.0.5 the server tells me that URL doesn't exist. That's why it feels like there is something deeper going on.

The specific site is myfitnesspal.com. I'll work up a minimal reproduction later since it's on my home array and I'm at the office. You'll need credentials on that site to reproduce it though. Well, I guess you don't necessarily need working credentials because it will likely not return a 404 if the credentials are invalid, it'll return a different status code.

4

I'm having (is it) the same problem.
My netgear router has local network access where I can log in to a http interface and reboot the router, if I have any public internet problems.
Over a year ago, I made this into a node-RED flow which monitored a ping response, and if it fails, then it reboots my router automatically.
It has worked perfectly for the past year or so, but I've noticed that since updating I'm getting a 401 now when it tries to login using using my credentials to obtain the auth.
This is the response;

<html>
<head>
<meta http-equiv='Content-Type' content='text/html; charset=utf-8'>
<title>401 Unauthorized</title></head>
<body onload="document.aForm.submit()"><h1>401 Unauthorized</h1>
<p>Access to this resource is denied, your client has not supplied the correct authentication.</p><form method="post" action="unauth.cgi?id=cbd9b421c011178ed87d2aa30392829363d6c77ea96f6cea3ec118699ec146c4" name="aForm"></form></body>
</html>
5

No @Paul-Reed , I don't think that is the same error. A 401 is not a 404. Please raise an issue with complete details of the flow you are using so it can be investigated properly.

6

In the body that says it's a 401 but the http status is 404? I will have to look in the body of what is returned to me to see if it says internally that it's unauthorized rather than not being found.

Also, rebooting the router when you lose internet? You have my attention. I lose internet two or three times a week and I never thought about trying to automate a reboot. I could try to do it via login to the cable modem or I could pick up a smart switch and route a power cycle through my Hubitat. Alternatively, I could talk to Cox and try to get them to come out and fix whatever is wrong with my cable but that's not as fun as solving it with automation. :slight_smile: Then again I could also just write a Hubitat app that monitors the connection and reboots the cable modem but where's the fun in going with the simpler solution.

8

@knolleary
This version works in 1.3.4 but fails in 2.0.5. If you use incorrect credentials then it returns a 200 in both versions but if you use valid credentials it will return a 302 with a payload with a short redirect notice under 1.3.4 but it returns a 404 with a full web page under 2.0.5.

The inject node at the beginning has properties for the username and password.

[{"id":"706f8b78.caf304","type":"tab","label":"Login Test","disabled":false,"info":""},{"id":"9f19cdc7.37621","type":"http request","z":"706f8b78.caf304","name":"Get Login","method":"GET","ret":"txt","paytoqs":"ignore","url":"https://www.myfitnesspal.com/account/login","tls":"","persist":false,"proxy":"","authType":"","x":100,"y":140,"wires":[["3587fd6f.9e1e42"]]},{"id":"3587fd6f.9e1e42","type":"html","z":"706f8b78.caf304","name":"Auth Token Extractor","property":"payload","outproperty":"authenticity_token","tag":"form.login > input[name='authenticity_token']","ret":"attr","as":"single","x":160,"y":200,"wires":[["80d18b85.901ac8"]]},{"id":"80d18b85.901ac8","type":"function","z":"706f8b78.caf304","name":"Prepare POST","func":"msg.token = msg.authenticity_token[0].value;\nmsg.url = \"https://www.myfitnesspal.com/account/login\";\nmsg.method = \"POST\";\nmsg.contentType = \"application/x-www-form-urlencoded\";\nmsg.payload = \"utf8=%E2%9C%93&authenticity_token=\" + encodeURIComponent(msg.token) + \"&username=\" + encodeURIComponent(msg.user) + \"&password=\" + encodeURIComponent(msg.password) + \"&remember_me=1\";\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":200,"y":260,"wires":[["a5d06021.66c82"]]},{"id":"a5d06021.66c82","type":"http request","z":"706f8b78.caf304","name":"POST","method":"use","ret":"txt","paytoqs":"ignore","url":"","tls":"","persist":false,"proxy":"","authType":"","x":270,"y":320,"wires":[["ce55b898.7b5558"]]},{"id":"27a14c06.19c8c4","type":"inject","z":"706f8b78.caf304","name":"Do It","props":[{"p":"payload"},{"p":"user","v":"","vt":"str"},{"p":"password","v":"","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payloadType":"date","x":90,"y":80,"wires":[["9f19cdc7.37621"]]},{"id":"ce55b898.7b5558","type":"debug","z":"706f8b78.caf304","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":290,"y":380,"wires":[]}]

Thank you so much for looking into this.

9

Please raise an issue with these details in.

10

Done: HTTPS POST request fails on 2.0.5 but worked on 1.3.4 · Issue #3102 · node-red/node-red (github.com)

11

For anyone else that encounters this, there is a workaround suggested on the Github issue by Ben Hardill and it was effective for me:

Work around is to add msg.followRedirects = false in the last function node.

It looks like request was not following the 302 redirect and got is by default

12

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.