Http request node flow broke after upgrade from 1.3.7 to 2.1.6

General
1

I have been running on the latest-10 docker image for a while and finally decided to upgrade to 14. When i did a http request flow that i had setup broke. It does a makes a POST request first and captures the response cookies then follows up with a GET where i parse out some information. The issue seems to be in the POST, i'm getting a undefined back for the msg.responseCookies. If i roll back to node 10 everything works perfect.

Does anyone have any ideas as to what could be going on? Below is the code with my credentials redacted.

[{"id":"6292aa1.62ac554","type":"debug","z":"da556b4c.191f68","name":"","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"responseUrl","targetType":"msg","statusVal":"","statusType":"auto","x":1300,"y":1520,"wires":[]},{"id":"dbdf01ae.12f148","type":"inject","z":"da556b4c.191f68","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":120,"y":1340,"wires":[["7ff0754d.65e44c"]]},{"id":"c601a809.c84928","type":"debug","z":"da556b4c.191f68","name":"","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"responseCookies","targetType":"msg","statusVal":"","statusType":"auto","x":660,"y":1280,"wires":[]},{"id":"7ff0754d.65e44c","type":"function","z":"da556b4c.191f68","name":"","func":"msg.headers = \n{ \"Host\" : \"myfirestar.com\",\n\"User-Agent\" : \"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:95.0) Gecko/20100101 Firefox/95.0\",\n\"Accept\" : \"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8\",\n\"Accept-Language\" : \"en-US,en;q=0.5\",\n\"Accept-Encoding\" : \"gzip, deflate, br\",\n\"Content-Type\" : \"application/x-www-form-urlencoded\",\n\"Content-Length\" : \"53\",\n\"Origin\" : \"https://myfirestar.com\",\n\"DNT\" : \"1\",\n\"Connection\" : \"keep-alive\",\n\"Referer\" : \"https://myfirestar.com/Account/LogOn?ReturnUrl=%2fHome%2fDashboard\",\n\"Cookie\" : \"ASP.NET_SessionId=cmk5cccdn5sezxwlmzn2cew3\",\n\"Upgrade-Insecure-Requests\" : \"1\",\n\"Sec-Fetch-Dest\" : \"document\",\n\"Sec-Fetch-Mode\" : \"navigate\",\n\"Sec-Fetch-Site\" : \"same-origin\",\n\"Sec-Fetch-User\" : \"?1\"}\nmsg.payload =  {\n\"UserName\":\"private\",\n\"Password\":\"private\",\n\"RememberMe\":\"false\"\n};\nreturn msg;\n","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":320,"y":1340,"wires":[["b5811490.60a988"]]},{"id":"b5811490.60a988","type":"http request","z":"da556b4c.191f68","name":"login","method":"POST","ret":"txt","paytoqs":"ignore","url":"https://myfirestar.com/Account/LogOn","tls":"","persist":false,"proxy":"","authType":"","x":490,"y":1340,"wires":[["c601a809.c84928","185697f9.20c0d8","1e4259f8.3df63e"]]},{"id":"1e4259f8.3df63e","type":"function","z":"da556b4c.191f68","name":"","func":"msg.headers = {\n\"Host\": \"myfirestar.com\",\n\"User-Agent\": \"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:95.0) Gecko/20100101 Firefox/95.0\",\n\"Accept\": \"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8\",\n\"Accept-Language\": \"en-US,en;q=0.5\",\n\"Accept-Encoding\": \"gzip, deflate, br\",\n\"DNT\": \"1\",\n\"Connection\": \"keep-alive\",\n\"Referer\": \"https://myfirestar.com/Home/Dashboard\",\n\"Upgrade-Insecure-Requests\": \"1\",\n\"Sec-Fetch-Dest\": \"document\",\n\"Sec-Fetch-Mode\": \"navigate\",\n\"Sec-Fetch-Site\": \"same-origin\",\n\"Sec-Fetch-User\": \"?1\"\n};\nmsg.cookies = msg.responseCookies;\nreturn msg;\n","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":985.3333129882812,"y":1340.3333435058594,"wires":[["6d7ac95.63aa838","1daafc02.437af4"]]},{"id":"1daafc02.437af4","type":"http request","z":"da556b4c.191f68","name":"","method":"GET","ret":"txt","paytoqs":"ignore","url":"https://myfirestar.com/Home/Chart/GX731124","tls":"","persist":false,"proxy":"","authType":"","x":1200,"y":1340,"wires":[["6292aa1.62ac554","e680b47a.3d3f88","a36b829a.20561","357bf00e.7e5d68","e07edfcb.53d7c8","19c996b9.523421","2f29ba66.799e0e","c29ef54c.c59cf","d803e44d.979b1","bc23c9ed.c3dbf","393f9b29.24d934"]]},{"id":"6d7ac95.63aa838","type":"debug","z":"da556b4c.191f68","name":"","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"cookies","targetType":"msg","statusVal":"","statusType":"auto","x":1140,"y":1400,"wires":[]},{"id":"e680b47a.3d3f88","type":"debug","z":"da556b4c.191f68","name":"","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":1280,"y":1560,"wires":[]},{"id":"a36b829a.20561","type":"debug","z":"da556b4c.191f68","name":"","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"cookies","targetType":"msg","statusVal":"","statusType":"auto","x":1280,"y":1480,"wires":[]},{"id":"185697f9.20c0d8","type":"debug","z":"da556b4c.191f68","name":"","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":630,"y":1240,"wires":[]},{"id":"357bf00e.7e5d68","type":"html","z":"da556b4c.191f68","name":"html - temperature","property":"payload","outproperty":"payload","tag":"#log-table > tbody:nth-child(2) > tr:nth-child(1) > td:nth-child(5) > .fahrenheit","ret":"text","as":"single","x":1660,"y":1220,"wires":[[]]},{"id":"e07edfcb.53d7c8","type":"html","z":"da556b4c.191f68","name":"html - reaction","property":"payload","outproperty":"payload","tag":"#log-table > tbody:nth-child(2) > tr:nth-child(1) > td:nth-child(6) > span","ret":"text","as":"single","x":1650,"y":1280,"wires":[[]]},{"id":"19c996b9.523421","type":"html","z":"da556b4c.191f68","name":"html - last update","property":"payload","outproperty":"payload","tag":"#log-table > tbody:nth-child(2) > tr:nth-child(1) > td:nth-child(1)","ret":"text","as":"single","x":1660,"y":1340,"wires":[[]]},{"id":"2f29ba66.799e0e","type":"html","z":"da556b4c.191f68","name":"html - status","property":"payload","outproperty":"payload","tag":"#log-table > tbody:nth-child(2) > tr:nth-child(1) > td:nth-child(2)","ret":"text","as":"single","x":1640,"y":1400,"wires":[[]]},{"id":"c29ef54c.c59cf","type":"html","z":"da556b4c.191f68","name":"html - mode","property":"payload","outproperty":"payload","tag":"#log-table > tbody:nth-child(2) > tr:nth-child(1) > td:nth-child(3)","ret":"text","as":"single","x":1640,"y":1460,"wires":[[]]},{"id":"d803e44d.979b1","type":"html","z":"da556b4c.191f68","name":"html - fan","property":"payload","outproperty":"payload","tag":"#log-table > tbody:nth-child(2) > tr:nth-child(1) > td:nth-child(4)","ret":"text","as":"single","x":1630,"y":1520,"wires":[[]]},{"id":"bc23c9ed.c3dbf","type":"html","z":"da556b4c.191f68","name":"html - first fault","property":"payload","outproperty":"payload","tag":"#log-table > tbody:nth-child(2) > tr:nth-child(1) > td:nth-child(8) > span:nth-child(1)","ret":"text","as":"single","x":1650,"y":1580,"wires":[[]]},{"id":"393f9b29.24d934","type":"html","z":"da556b4c.191f68","name":"html - second fault","property":"payload","outproperty":"payload","tag":"#log-table > tbody:nth-child(2) > tr:nth-child(1) > td:nth-child(8) > span:nth-child(2)","ret":"text","as":"single","x":1660,"y":1640,"wires":[[]]}]
2

The underlying library "request" was replaced with "got"

You might have stumbled upon a bug?

Are there any other clues in the other msg properties?

Is a catch node able to catch any error?

Perhaps you could capture a wireshark of both version to analyse the difference?

3

I had seen the request module was being replaced with "got" and thought the same. A catch node doesn't catch any errors. I'll try to do a wireshark capture with both versions to see what it catches.

Also i tried it on the newest node16 release and it behaves the same as 14.

4

The version of Node.js won't be relevant here - the question is what version of Node-RED you are using.

Given we don't provide a Node 10 container for NOde-RED 2.x that introduced the change for the got module, then that probably explains why that correlates to what you're seeing.

The question is what is actually happening to cause the change in behaviour.

5

It looks like it was working in 1.3.7 and is now not working in 2.1.6 upon update.

I'm struggling to get wireshark to show anything valuable with my working configuration in 1.3.7. I'm not a experienced user of wireshark it just seems everything is pretty cryptic when i look at the transmission data. I don't know if its because this is https instead of http?

6

What status code do you get from the post? (Both versions of node-red)

7

I don't get any status code. Below the block is just says requesting and completes without error.

8

So there is no debug output?

But you said...

Is your debug set to show complete msg object?

9

Yea i get a debug message. The response for complete msg object is html and i can see that i got logged in. However since i'm not receiving the responseCookies my next GET request node doesn't complete since i'm not passing cookies.

10

That is not likely. msg is always an object containing properties (usually payload, topic etc)

Double check that the debug (immediately after your http post request) is definitely set to show "complete message" expand all the properties of the msg & screenshot it.

11

so the complete msg object payload is html. I did'nt scroll down far enough to see the response headers. I'm seeing a status code of 200 on my POST and GET request.

Here is the post request msg object,

{"_msgid":"ec828d6f91b00937","payload":"\r\n<!DOCTYPE html>\r\n<html class=\"firefox firefox95\">\r\n<head>\r\n    <meta charset=\"utf-8\" />\r\n    <meta http-equiv=\"X-UA-Compatible\" content=\"IE=edge,chrome=1\" />\r\n\r\n    <title>FireStar | Dashboard |  Central Boiler, Inc.</title>\r\n    <meta name=\"description\" content=\"FireStar Website\">\r\n\t<meta name=\"author\" content=\"Central Boiler, Inc.\">\r\n    <meta name=\"robots\" content=\"noindex\">\r\n    <meta name=\"viewport\" content=\"width=420, user-scalable=no\">\r\n\r\n    <link rel=\"stylesheet\" href=\"/Content/Responsive.css?v=3\" />\r\n    \r\n\r\n    <script src=\"/Scripts/modernizr.2.71.plusmediaqueries.js\"></script>\r\n</head>\r\n<body>\r\n    \r\n    <div class=\"sidebar left tablet desktop\">\r\n        <h4>Welcome, kinser06!</h4>\r\n            <a href=\"/Account/LogOff\" class=\"pill\"><i class=\"icon-signout\"></i> Log Off</a>\r\n        <div style=\"height: 20px;\"></div>\r\n        <ul class=\"nav unstyled\">\r\n            <li><a href=\"#\" class=\"btn screen-display active\" data-link=\"dashboard\"><i class=\"icon-dashboard\"></i>Dashboard...","topic":"","headers":{"cache-control":"private, s-maxage=0","content-type":"text/html; charset=utf-8","server":"Microsoft-IIS/7.5","x-aspnetmvc-version":"3.0","x-aspnet-version":"4.0.30319","x-powered-by":"ASP.NET","x-frame-options":"deny","date":"Wed, 19 Jan 2022 23:03:30 GMT","content-length":"81739","x-node-red-request-node":"febd0aa2"},"statusCode":200,"responseUrl":"https://myfirestar.com/Home/Dashboard","redirectList":[{"location":"/","cookies":{".ASPXAUTH":{"path":"/","value":"13C5ACAF8F576E624A040D6B211A0394FD4F0F0F4BB966C6968DCC154C12EB566AD3181EA2019FD4C444DD9426B64AA7CE9522DEA43C6C2FE305F2D9F339D7C89B60BEC75124BD779400C35D825C5241D286DF19A4F551E7E4AC67EABAFE9404BCE5EC7C0F9A6C5E6E2586FFF02AF8B4D4AA6FE477258FCBE65DA131DAF5792446955632572C996372F7F84F584B609A"}}},{"location":"/Home/Dashboard","cookies":{"ASP.NET_SessionId":{"path":"/","SameSite":"Lax","value":"t2jipbnsr1xff2xbwvs1m2do"}}}],"retry":0}

Here is the GET request msg object,

{"_msgid":"ec828d6f91b00937","payload":"<!doctype html>\r\n<!--[if lt IE 8]>   <html class=\"lt-ie9 lt-ie8 no-js\" lang=\"en\">    <![endif]-->\r\n<!--[if IE 8]> \t\t<html class=\"lt-ie9 no-js\" lang=\"en\">           <![endif]-->\r\n<!--[if gt IE 8]><!--> <html class=\"no-js\" lang=\"en\"> \t\t    <!--<![endif]-->\r\n<head>\r\n    <meta charset=\"utf-8\">\r\n    <meta http-equiv=\"X-UA-Compatible\" content=\"IE=edge,chrome=1\">\r\n    \r\n    <title>FireStar | Log On | Central Boiler, Inc.</title>\r\n    <meta name=\"description\" content=\"FireStar Website\">\r\n\t<meta name=\"author\" content=\"Central Boiler, Inc.\">\r\n\r\n    <meta name=\"viewport\" content=\"width=device-width, minimum-scale=1.0, maximum-scale=1.0\">\r\n    \r\n    <link rel=\"stylesheet\" href=\"/Content/Site.css?v=2\">\r\n    \r\n    <script src=\"//cdnjs.cloudflare.com/ajax/libs/modernizr/2.6.2/modernizr.min.js\"></script>\r\n<script>\r\n    (function (Modernizr) {\r\n        if (!Modernizr) {\r\n            document.write(unescape('%3Cscript src=\"/Scripts/modernizr-2.6.2.js\"%3E%3C/script%3E'));\r\n        }\r\n    } (this.Moderniz...","topic":"","headers":{"cache-control":"private","content-type":"text/html; charset=utf-8","server":"Microsoft-IIS/7.5","x-aspnetmvc-version":"3.0","x-aspnet-version":"4.0.30319","x-powered-by":"ASP.NET","x-frame-options":"deny","date":"Wed, 19 Jan 2022 23:03:31 GMT","content-length":"6346","x-node-red-request-node":"643b39a8"},"statusCode":200,"responseUrl":"https://myfirestar.com/Account/LogOn?ReturnUrl=%2fHome%2fChart%2fGX731124","redirectList":[{"location":"/Account/LogOn?ReturnUrl=%2fHome%2fChart%2fGX731124"}],"retry":0,"_event":"node:dcc1b79fbd0301cb"}
12

Ok, so now if you do the same thing in earlier node-red, how do the full msg objects differ?

A very quick look on my phone, I did see some cookies in that response (inside redirectList)

1 Like
13

So the big difference is in the redirect list.

1.3.7 (working)

{"_msgid":"2acb75f9.2218ea",
    "payload":"<html><head><title>Object moved</title></head><body>\r\n<h2>Object moved to <a href=\"/\">here</a>.</h2>\r\n</body></html>\r\n",
"topic":"",
"headers":{
    "cache-control":"private",
    "content-type":"text/html; charset=utf-8",
    "location":"/",
    "server":"Microsoft-IIS/7.5",
    "x-aspnetmvc-version":"3.0",
    "x-aspnet-version":"4.0.30319",
    "set-cookie":[".ASPXAUTH=85ABD959FD61A0F7011EEA3CC8E56D1107E6883E089D84C7B969F5A47D02586E450EDF2082168B6BB02271798D9138C7BA1052E9D67E68A7617FB8E313BB8FE889C393FD4CCC2B43C1B9BDDCBA801852EEA8CEF54527405F4F140F1FA0DE0B42AE7D0DD3578B3BB08D21C59EF403A9C126D33D4BAC0D45BC9D93162466E5C8BE133C0931E71FEEE7C4F78D5E9BBF6B43; path=/"],
    "x-powered-by":"ASP.NET",
    "x-frame-options":"deny",
    "date":"Wed, 19 Jan 2022 23:21:16 GMT",
    "content-length":"118",
    "x-node-red-request-node":"2184d20e"},
"statusCode":302,
"responseUrl":"https://myfirestar.com/Account/LogOn",
"redirectList":[],
"responseCookies":{
    ".ASPXAUTH":{
        "path":"/",
        "value":"85ABD959FD61A0F7011EEA3CC8E56D1107E6883E089D84C7B969F5A47D02586E450EDF2082168B6BB02271798D9138C7BA1052E9D67E68A7617FB8E313BB8FE889C393FD4CCC2B43C1B9BDDCBA801852EEA8CEF54527405F4F140F1FA0DE0B42AE7D0DD3578B3BB08D21C59EF403A9C126D33D4BAC0D45BC9D93162466E5C8BE133C0931E71FEEE7C4F78D5E9BBF6B43"
        }}}

2.1.6 has a diffrent response URL and the redirect list is quite long

{"_msgid":"ec828d6f91b00937",
    "payload":"\r\n<!DOCTYPE html>\r\n<html class=\"firefox firefox95\">\r\n<head>\r\n    <meta charset=\"utf-8\" />\r\n    <meta http-equiv=\"X-UA-Compatible\" content=\"IE=edge,chrome=1\" />\r\n\r\n    <title>FireStar | Dashboard |  Central Boiler, Inc.</title>\r\n    <meta name=\"description\" content=\"FireStar Website\">\r\n\t<meta name=\"author\" content=\"Central Boiler, Inc.\">\r\n    <meta name=\"robots\" content=\"noindex\">\r\n    <meta name=\"viewport\" content=\"width=420, user-scalable=no\">\r\n\r\n    <link rel=\"stylesheet\" href=\"/Content/Responsive.css?v=3\" />\r\n    \r\n\r\n    <script src=\"/Scripts/modernizr.2.71.plusmediaqueries.js\"></script>\r\n</head>\r\n<body>\r\n    \r\n    <div class=\"sidebar left tablet desktop\">\r\n        <h4>Welcome, kinser06!</h4>\r\n            <a href=\"/Account/LogOff\" class=\"pill\"><i class=\"icon-signout\"></i> Log Off</a>\r\n        <div style=\"height: 20px;\"></div>\r\n        <ul class=\"nav unstyled\">\r\n            <li><a href=\"#\" class=\"btn screen-display active\" data-link=\"dashboard\"><i class=\"icon-dashboard\"></i>Dashboard...",
"topic":"",
"headers":{
    "cache-control":"private, s-maxage=0",
    "content-type":"text/html; charset=utf-8",
    "server":"Microsoft-IIS/7.5",
    "x-aspnetmvc-version":"3.0",
    "x-aspnet-version":"4.0.30319",
    "x-powered-by":"ASP.NET",
    "x-frame-options":"deny",
    "date":"Wed, 19 Jan 2022 23:03:30 GMT",
    "content-length":"81739",
    "x-node-red-request-node":"febd0aa2"},
"statusCode":200,
"responseUrl":"https://myfirestar.com/Home/Dashboard",
"redirectList":[{
    "location":"/",
    "cookies":{
        ".ASPXAUTH":{
            "path":"/",
            "value":"13C5ACAF8F576E624A040D6B211A0394FD4F0F0F4BB966C6968DCC154C12EB566AD3181EA2019FD4C444DD9426B64AA7CE9522DEA43C6C2FE305F2D9F339D7C89B60BEC75124BD779400C35D825C5241D286DF19A4F551E7E4AC67EABAFE9404BCE5EC7C0F9A6C5E6E2586FFF02AF8B4D4AA6FE477258FCBE65DA131DAF5792446955632572C996372F7F84F584B609A"}}},
            {"location":"/Home/Dashboard",
                "cookies":{
                    "ASP.NET_SessionId":{
                        "path":"/",
                        "SameSite":"Lax",
                        "value":"t2jipbnsr1xff2xbwvs1m2do"}}}],
"retry":0}
14

Any idea for a work around on this?

15

apparently the http's cookie property has changed from responseCookies
so try before the second http-request (the GET request) to use a Function node with
msg.cookies = msg.redirectList[0].cookies

3 Likes
16

That worked perfectly! Thanks so much

17

Did you not see this :point_up:

If you aren't sure how to get that path to the property of interest in the msg (in your case redirectList), use the debug panel...

There’s a great page in the docs that will explain how to use the debug panel to find the right path to any data item.

Pay particular attention to the part about the buttons that appear under your mouse pointer when you over hover a debug message property in the sidebar.

BX00Cy7yHi

https://nodered.org/docs/user-guide/messages

18

all credit goes to Steve because he guided you through compering the replies between the two versions for the initial POST request :wink:

maybe we need an update of the help file for the node if this change of cookie location applies for all replies with got ?

1 Like
19

Sorry guys, i'm a fairly new user and have stumbled through most of my flows so far. I did lots of searching and reading the help file for that node before i decided to post. It kinda through me off that i was seeing a status code 302 on the old verison and code 200 on the new. I guess i just assumed it was a bug.

Thanks again for the help in getting it working.

1 Like
20

No. Not at all. I just wanted to be sure the OP understood the process.

Your answer is the right solution.