Reusing set-cookie from response headers in HTTP request

[josechu]

Hi all,

I'm struggling a bit to create an http flow in node-red... I've been fighting for some days, going ahead, but not achieving the final result, so any help will be a lot

This is tied to this issue, but now closed.

I'm trying to fetch info from a web server which login is controlled by a cookie parameter set as http header, as usual.

When loading the login page the server sends a "set-cokie" parameter in the response headers. I'd need to reuse that parameter which comes in this form:

set-cookie: ASPSESSIONIDCUSRARBR=OLJCEELALEKFHOJOCEEEKAHH; secure; path=/

I'd need to reuse that cookie value (removing the last part I guess) to make requests adding in headers of the http request in this format (this is the example from curl command):

  -H 'cookie: ASPSESSIONIDCUSRARBR=OLJCEELALEKFHOJOCEEEKAHH'

I tried several ways, some cleaner than others... but non worked

Some of the ideas... first one was to try to use it directly:

Other idea has been (as I usually inject headers in HTPP requests in node-red) with a change node... and for this i was trying to change msg.headers with "'cookie: " & $msg.headers["set-cookie"][0] & "'"... but it doesnt come with the cookie value in the middle of the sentence...

Last idea Im trying is with a function...

msg.headers = msg.headers || {};
msg.headers["cookie"] = msg.headers["set-cookie"]
delete msg.payload

let avcookie = ""
let newMsg = {}  // clean new msg

// filter cookies from previous reply 
msg.headers["cookie"].forEach(el => {

    if (el.startsWith("ASPSESSIONID")) avcookie = el.split(";")[0]
})

newMsg.headers = {
    "Cookie": `${avcookie}`,
}
//delete msg.headers["set-cookie"]
return newMsg;

And then with a change nod using newMsg as msg.headers...

But non of them worked, all the ideas come from other msgs and forums... thats why some are better/cleaner than others.

I'd like to use the more simple option that exists... if it can be without a function and reusing that set-cookie value somehow... that would be the best

I'm suspecting that my issue is there in the cookie thing... but there is other part of the login request sent in the initial POST (after getting the set-cookie in a previous GET), when doing all of this manually is like:

  -H 'cookie: ASPSESSIONIDCUSRARBR=OLJCEELALEKFHOJOCEEEKAHH \
  --data-raw 'nic=EMAIL&pwd=PASSWORD&Enviar=Entrar+%2F+Enter'

This part im adding it as msg.payload with a change node injected in login page. As an string, without the simple quotes. Maybe all the headers cookie was right in some if the tries and the issue was here now that im thinking xD, I hope not!

Thanks!

[josechu]

The full flow in curl commands, just if anyone needs to know how this is done from a manual form:

First to get the cookie I do a normal GET, and get it in responde headers (it changes on every request as far as I see):

< HTTP/2 200
< cache-control: Private
< pragma: no-cache
< content-type: text/html
< expires: Wed, 05 Jan 2000 11:12:12 GMT
< server: Microsoft-IIS/10.0
< set-cookie: ASPSESSIONIDCUSRARBR=OLJCEELALEKFHOJOCEEEKAHH; secure; path=/
< x-powered-by: ASP.NET
< date: Thu, 08 Sep 2022 13:40:45 GMT

Then to login:

curl 'https://URL/default.asp' \
  -H 'cookie: ASPSESSIONIDCUSRARBR=OLJCEELALEKFHOJOCEEEKAHH' \
  --data-raw 'nic=EMAIL&pwd=PASSWORD&Enviar=Entrar+%2F+Enter'

Then some middle webpage that loads nothing but its needed as far as I've tested:

curl 'https://URL/cargaDatos.asp' \
  -H 'cookie: ASPSESSIONIDCUSRARBR=OLJCEELALEKFHOJOCEEEKAHH

And finally the webpage that give the HTML content where im parsing some stuff:

curl 'https://URL/inicio.asp' \
  -H 'cookie: ASPSESSIONIDCUSRARBR=OLJCEELALEKFHOJOCEEEKAHH'

When doing the 3 GETs... one after the other manually in this order (as seen in Chrome inspect), It works,

[bakman2]

If you check the request node documentation:

Cookie handling

The cookies property passed to the node must be an object of name/value pairs. The value can be either a string to set the value of the cookie or it can be an object with a single value property.

Any cookies returned by the request are passed back under the responseCookies property.

[josechu]

Thanks for your reply, yes that was other way I was playing with, so many ideas and i forgot some for sure in the initial post:

But it provides very similar content as in msg.headers["set-cookie"] so I continued with the other, the thing is that I don't know how to reuse it in a Request after fetching the cookie in either way.

If I use it as headers for the next request like this:

It doesnt work... maybe the unnamed object (first screenshot of this post) should be renamed as "cookie" as in the curls in my second post? or I need more coffee?

[bakman2]

Could you try:

msg.headers = {cookie: msg.responseCookies.ASPSESSIONIDCUSRARBR.value}

[josechu]

Do you mean here? It gives a node-red error, it doesnt seems the format that field expects

I guess you mean in other place, right?

[josechu]

Or this? but same error, about json format maybe, something i can investigate... but is this the place you mean?

[bakman2]

No either in a function node or set the change node to J as in jsonata and set the value to:

{"cookie": responseCookies.ASPSESSIONIDCUSRARBR.value}

[josechu]

btw, from this image:

Both values change on every request, so we should reuse the ASPSESSIONIDXXXX and the following value also, both change.

[bakman2]

Yes I was wondering if that was the case (as it should change with each new session) but your examples all showed the same.

Then use jsonata in the change node and set the value to:

{"cookie": **.value}

[josechu]

better yeah, but we'd need to maintain both variables, not just the value, also the variable name:

Because the cookie value must be both fields, like in the examples of the first posts.

Thanks so much btw, this gives me area to investigate further

[josechu]

Same, only one value (seems the second, not the ASPSESSIONIDXXXX):

Update, the difference between before and after change node:

[bakman2]

Try it like this:

{
    "cookie":*.$keys()&"="&**.value
}

[josechu]

This last one im still trying to understand it , but doesnt work:

[bakman2]

Sorry i forgot about the rest of the object

{
    "cookie":responseCookies.$keys()[0]&"="&**.value
}

edit
This is more error proof:

{
    "cookie":responseCookies.$keys()[0]&"="&responseCookies.*.value
}

[josechu]

@bakman2

{
    "cookie":*.$keys()&"="&**.value
}

With that I get this:

Not sure if is same format as in the curls from my first post :S, doesnt work anyway.

[josechu]

And now I have tried this: (trying to invent... but gave no error at least for creating the parameter hehehe)

And I see the cookie parameter created in the headers (among rest of responseHeaders stuff)... but I get other error, related to this or maybe the raw-data with the login info?

[bakman2]

Not sure if is same format as in the curls from my first post :S, doesnt work anyway.

it is the same format.

the error comes from a different node (default.asp) ?

[josechu]

Maybe all this is from a different thing... oh my... Could be the raw data injected for the login?
In my manual curl is something like:

curl 'https://URL/default.asp' \
  -H 'cookie: ASPSESSIONIDCUSRARBR=OLJCEELALEKFHOJOCEEEKAHH' \
  --data-raw 'nic=EMAIL&pwd=PASSWORD&Enviar=Entrar+%2F+Enter'

So they way I inject it in the HTTP request node is this way:

I have tried also without quotes... or in JSON format like {"nic":"EMAIL","pwd":"PASS","Enviar":"Entrar+%2F+Enter"}
maybe other format is needed?

BTW, sorry I didnt reply earlier yesterday... but faced this thing hehe

[bakman2]

data-raw is a POST request, needs to be an object.