How to fill in and submit the login web form?

[TataGEEK]

Hi, I would like to log in to the site and then perform scraping. Unfortunately the site doesn't provide any API. The form on the page expects the variables "username" and "password". Then there is a CSRF token in a hidden form field.

If I use the Postman app and enter my login credentials and a valid CSRF token in body/form-data, I can log in. The page status returns 200 and I get a valid cookie.
The cURL code is as follows:

curl --location --request POST 'https://schoolclassname.tridnistranky.cz/prihlaseni/' \
--form 'login="username"' \
--form 'password="password"' \
--form 'csrf="valid csrf token"'

Now I'd like to automate the same thing using node-red. I created a flow that gets a valid csrf token and sends it back to the form. Unfortunately, I'm stuck at this point and don't know exactly how to submit the form. I've tried several methods and none of them work.

var csrf = msg.payload[0].value;
var login = 'username';
var password = 'password';

msg.followRedirects = true

msg.headers = {};
//msg.headers['Content-Type'] = 'application/x-www-form-urlencoded';
//msg.headers['Accept'] = 'application/json';
//msg.headers['X-Auth-User'] = login;
//msg.headers['X-Auth-Key'] = password;
//msg.headers['X-CSRF-Token'] = csrf;

//msg.payload = {
//    'login': login,
//    'password': password,
//    'csrf': csrf
//}

msg.payload = "login=" + login;
msg.payload += "&password=" + password;
msg.payload += "&csrf=" + csrf;

return msg;

Here is the code for the function I use to create the headers. I tried several variations and combinations. I've left the previous variations as comments. I also tried defining the content-type of the header up in the "http request" node.

Here is my whole current flow.

[{"id":"630f1967ec6bb45e","type":"tab","label":"Flow 2","disabled":false,"info":"","env":[]},{"id":"dffb09e214114abb","type":"inject","z":"630f1967ec6bb45e","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":120,"y":40,"wires":[["a5fc156472af18a5"]]},{"id":"a5fc156472af18a5","type":"http request","z":"630f1967ec6bb45e","name":"","method":"GET","ret":"txt","paytoqs":"ignore","url":"https://schoolclassname.tridnistranky.cz/prihlaseni/","tls":"","persist":false,"proxy":"","insecureHTTPParser":false,"authType":"","senderr":false,"headers":[],"credentials":{},"x":130,"y":100,"wires":[["1c093d059381b830"]]},{"id":"54b05baf06a47cf9","type":"debug","z":"630f1967ec6bb45e","name":"debug 2","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":400,"y":280,"wires":[]},{"id":"1c093d059381b830","type":"html","z":"630f1967ec6bb45e","name":"Raed CSRF","property":"payload","outproperty":"payload","tag":"form#login-form>input#csrf","ret":"attr","as":"single","x":130,"y":160,"wires":[["ee1da4af691fdae1"]]},{"id":"ee1da4af691fdae1","type":"function","z":"630f1967ec6bb45e","name":"set login variables","func":"var csrf = msg.payload[0].value;\nvar login = 'username';\nvar password = 'password';\n\nmsg.followRedirects = true\n\n//msg.headers = {};\n//msg.headers['Content-Type'] = 'application/x-www-form-urlencoded';\n//msg.headers['Accept'] = 'application/json';\n\n//msg.headers['X-Auth-User'] = login;\n//msg.headers['X-Auth-Key'] = password;\n//msg.headers['X-CSRF-Token'] = csrf;\n\n//msg.payload = {\n//    'login': login,\n//    'password': password,\n//    'csrf': csrf\n//}\n\nmsg.payload = \"login=\" + login;\nmsg.payload += \"&password=\" + password;\nmsg.payload += \"&csrf=\" + csrf;\n\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":150,"y":220,"wires":[["37575f27a5835a67","bb3514dd0702ade4"]]},{"id":"bb3514dd0702ade4","type":"http request","z":"630f1967ec6bb45e","name":"","method":"POST","ret":"txt","paytoqs":"query","url":"https://schoolclassname.tridnistranky.cz/prihlaseni/","tls":"","persist":false,"proxy":"","insecureHTTPParser":false,"authType":"","senderr":false,"headers":[{"keyType":"Content-Type","keyValue":"","valueType":"other","valueValue":"application/x-www-form-urlencoded"}],"credentials":{},"x":130,"y":280,"wires":[["54b05baf06a47cf9"]]},{"id":"37575f27a5835a67","type":"debug","z":"630f1967ec6bb45e","name":"debug 1","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":400,"y":220,"wires":[]}]

If I log in to the site in my browser and watch what happens, I see the following redirection.

Currently nodered is returning me error 405.

Can you please check my node-red code, it's the first flow I'm creating. I've looked through several threads here, but haven't found an answer in any of them.
Thank you very much for your advice.

[marcus-j-davies]

Hi @TataGEEK Welcome to the forums.

Without having access to the resource ourself - debugging these can sometimes be difficult, as like your doing, having access the network frames helps - a lot!

However, what happens if you disable followRedirects, and instead store the cookie(s), and then GET the target pages after you get your token (remembering to provide the cookie headers)?

Just thinking if the redirect is causing some issues

[TotallyInformation]

The body of the POST contains the data so I don't think you need to do as much as you are.

Simply make the msg.payload an object

{
    "login": login,
    "password": password,
    "csrf": csrf
}

and see if that works.

[TataGEEK]

Thank you for your reply. If I change the message format, the following happens:

1. test

{"login":"name","password":"pass","csrf":"bla bla bla"}
MaxRedirectsError: Redirected 21 times. Aborting.

2. test

login=name&password=pass&csrf=bla bla bla
405 Method Not Allowed
The method POST is not allowed for this resource. 

Actually, those were my very first attempts before I even tried to understand what format to use for the header. I read somewhere that Node-red usually sets the headers itself and correctly.

[TataGEEK]

Thank you for the welcome and for your response. I have to say that I don't actually know if using "msg.followRedirects = true;" is the right way to go. I was trying to solve the "MaxRedirectsError" with this (see my previous answer), but it looks like the cause of my problems is somewhere else.

I found a solution with "followRedirects" on a GitHub Issue. It's probably the wrong way for me.

node-red http request not handling HTTP 302 redirects

[bakman2]

I think the flow is:

Login; user/pass, it generates a csrf token, which it hands back to you with a 302, which has a location header.
From here you need to perform a GET request with a csrf-token header and a csrf cookie to interact with the end-point.

If you set a debug node to display the complete message, you should be able to read all the header/cookie information that is returned.

[TataGEEK]

Thanks for the tip with the "Debug complete msg object". I've been using Node-red for a few days and everything is new to me.

The first place I see some cookies is "debug 2". Could this mean that the cookie says that I failed to log in? Or for what reason the cookie is created after error 405?

Your idea of assigning a CSRF token probably doesn't apply here, the site will accept any CSRF token that has been valid in the past. In Postman I can log in just fine with a CSRF that was valid yesterday.