Is there any way to automatically authenticate myself in a site with a node?

I took a quick look at that site, and it looks like a html form in the page, where info is entered and then POSTed back to login, thus setting a (session) cookie where then the next values can be received. So basically it's a scraping job with just a regular login system. I've done a lot of scraping jobs for all kinds of projects so far, several even work related. Is there a particular reason why you want the "get the output" part to be done in Node-RED elements rather than utilising another tool for that?

If I had to do this task in a work related context I would see a couple options, with a first step shared by all:
0. Reverse engineer the website and figure out how exactly the login system works, if things are stored as cookies and if those cookies are easily transferred or if you would have to simulate the login process on each attempt.
Options:

  1. Write a custom script in a language of choice that handles the logging in process and gathering the value, then make it so that this script gets called and returns the value to node-red
  2. Figure out if you can send a POST request through the HTTP request node where cookies are shared to other nodes in the flow. I haven't played with that yet, so I can't give the answer to it (mostly because this wouldn't be my own preferred option). If yes, send a POST request to the where the login ends up, then share that session with the next node which will be a GET to that b2b page, then follows the rest of your current flow.
  3. Write the entire calling setup in a function node, where callbacks are used to forward each step of the http processing until you get the data you need, then forward it to that switch node (though take a look at the RBE node as well because I think it does exactly what you want to do with that switch)
  4. Handle this entire flow outside of node red (or perhaps partially in node red with just the timestamp node and executing your script)

As for the "use authentication" on the http request node, as far as I know that one is to pass a username/password over HTTP Basic Auth to a web page, which is not the case for the website you mentioned here. HTTP Basic Auth is (simplified) when the web browser gives you a popup that says "you need to login to open this page" and you have to enter a username and password in that popup before you can get control of your web browser back.

(Note: I'm a Q/A engineer by trade, and automating websites/write and run automated tests on websites was part of my job description :stuck_out_tongue: Any more questions, just ask.)

2 Likes