Automating a website with Node-RED (login, clicks, download)

General
1

Hello everyone,

I have a question about Node-RED and hope someone can help me.

I would like to automate the interaction with a website. Specifically, I want to perform the following steps:

  1. Open a website
  2. Enter a username and password
  3. Click the “Login” button
  4. Scroll or navigate on the page
  5. Download a file

My question is: Is it possible to perform these actions using Node-RED?
If yes, which nodes or methods would you recommend for this?

I have already worked a bit with HTTP requests, but I am not sure how to properly handle login, button clicks, scrolling, and file downloads.

Thank you very much for your help!

2

Yes. There are some puppeteer nodes out there.

1 Like
3

What you describe seems more like RPA (Robotic Process Automation), and there are quite a few tools which allow you to record & replay logical sequences of data input, mouse clicks etc.
Node-red is more for automating the "essence" of the operation, rather than being a dumb robot which just scrapes screens & presses buttons.
The rule of thumb is to use RPA when there are no available APIs or access to data. Use RPA for legacy systems, or where the sequences are rigid and don't change much

4

Depending on the login (nowadays login is more complex than the http request node is most likely to be able to handle due to security), puppeteer may be the 'easiest' approach, see this example how to do it using a function node. (there are nodes for node-red available but those either don't work most of the time, or are outdated).

5

Hello!

Thank you for your reply.

What do I need to install? And can Node-RED run in a Docker container, or does it require a full operating system?

6

However, since this proposed solution (puppeteer via function node) will essentially fire up a headless browser, there may be components not available inside a bare/basic image. You would need to try it out.

7

I realise this all totally depends on the auth and how the file is generated/served but using the standard HTTP Request node would be my first approach (long before puppeteer/headless browser).

In simple terms:

  • login (get token/cookie/whatever)
  • http request the file to download (and include the token/cookie/whatever in headers)
8

OK.

Do you have an example?

9

It is impossible for me to know the auth type, the structure of the file download page or if the file is even served as a resource (or generated in script).

You need to understand what is actually happening.

You can achieve this by:

  • open your browser, open the devtools and switch to the Network panel
  • navigate to the page
  • log in
    • see what headers are added/returned
  • download a file
    • see if the file appears as an entry in the network panel
    • grab its URL
    • inspect the headers sent with this request

In Node-RED

  • try the file URL it in a simple flow inject -> http request (set URL to file URL) --> debug
  • try adding any headers to the http request (those inspected on the file download test in your browser)

Once you manage to do this manually, you can automate the steps. e.g. suppose the login makes a POST with user/pass and the result is a particular cookie you need to include in every request, then you would perform the login, store the cookie, apply the cookie, download the file.