Node red Web Scrapping with login

General
1

Hi all!

I'm working on a project were i need the following things:

  1. Login in a website with emai and password.
  2. Click on a PDF bill, this will open a new tab
  3. Take a full window screen shot of the new tab PDF
  4. Save the screenshot on a folder whatever it will be
  5. Send that PDF via email

I know is a bit tricky to make it work and i know there are some nodes that can help me out. Maybe pupeteer or some other tool. I'm new about this project.

Thanks a lot and i hope someone could help me.

Regards!

2

Have you considered downloading the pdf file instead, that might be much simpler.

3

Yes that could be an option.

First off i need to login to the website and click on the PDF that i need.

This is what i need to get.

Screen Shot 2021-04-29 at 17.32.28
Screen Shot 2021-04-29 at 17.32.281171×497 54.2 KB

If i click on the PDF icon then another new tab opens and shows me the PDF. And i can saves as...there too.

Regards

4

Marty,

I use puppeteer for a similar type of job and it is amazing . Have not tried integration into node-red but wise case just call your puppeteer script as an executable passing in paths and get it to write a PDF . With puppeteer there is a bit of a learning curve but your requirements are not to bad. This may be 20-30 lines of code (really depends on how easy it is to find the first PDF

It is tricky for me to share an example script (a bit of work to make it generalised but happy to help with puppeteer as (I am not an expert)

Btw as Colin says to download the PDF is about the same work with puppeteer and has the advantage of having the source document you can the potentially parse using a node-red node

5

Great @chrisn-au i will try pupeteer also as Selenium. I used to do some scripting with python and Selenium back in the days. Now i want to do it in NodeRed bacause is easier and reliable. If i do it in python as a service i need to do some Docker, cron, etc...Node red comes so solve a part of that.

So, how can i start with pupeteer as a guide?. Do you a simple flow example to get some elements on google or so?

Regards!!

6

To start with have a look at GitHub - checkly/puppeteer-examples: Puppeteer example scripts for running Headless Chrome from Node. there look like some great examples . It is archived now but I am sure the core capability should just work.

If this doesn’t get you going let me know and I will pull out some of my sample stuff (just not trivial to share given custom nature of the beast )

Sounds like you are going to deploy using docket - a great idea (although it was one of the last things I moved to Docker) . To integrate the node-red flows into a Docker world I would add a http endpoint to your puppeteer script. Then you simply expose the http port in docker and you should be able to call from node-red. Depending on the timing for your puppeteer script you may even get the http call to return the PDF or you get the puppeteer to call a http endpoint in node red

7

Ok… so i will have a look at the url you send and see what i can understand.

Yes, i use nodered with docker. Please if you have a basic example so i can start with it will be great!
Add a flowline here.

Regards!

8

Anyone?

Im trully lost with this project. I need first to login here Telecentro | Inicio
How can i fill the user and password fields in headless mode?

Any clues?

Thanks!

9

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.