Node-red-contrib-simple-webdriver don't work on my pi zero

General
1

Hello everyone,

I m beginner with node-red, today I m trying to use node-red-contrib-simple-webdriver to make a scrapping bot. I installed chromium and chromedriver but I have this error

"Can't open an instance of chromium"

"WebDriverResponseError: session not created : Unable to find a matching set of capabilities"

Any one would know how to do ? and if it's possible to do it on raspberry pi zero without graphic environement.

Thank you!

2

Forgive me if I misunderstand you, but you can do web scraping with the Html node.

There are some example flows available via Import, examples, node-red, parser.

3

Hi Jbudd,

I need to scrap website like X / facebook, I don't think it's possible with html request is it ?

4

Sorry, I don't know, my use is limited to extracting data from weather reports and rail movements.
I'm sure other people have much greater experience.

5

You need more than http-request IF the site you are trying to scrape requires JavaScript to work.

Check to see if you have access to an "API". If there is one, you can usually get away without.

6

haha thank you anyway for answer, even if it's possible in an other way I m still curious to know how to use this node module. I don't understand why it doesnt work that's my problems, I will continue search to see where I fail.

7

Yes I can use api but they are expensive and I need to refresh data often to check update. This why I wanted to try to use selenium, I read it was possible to use it even with headless browser this what I m trying to do :slight_smile:

8

Should be possible on a headless Pi. Try on a simple request initially to make sure that the complexities of X/Facebook don't get in the way.

There are a few similar nodes you could also try

https://flows.nodered.org/search?term=selenium

9

Yes thank you I first try to make a connexion to google lol, I m stuck at the first step opening the browser haha

10

After having several (unexplained) issues with simple-webdriver, I would recommend to use the puppeteer node instead.

11

I finaly made it work after a new install of everythings, but work only with chrome browser and not chromium....and it's really slow on pi 0, I tried puppeteer but seems pi zero is too slow for it too.