Web scraping flow/nodes with login before getting data

Node-RED flow.
I need a "function" or "http request", that can login with username/Password to this website brewcreator.com and the get your "profilename" from the dashboard "HELLO, YOUR NAME!" so I get that as a output to a debug node or that can be passed to a "switch" node.
The blue circle (screenshot) is what im looking to get, bc. if you make a profile, you don't have the "Graviator" device.

But I have added a orange circle, bc. this is the true object im looking to get after login, I have added a screenshot that's show the source code to that info.

But if you create an account, you will only be able to see "Hello" and your name and not, as in my picture, see info from my graviator.
A graviator is a modern device that can measure one's brewing process of a beer and here I would like to be able to retrieve its temperature (orange circle) so that I can start a ZBMINI via a switch, depending on what temperature I get.

So what I'm interested in is actually just a flow where you can login to the webpage with your information and then "Hello..." your name is displayed in a debug node or switch.
But have added a screenshot that shows the true ID i'm looking to get.

I don't know what this will cost, but write... I can pay by paypal.

Can someone help, and what will it cost? :slight_smile:

Just to explore some ideas (for those who may grab this project )

are these graviator devices a product of brewcreator.com
or do these devices have specific relationships with brewcreator.com

Im thinking if these devices have a documented API instead of resorting to web scraping
Perhaps a little more on these graviator things may help

When you add the Graviator to your account, you have a SN on the Graviator, that you add to the website, and then the website will open for the Graviator, that connected to my home network WiFi.

There are nodes for puppeteer which would allow you to remotely control a website beyond just getting pages with a http request.

Hi @nodeautomata yes, but puppeteer is not working on a QNAP nas.
You can add the install/node and set it up, when you then trigger the launch node, you will get an error.
See image, so have tried that.

Maybe because you didn't click checkbox in launch node to run headless and/or chrome not properly installed to run headless.

Puppeteer might be trying to fire up chrome in a gui desktop which doesn't exist.

Setting up a second computer running node red with a gui and chrome makes debugging your spider much easier btw. You will actually see chrome open and do stuff so you can tell what's going on.

Puppeteer basically runs chrome remotely.

The Node has no headless in it and can see that, there is an error in the code for the node.

It's a checkbox in the browser-launch node. I forget the exact wording.

Did you try a simple http request node? If you're not dealing with the browser rendering javascript then you might be able to keep simple and just request the page and extract the data.

Puppeteer is every bit as cool as it is a pain in the ass to set up and keep going.


Puppeteer (as already mentioned) is one way to get the data. Maybe a direct-access API is available? Then you can use HTTP request, some tokens / cookies etc. and request the data directly. Some APIs restrict calls by volume, token, GeoIP etc. so keep that in mind.

You can see the API calls being made by using browser Developer Tools (F12 in Chrome).