Extract Data from a Webpage

General
1

Hello,
with node-red, I want to extract data from this webpage:

[Kanton Aargau - Hydrometrie]

I'm interested in "Abfluss", "Pegel" and "Wassertemperatur", see attached photo
Because it seems not to be ordinary html (load html and parse for keyword doesn't work), I got stuck.

parse
parse1292Ă—956 215 KB

Thanks for your support...
Roger

2

Unless
First thing to do is to check whether the page works without JavaScript. Try grabbing the whole page using the request node and see what you get.

Sorry, ignore the below. A quick check shows that without JavaScript, you get no content in the page.

However, better news, the data itself seems to come from this URL which returns JSON data:

https://www.ag.ch/app/hydrometrie/kiwis/KiWIS?datasource=0&service=kisters&type=queryServices&request=getTimeseriesValueLayer&format=json&crs=local&timeseriesgroup_id=39016&metadata=true&md_returnfields=custom_attributes%2Cstation_id%2Cstation_no%2Cstation_name%2Cparametertype_id%2Cparametertype_name%2Cts_unitsymbol%2Cts_shortname&orderby=station_name

You can fetch the data with the request node:

image
image394Ă—442 24 KB

Then you just need to find the bits you want.

3

Thank you so much, it is working.
Finally, it was all about to know the right URL. I wasn't able to get this.

4

When trying to scape websites:

  1. Check if the site works without JavaScript
  2. If it does, find the right CSS Selector using browser dev tools
  3. If it does not, use the browser dev tools to start hunting through the JavaScript to see if you can spot the request to get the data. Test to see if you can request that data without the web page (you might not be able to due to security settings). If you can, all good.