Hi guys,
i have tried to read out this page per http request
"Bad Schallerbach - GEM2GO WEB - Zentrum - Bürger<br>service - Abfall & Entsorgung - Abfalltermine"
But the information about the schedule is not included in the request answer - maybe it will be cut off?!
Is there another way, to get this information?
Thanks guys!
OK - with the HTML Parser i can see the information in de Debug menu.
My problem is now, that i cant get the right information. maybe because of the cookie query.
I have the correct link in node red but in the debug i get information from the wrong street, also because of the cookie problem i think.
These are the information i get:
To get more responses for help it would be best to capture the html page, then construct a sample flow with your html node and selectors you have tried, put the captured http request info in an inject node or template node. Also supply what info you are trying to extract, may be highlight it in a screenshot. This means others then can test and will know what info you are trying to extract.
I hope i unterstood you correctly
[{"id":"c022c193e847067e","type":"http request","z":"f1c196de.aec348","name":"","method":"GET","ret":"txt","paytoqs":"ignore","url":"https://www.bad-schallerbach.at/system/web/kalender.aspx?sprache=1&menuonr=226753868&typids=226758838,226758840,226758842,226758844","tls":"","persist":false,"proxy":"","authType":"","x":390,"y":760,"wires":[["57a7c2021e75034a"]]},{"id":"57a7c2021e75034a","type":"html","z":"f1c196de.aec348","name":"","property":"payload","outproperty":"payload","tag":"p","ret":"text","as":"single","x":570,"y":760,"wires":[["b5a6516a49411875"]]},{"id":"b5a6516a49411875","type":"debug","z":"f1c196de.aec348","name":"","active":true,"tosidebar":true,"console":true,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":750,"y":760,"wires":[]},{"id":"28167ca2afd2ce12","type":"inject","z":"f1c196de.aec348","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":160,"y":760,"wires":[["c022c193e847067e"]]}]
i want the yellow highlited information. There should be more then 1 day available
Here is an example:
You need to capture the request returned payload, as a login is required. No one can see the returned payload in full so can not see the source.
save the return payload of the request node to file using a file write node, then upload here so we can see the source.
This is the page(with no styling) your http request node returns for me, there is no table like the image you show
Hauptinhalt: Kurztaste 1
Hauptnavigation: Kurztaste 2
Metanavigation: Kurztaste 3
Suche: Kurztaste 4
Bad_Schallerbach_logo_neu
Bürger
service
Informationen
Amtstafel
Bad Schallerbach CARD
Fahrpläne
Gemeindezeitung
Neuigkeiten
Ortsplan
Wasserversorgung
Wohnungsangelegenheiten
E-Government
Duale Zustellung
Formulare
Gem2Go
Zivilschutz SMS
Digitales Österreich
Handysignatur
Amtssignatur
Abfall & Entsorgung
Altstoffsammelzentrum
Bezirksabfallverband
Grünschnitt
Abfalltermine
Jahresplan-Druck
Müllinfoservice
Abgaben &
Verordnungen
Gebührentabelle
Jugendtaxi
Studentenförderung
Umweltförderung
Verordnungen
Voranschlag
Rechnungsabschluss
Verwaltung
& Politik
Verwaltung
Amtsleitung
Bauabteilung
Bürgerservice
Finanzabteilung
Dienstleistungszentrum
Öffnungszeiten
Mitarbeiter
Politik
Bürgermeister
Vizebürgermeister
Gemeindevorstand
Gemeinderat
Ausschüsse
Mandatsverteilung
Unsere
Gemeinde
Wissenswertes
Geschichte
Zahlen und Fakten
Städtepartnerschaft
Verdiente Persönlichkeiten
Fotogalerie
Kinder & Jugend
Ferienpass
Ferienbetreuungsangebote
Jugendtreff
Jugendtaxi
Studentenförderung
Bildungseinrichtungen
Pfarrcaritas-Kindergarten
Volksschule
Mittelschule
Schulische Tagesbetreuung
Landesmusikschule
Gesundheit und Soziales
Ärzte & Apotheken
Essen auf Rädern
Kur & Rehaeinrichtungen
Betreubares Wohnen
Alten- und Pflegeheim
Notrufnummern
Kultur
& Freizeit
Kultur
Veranstaltungen
Neue Veranstaltung
Veranstaltung ändern
Newsletter abonnieren
Veranstaltungszentrum ATRIUM
Abendmusik in der Magdalenabergkriche
Internationaler Musiksommer
Kirche & Pfarre
Vereine
Freizeit
Bad Schallerbacher Bibliothek
Gastronomie
Unterkünfte
Tourismusinfo Vitalwelt
Wirtschaft
Spielplätze
Sportstätten
Bad_Schallerbach_logo_neu
Sommer 4
Sommer 3
Sommer 2
Sommer 1
Kultur
& Freizeit
Kultur
Veranstaltungen
Neue Veranstaltung
Veranstaltung ändern
Newsletter abonnieren
Veranstaltungszentrum ATRIUM
Abendmusik in der Magdalenabergkriche
Internationaler Musiksommer
Kirche & Pfarre
Vereine
Freizeit
Bad Schallerbacher Bibliothek
Gastronomie
Unterkünfte
Tourismusinfo Vitalwelt
Wirtschaft
Spielplätze
Sportstätten
Zentrum > Kultur
& Freizeit > Kultur > Veranstaltungen > Veranstaltung ändern
Aufgrund Ihrer aktuellen Cookie Einstellungen kann der Inhalt nicht angezeigt werden. Klicken sie hier um Ihre Einstellungen anzupassen.
Kontakt
Bad Schallerbach
Rathausplatz 1
4701 Bad Schallerbach
+43 7249 485 550
gemeinde@bad-schallerbach.at
Wappen groß transparent
Öffnungszeiten
Öffnungszeiten
Mo 08:00 Uhr - 12:00 Uhr, 14:00 Uhr - 18:00 Uhr
Di, Mi, Fr 08:00 Uhr - 12:00 Uhr
Do 08:00 Uhr - 12:00 Uhr, 14:00 Uhr - 17:00 Uhr
Das nächste Mal geöffnet:
Mi, 17.08.2022 ab 08:00 Uhr
Ortsplan
ooe_karte_bad-schallerbach
Gem2Go
Logo
DatenschutzSitemapCookiesImpressumBarrierefreiheit
Sorry, that was my bad. I marked the worng infromation
I would like to know when the garbage is picked up - not the opening hours of the municipality
There is no login necessary. The page looks like this:
You have to choose a street and a housenumber to see the schedule.
I get the same reqeust as you - from my debug node.
Thats the sourcecode of the page:
Settings from the HTTP Node
That is not what is being returned form the http request, all i get is what i posted above, I think you will find that there is a separte javascript call to another url to return the table data, you will need to look in the developers tool in browser to find the correct url needed.
Once you find the correct url that returns the table the selector would be td.td_kal
I couldnt figure out either how the page requests that data ..
an alternative is to use a module called Puppeteer .. a tool based on Cromium browser that can accept commands programmatically .. mostly used for website functionality testing
We can use this npm module in a Function node ( functionExternalModules )
Test Flow:
[{"id":"7c406736d4aa3380","type":"inject","z":"54efb553244c241f","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":360,"y":4320,"wires":[["9fb94f39f2c85611"]]},{"id":"9fb94f39f2c85611","type":"function","z":"54efb553244c241f","name":"puppeteer","func":"node.status({fill:\"green\",shape:\"dot\",text:\"Processing...\"});\n\nconst browser = await puppeteer.launch({ headless: true });\nconst page = await browser.newPage();\nawait page.goto('https://www.bad-schallerbach.at/system/web/kalender.aspx?sprache=1&menuonr=226753868&typids=226758838,226758840,226758842,226758844/', {\n waitUntil: 'networkidle2' // <-- good practice to wait for page to fully load \n});\n\nawait page.click('.bemCookieOverlay__btn--save');\n\nawait page.waitForTimeout(3000)\n\nmsg.payload = await page.evaluate(() => {\n // @ts-ignore\n return Array.from(document.querySelectorAll(\".ris_table tr\")).map(el => el.innerText)\n})\n\n// make the data nice ;)\nmsg.payload = msg.payload.map(el => { return { \"DATUM\": el.split(\"\\n\")[0], \"TERMIN\": el.split(\"\\n\")[1] } })\n\nawait browser.close();\n\nnode.status({});\n\nreturn msg;\n\n\n","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[{"var":"puppeteer","module":"puppeteer"}],"x":550,"y":4320,"wires":[["9620fe8243f9691e"]]},{"id":"9620fe8243f9691e","type":"debug","z":"54efb553244c241f","name":"debug","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":770,"y":4320,"wires":[]}]
You are running an older version of Nodejs that is not supported by the module
Since you are on a PI you can run the official installation script with the --node14 parameter to update to a compatible version
bash <(curl -sL https://raw.githubusercontent.com/node-red/linux-installers/master/deb/update-nodejs-and-nodered) --node14
What you see in your browser is the html and JS and the browser runs the JS and updates the source and displays it..
The http request node does not run the JS, therefore the output does not have the returned ddata from any of the JS code.
@UnborN flow should do the trick as it can run the JS.
ah ok, i think i unterstand what u mean. Thanks for that.
I will update my nodejs and try it again
Thanks for the command, that was very helpful!
Now i get the following error msg
sudo apt-get install chromium-browser - was not helpful
I found this
here:
but it sticks
i have a spare raspberry pi but i havent tested it there yet
my tests were done on a windows system
you can try some of the suggestions in this article : Puppeteer on Raspbian
i dont know whats the problem, but it wont work.
[{"id":"7c406736d4aa3380","type":"inject","z":"f1c196de.aec348","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payloadType":"date","x":260,"y":860,"wires":[["9fb94f39f2c85611"]]},{"id":"9fb94f39f2c85611","type":"function","z":"f1c196de.aec348","name":"puppeteer","func":"node.status({fill:\"green\",shape:\"dot\",text:\"Processing...\"});\n\nconst browser = await puppeteer.launch({\n headless: true,\n executablePath: '/usr/bin/chromium-browser',\n args: ['--no-sandbox', '--disable-setuid-sandbox']\n});\nconst page = await browser.newPage();\nawait page.goto('https://www.bad-schallerbach.at/system/web/kalender.aspx?sprache=1&menuonr=226753868&typids=226758838,226758840,226758842,226758844/', {\n waitUntil: 'networkidle2' // <-- good practice to wait for page to fully load \n});\n\nawait page.click('.bemCookieOverlay__btn--save');\n\nawait page.waitForTimeout(3000)\n\nmsg.payload = await page.evaluate(() => {\n // @ts-ignore\n return Array.from(document.querySelectorAll(\".ris_table tr\")).map(el => el.innerText)\n})\n\n// make the data nice ;)\nmsg.payload = msg.payload.map(el => { return { \"DATUM\": el.split(\"\\n\")[0], \"TERMIN\": el.split(\"\\n\")[1] } })\n\nawait browser.close();\n\nnode.status({});\n\nreturn msg;\n\n\n","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[{"var":"puppeteer","module":"puppeteer"}],"x":450,"y":860,"wires":[["9620fe8243f9691e"]]},{"id":"9620fe8243f9691e","type":"debug","z":"f1c196de.aec348","name":"debug","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":670,"y":860,"wires":[]}]
Since the update i have another problem... node red autostart doesn't work anymore
sudo systemctl enable nodered.service
dont work anymore
and the next problem is that node red is maybe crashing after a few minutes
since the update i can only start node-red with ./node-red not with node-red-start
ok .. one thing at a time ...
sudo systemctl disable nodered.service
to disable the service until we figure whats wrong with that
run
which node-red
show us the result
also run
node-red
does it start manually ? show us the log ..
Starting Nodered manually was successful
I have used your mentioned cmd a 2nd time and now it seems to be working.
I can start it as a service and and autostart should now work again
ok .. the problem is that your node red is installed under nvm ?
which stands for Node Version Manager which is known to cause problems when running with node-red and updating with the official script.
I would recommend removing NVM (instructions here) but first make a backup of your flows and any related files from /home/pi/.node-red/
Then re-run the script to install NR properly.
regarding puppeteer .. i tried several things on my PI but i couldnt get cromium to run .. seems to be an issue .. i'll keep reading for a solution .. if there is one
I couldnt figure out either how the page requests that data ..
Need to set/accept a cookie first, same issue will exist with puppeteer.
Copyright OpenJS Foundation and Node-RED contributors. All rights reserved. The OpenJS Foundation has registered trademarks and uses trademarks. For a list of trademarks of the OpenJS Foundation, please see our Trademark Policy and Trademark List. Trademarks and logos not indicated on the list of OpenJS Foundation trademarks are trademarks™ or registered® trademarks of their respective holders. Use of them does not imply any affiliation with or endorsement by them.
The OpenJS Foundation | Terms of Use | Privacy Policy | OpenJS Foundation Bylaws | Trademark Policy | Trademark List | Cookie Policy
