# Search on webpage and retrieve data

**URL:** https://discourse.nodered.org/t/search-on-webpage-and-retrieve-data/52091
**Category:** General
**Tags:** http-request
**Created:** [11 October 2021 11:54 UTC](https://discourse.nodered.org/t/search-on-webpage-and-retrieve-data/52091 "2021-10-11T11:54:12Z")
**Posts on this page:** 1
**Showing post:** 3

<div class="post-metadata">

### Author: ![UnborN](https://avatars.discourse-cdn.com/v4/letter/u/4491bb/32.png) [@UnborN](https://discourse.nodered.org/u/UnborN)
#### Post date: [11 October 2021 17:54 UTC](https://discourse.nodered.org/t/search-on-webpage-and-retrieve-data/52091/3 "2021-10-11T17:54:06Z")

</div>

If you check your browsers developer tools network tab, you'll see that the browser first makes some sort of authentication request to get a JWT access token.  
then it makes the actual request to get the json data.

 ![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/5/d/5d6b7c59352dd5bacf3c4e9bee6afc335c4ba9b0.png)

You can emulate the whole request process with a flow like the example below :

 ![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/6/8/680112a40cd60556de0ac6a27da6f71e6dedee91.png)

```auto
[{"id":"1c44140cb8acf2cb","type":"inject","z":"5847b7aa62131d37","name":"","props":[{"p":"payload"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payloadType":"date","x":160,"y":640,"wires":[["6da13aa2886382d1"]]},{"id":"b7762aa66de6a566","type":"http request","z":"5847b7aa62131d37","name":"auth req","method":"POST","ret":"txt","paytoqs":"body","url":"https://authwebservice.cardiff.gov.uk/AuthenticationWebService.asmx?op=GetJW","tls":"","persist":true,"proxy":"","authType":"","senderr":false,"x":400,"y":700,"wires":[["682b5c419c834422"]]},{"id":"33fc229af1bd9858","type":"debug","z":"5847b7aa62131d37","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":630,"y":780,"wires":[]},{"id":"6da13aa2886382d1","type":"function","z":"5847b7aa62131d37","name":"","func":"msg.headers = {\n \"Content-Length\": \"284\",\n \"Accept-Encoding\": \"gzip, deflate, br\",\n \"Referer\": \"https://www.cardiff.gov.uk/\",\n \"Origin\": \"https://www.cardiff.gov.uk\",\n \"Content-Type\": `text/xml; charset=\"utf-8\"`,\n \"Sec-Fetch-Dest\": \"empty\",\n \"Sec-Fetch-Mode\": \"no-cors\",\n \"Sec-Fetch-Site\": \"same-site\",\n \"Pragma\": \"no-cache\",\n \"Cache-Control\": \"no-cache\"\n}\n\nmsg.payload = `<?xml version='1.0' encoding='utf-8'?><soap:Envelope xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance' xmlns:xsd='http://www.w3.org/2001/XMLSchema' xmlns:soap='http://schemas.xmlsoap.org/soap/envelope/'><soap:Body><GetJWT xmlns='http://tempuri.org/' /></soap:Body></soap:Envelope>`\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":260,"y":700,"wires":[["b7762aa66de6a566"]]},{"id":"682b5c419c834422","type":"xml","z":"5847b7aa62131d37","name":"","property":"payload","attr":"","chr":"","x":530,"y":700,"wires":[["24b162b725bfb0a2"]]},{"id":"24b162b725bfb0a2","type":"function","z":"5847b7aa62131d37","name":"","func":"let access_token = JSON.parse(msg.payload[\"soap:Envelope\"][\"soap:Body\"][0].GetJWTResponse[0].GetJWTResult).access_token\n\nmsg.payload = {\n \"systemReference\": \"web\",\n \"language\": \"eng\",\n \"uprn\": 100100073989 // code for address ?? you need to find it from browser\n}\n\nmsg.headers = {\n \"Authorization\": `Bearer ${access_token}`\n}\n\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":260,"y":780,"wires":[["b091cd6df0893112"]]},{"id":"b091cd6df0893112","type":"http request","z":"5847b7aa62131d37","name":"WasteCollection req","method":"POST","ret":"obj","paytoqs":"body","url":"https://api.cardiff.gov.uk/WasteManagement/api/WasteCollection","tls":"","persist":false,"proxy":"","authType":"","senderr":false,"x":440,"y":780,"wires":[["33fc229af1bd9858"]]}]

```

The data is in `msg.payload.collectionWeeks`

One small problem is that you have to make a request of your real postcode and address (from the browser) and see in its dev tools the `uprn` code. Use that in the Function node to get the correct data relating to your address.

![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/b/1/b129bc7e6018586ff39d4b87cd00aabff3d1021b.png)

---

_[View the full topic](https://discourse.nodered.org/t/search-on-webpage-and-retrieve-data/52091)._