# Web scraping issue

**URL:** <https://discourse.nodered.org/t/web-scraping-issue/37945>\
**Category:** General\
**Created:** [23 December 2020 21:20 UTC](https://discourse.nodered.org/t/web-scraping-issue/37945 "2020-12-23T21:20:14Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![Marty1982](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/marty1982/32/20162_2.png) [@Marty1982](https://discourse.nodered.org/u/Marty1982)\
**Post date:** [23 December 2020 21:20 UTC](https://discourse.nodered.org/t/web-scraping-issue/37945/1 "2020-12-23T21:20:14Z")

</div>

Hello all node-red maniacs!

I'm web scrapping a little bit and i found a rock in my shoe!.

I have this...

![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/b/1/b13f266b910b9a21de54d2383ddfd51d7af431b5.png)

And i want to get the value from that table..this is how it sees from the UI.  
 ![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/2/1/210cf35d8823a4b5865485f00e17f2e953e8a6e6.png)

So...in my node-red nodes i have this:

 ![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/0/3/0310e2cf0c180ec993a6dbdf0cad36c8bcd73805.png)

But i only got and empy value, i need to got the same value from what i sees on the UI.

This is how my node is configured to get that element information.

 ![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/5/4/54e5034e5e092a98331787bcb57f14df13013510.png)

Thanks a lot!!

---

<div class="post-metadata">

**Author:** ![Steve-Mcl](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/steve-mcl/32/4826_2.png) [@Steve-Mcl](https://discourse.nodered.org/u/Steve-Mcl)\
**Post date:** [23 December 2020 21:43 UTC](https://discourse.nodered.org/t/web-scraping-issue/37945/2 "2020-12-23T21:43:45Z")

</div>

what is the source of the web page? Can you share a link?

I suspect the value is not present in the DOM on load (but rather populated via javascript)

If you open devtools (F12) open the network tab & refresh the page, you might be able to see where it is coming from

This is how I help the last person with similar request...

> [@Web scraping to grab an items price, but coming back empty](https://discourse.nodered.org/t/web-scraping-to-grab-an-items-price-but-coming-back-empty/34894/4):
>
> As i thought, value is NOT present in the HTML (it is updated in a later background call to another URL) The actual URL is https://www.lowes.com/pd/1000698158/productdetail/2209/Guest so do a request to that URL, set the request node to JSON then access the values from msg.payload.productDetails["1000698158"].price.itemPrice

---

<div class="post-metadata">

**Author:** ![Marty1982](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/marty1982/32/20162_2.png) [@Marty1982](https://discourse.nodered.org/u/Marty1982)\
**Post date:** [23 December 2020 22:09 UTC](https://discourse.nodered.org/t/web-scraping-issue/37945/3 "2020-12-23T22:09:08Z")

</div>

The source is local, so i cannot share it here.

This is what i got from devtools.

 ![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/5/e/5e8155d75f7ca30a3f3d3e4b775f2f460ca92816.png)

---

<div class="post-metadata">

**Author:** ![Steve-Mcl](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/steve-mcl/32/4826_2.png) [@Steve-Mcl](https://discourse.nodered.org/u/Steve-Mcl)\
**Post date:** [23 December 2020 22:31 UTC](https://discourse.nodered.org/t/web-scraping-issue/37945/4 "2020-12-23T22:31:01Z")

</div>

ok, so first things first. The HTTP Request node grabs what you ask for. In this case you request a WEB page & thats what you get. BUT - the HTTP request node is NOT a browser so it doesnt do Javascript \<\< keep that in mind.

...

Using an example, if you look at the response (like the HTTP Request node would get) you can see this page has no prices in the HTML (they get populated by JS later on)

When I select the HTML (usually the first item in the `network` list) - we can see there is nothing inside the DIV...

![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/9/6/96c5df01d113733d5fa30f0cd025467e185b08fb.png)

  

However, if i look at the `Elements` tab, the `div` actually has content...

 ![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/9/0/90427d075f138d573a0e1ee5f927b3ef2fe79f92.png)

  

* * *

So have sift through the network transfers & see if you can spot the value you are interested in. Hopefully it is in the HTML if not, then see if you can find the value somewhere else in the network items.

---

<div class="post-metadata">

**Author:** ![Marty1982](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/marty1982/32/20162_2.png) [@Marty1982](https://discourse.nodered.org/u/Marty1982)\
**Post date:** [23 December 2020 22:54 UTC](https://discourse.nodered.org/t/web-scraping-issue/37945/5 "2020-12-23T22:54:43Z")

</div>

Ok, i see that the network option shows that the content is empty, so maybe there it comes my issue and i cannot get the value that im looking for (program version).

 ![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/a/f/af77784f927a3c56909622365d74d111bfea0ac0.png)

Could it be that?

---

<div class="post-metadata">

**Author:** ![Steve-Mcl](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/steve-mcl/32/4826_2.png) [@Steve-Mcl](https://discourse.nodered.org/u/Steve-Mcl)\
**Post date:** [23 December 2020 22:56 UTC](https://discourse.nodered.org/t/web-scraping-issue/37945/6 "2020-12-23T22:56:29Z")

</div>

> [@Marty1982](#):
>
> Could it be that?

Yes. that is why you cannot get it. The value is populated later (somehow) - most likely something in one of those scripts. Have you looked at them? Click them then click "response" tab - see if you can spot the info you are interested in.

---

<div class="post-metadata">

**Author:** ![Marty1982](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/marty1982/32/20162_2.png) [@Marty1982](https://discourse.nodered.org/u/Marty1982)\
**Post date:** [23 December 2020 23:39 UTC](https://discourse.nodered.org/t/web-scraping-issue/37945/7 "2020-12-23T23:39:56Z")

</div>

Uhmm ok..now i get it.

Inside de scripts there is nothing that brings me the info that i want.  
So maybe there is a matter of wait for the information to come up while the UI is loading?

Is there a node that can help me out?

---

<div class="post-metadata">

**Author:** ![Steve-Mcl](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/steve-mcl/32/4826_2.png) [@Steve-Mcl](https://discourse.nodered.org/u/Steve-Mcl)\
**Post date:** [23 December 2020 23:43 UTC](https://discourse.nodered.org/t/web-scraping-issue/37945/8 "2020-12-23T23:43:09Z")

</div>

Does this firmware have MQTT? Perhaps you can just have the value sent to your node-red? If you dont know, then search `sonoff tasmota MQTT` (I assume it does as it stands for **T** heo- **A** rends- **S** onoff- **M** QTT- **OTA** )

Or perhaps it has a HTTP API where you can ask for it - search `sonoff tasmota REST api`

---

<div class="post-metadata">

**Author:** ![Marty1982](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/marty1982/32/20162_2.png) [@Marty1982](https://discourse.nodered.org/u/Marty1982)\
**Post date:** [23 December 2020 23:45 UTC](https://discourse.nodered.org/t/web-scraping-issue/37945/9 "2020-12-23T23:45:58Z")

</div>

uhm ok, i should look for that link with homeassistant or mqtt so i can get the info.

---

<div class="post-metadata">

**Author:** ![Marty1982](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/marty1982/32/20162_2.png) [@Marty1982](https://discourse.nodered.org/u/Marty1982)\
**Post date:** [23 December 2020 23:58 UTC](https://discourse.nodered.org/t/web-scraping-issue/37945/10 "2020-12-23T23:58:20Z")

</div>

So you mean parse a Json so i can get that info?.

---

<div class="post-metadata">

**Author:** ![Steve-Mcl](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/steve-mcl/32/4826_2.png) [@Steve-Mcl](https://discourse.nodered.org/u/Steve-Mcl)\
**Post date:** [24 December 2020 00:16 UTC](https://discourse.nodered.org/t/web-scraping-issue/37945/11 "2020-12-24T00:16:11Z")

</div>

> [@Marty1982](#):
>
> So you mean parse a Json so i can get that info?

If you get what you want in JSON then you can just grab the value directly

There’s a great page in the docs that will explain how to use the debug panel to find the right path to any data item.

[https://nodered.org/docs/user-guide/messages](https://nodered.org/docs/user-guide/messages)

---

<div class="post-metadata">

**Author:** ![Marty1982](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/marty1982/32/20162_2.png) [@Marty1982](https://discourse.nodered.org/u/Marty1982)\
**Post date:** [28 December 2020 17:54 UTC](https://discourse.nodered.org/t/web-scraping-issue/37945/12 "2020-12-28T17:54:45Z")

</div>

Thanks Steve...

Somehow i cannot see that information i need to get the thing i want.  
I downloaded the puppetter node but i dont know how to use it either.

Im kinda stuck here...im still looking some information.

Regards!

---

<div class="post-metadata">

**Author:** ![Marty1982](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/marty1982/32/20162_2.png) [@Marty1982](https://discourse.nodered.org/u/Marty1982)\
**Post date:** [11 January 2021 19:54 UTC](https://discourse.nodered.org/t/web-scraping-issue/37945/13 "2021-01-11T19:54:04Z")

</div>

Hi all!

I returning with another web scrapping issue...  
i want to obtain this result stored inside a table but i have no luck.

[http://ota.tasmota.com/tasmota/release/](http://ota.tasmota.com/tasmota/release/)

 ![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/e/9/e9a83f62bcdff937472c63fdab95374240f5b441.png)

I have this flow.

Thanks a lot in advance.

```auto
[{"id":"6e1953b6.ceb55c","type":"html","z":"13e3e683.083059","name":"Check Web Site Tasmota Version","property":"","outproperty":"","tag":"<td>9.1.0</td>","ret":"text","as":"single","x":480,"y":960,"wires":[["3e9f13cb.37838c"]]},{"id":"c087b2bc.6d9e9","type":"http request","z":"13e3e683.083059","name":"Get thehackbox.org/tasmota/release/","method":"GET","ret":"txt","paytoqs":"ignore","url":"http://ota.tasmota.com/tasmota/release/","tls":"","persist":false,"proxy":"","authType":"","x":490,"y":920,"wires":[["6e1953b6.ceb55c"]]},{"id":"2fb8e118.b3cace","type":"comment","z":"13e3e683.083059","name":"Check Tasmota Web Version","info":"","x":160,"y":840,"wires":[]},{"id":"f222f5ec.174c98","type":"delay","z":"13e3e683.083059","name":"","pauseType":"delay","timeout":"2","timeoutUnits":"seconds","rate":"1","nbRateUnits":"1","rateUnits":"second","randomFirst":"1","randomLast":"5","randomUnits":"seconds","drop":false,"x":400,"y":880,"wires":[["c087b2bc.6d9e9"]]},{"id":"3de72ed6.f82a52","type":"inject","z":"13e3e683.083059","name":"Repeat every 15 days","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"1296000","crontab":"","once":false,"onceDelay":"","topic":"","payload":"2","payloadType":"num","x":170,"y":880,"wires":[["f222f5ec.174c98"]]},{"id":"3e9f13cb.37838c","type":"debug","z":"13e3e683.083059","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":750,"y":960,"wires":[]}]

```

The idea is to store that 9.0.1 or whatever value into a global variable so i can use it later on another nodes.

---

<div class="post-metadata">

**Author:** ![dceejay](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/dceejay/32/38_2.png) [@dceejay](https://discourse.nodered.org/u/dceejay)\
**Post date:** [11 January 2021 22:23 UTC](https://discourse.nodered.org/t/web-scraping-issue/37945/14 "2021-01-11T22:23:47Z")

</div>

This should get you most of the way there

```auto
[{"id":"3b5babc1.a63754","type":"inject","z":"8bfe9025.7255","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":120,"y":1260,"wires":[["3f9fa0bd.3c963"]]},{"id":"3f9fa0bd.3c963","type":"http request","z":"8bfe9025.7255","name":"","method":"GET","ret":"txt","paytoqs":"ignore","url":"http://ota.tasmota.com/tasmota/release/","tls":"","persist":false,"proxy":"","authType":"","x":270,"y":1260,"wires":[["1e52ea36.0f7356"]]},{"id":"1e52ea36.0f7356","type":"html","z":"8bfe9025.7255","name":"","property":"payload","outproperty":"payload","tag":"tr","ret":"html","as":"multi","x":410,"y":1260,"wires":[["54759709.b0cf28"]]},{"id":"5ec524fe.b237fc","type":"debug","z":"8bfe9025.7255","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":610,"y":1340,"wires":[]},{"id":"54759709.b0cf28","type":"html","z":"8bfe9025.7255","name":"","property":"payload","outproperty":"payload","tag":"td","ret":"html","as":"single","x":170,"y":1340,"wires":[["bb6e79ef.6cb5b8"]]},{"id":"bb6e79ef.6cb5b8","type":"switch","z":"8bfe9025.7255","name":"","property":"payload[0]","propertyType":"msg","rules":[{"t":"cont","v":"tasmota.bin","vt":"str"}],"checkall":"true","repair":false,"outputs":1,"x":310,"y":1340,"wires":[["5ec524fe.b237fc"]]}]

```

---

<div class="post-metadata">

**Author:** ![Marty1982](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/marty1982/32/20162_2.png) [@Marty1982](https://discourse.nodered.org/u/Marty1982)\
**Post date:** [11 January 2021 23:46 UTC](https://discourse.nodered.org/t/web-scraping-issue/37945/15 "2021-01-11T23:46:52Z")

</div>

Great approach!!

How can i get only this value?

 ![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/7/8/78aef463daf97a3ac2e16e3faaaf0a95abcc3539.png)

---

<div class="post-metadata">

**Author:** ![Steve-Mcl](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/steve-mcl/32/4826_2.png) [@Steve-Mcl](https://discourse.nodered.org/u/Steve-Mcl)\
**Post date:** [12 January 2021 00:06 UTC](https://discourse.nodered.org/t/web-scraping-issue/37945/16 "2021-01-12T00:06:20Z")

</div>

There’s a great page in the docs that will explain how to use the debug panel to find the right path to any data item.

[https://nodered.org/docs/user-guide/messages](https://nodered.org/docs/user-guide/messages)

Too be clear, read the part about the copy path button. Then you can paste that into a change node to move out copy it wherever you wish. Or paste it into a UI node or however you need to use it

---

<div class="post-metadata">

**Author:** ![kuema](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/kuema/32/6542_2.png) [@kuema](https://discourse.nodered.org/u/kuema)\
**Post date:** [12 January 2021 07:47 UTC](https://discourse.nodered.org/t/web-scraping-issue/37945/17 "2021-01-12T07:47:14Z")

</div>

With Tasmota, you can easily request the status via MQTT or web requests. No need to scrap the web interface.

Try sending the `status` command, you should get back a JSON  
[https://tasmota.github.io/docs/Commands/#with-web-requests](https://tasmota.github.io/docs/Commands/#with-web-requests)

The command list: [Commands - Tasmota](https://tasmota.github.io/docs/Commands/#management)

---

<div class="post-metadata">

**Author:** ![dceejay](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/dceejay/32/38_2.png) [@dceejay](https://discourse.nodered.org/u/dceejay)\
**Post date:** [12 January 2021 09:42 UTC](https://discourse.nodered.org/t/web-scraping-issue/37945/18 "2021-01-12T09:42:09Z")

</div>

I think this is about checking/grabbing latest versions of code from the web site rather than actually running devices... but i may be wrong.

---

<div class="post-metadata">

**Author:** ![kuema](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/kuema/32/6542_2.png) [@kuema](https://discourse.nodered.org/u/kuema)\
**Post date:** [12 January 2021 09:59 UTC](https://discourse.nodered.org/t/web-scraping-issue/37945/19 "2021-01-12T09:59:55Z")

</div>

Both, actually. At least as I understood it. 😀

First, the Tasmota device's web interface, hence the suggestion grabbing it via the provided API.

Second, the current version presented by the Github project page. Does Github offer any machine-readable API for that, maybe? 🤔

---

<div class="post-metadata">

**Author:** ![zenofmud](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/zenofmud/32/316_2.png) [@zenofmud](https://discourse.nodered.org/u/zenofmud)\
**Post date:** [12 January 2021 10:50 UTC](https://discourse.nodered.org/t/web-scraping-issue/37945/20 "2021-01-12T10:50:44Z")

</div>

This sub flow is how I've been grabbing the latest release of Tasmota in my ['ET Display Home' a flow to help manage your ESPeasy, Sonoff/Tasmota and Homie flashed devices (now on GitHub)](https://discourse.nodered.org/t/et-display-home-a-flow-to-help-manage-your-espeasy-sonoff-tasmota-and-homie-flashed-devices-now-on-github/1893) flow since I first published the flow back in 2018

```auto
[{"id":"fc3e43d5.47c3e","type":"http request","z":"fa6ef639.63d4e8","name":"Get SonOff releases","method":"GET","ret":"txt","paytoqs":false,"url":"","tls":"","proxy":"","authType":"","x":600,"y":220,"wires":[["3a4c8bfe.cdc6ac"]]},{"id":"a404b866.174848","type":"change","z":"fa6ef639.63d4e8","name":"","rules":[{"t":"set","p":"url","pt":"msg","to":"https://github.com/arendst/Sonoff-Tasmota/releases","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":570,"y":180,"wires":[["fc3e43d5.47c3e"]]},{"id":"3aa39bda.588e14","type":"inject","z":"fa6ef639.63d4e8","name":"Check for new Tasmota release","topic":"","payload":"1","payloadType":"str","repeat":"21600","crontab":"","once":true,"onceDelay":0.1,"x":640,"y":100,"wires":[["a404b866.174848"]]},{"id":"3a4c8bfe.cdc6ac","type":"function","z":"fa6ef639.63d4e8","name":"extract latest version","func":"var n = msg.payload.indexOf('<a href=\"/arendst/Tasmota/tree/');\n//node.warn(\"n=\" + n);\nvar str = '<a href=\"/arendst/Tasmota/tree/';\nvar l = str.length;\nn += l;\n//node.warn(\"nl=\" + n);\n//str = msg.payload.substring(n, 13);\n//node.warn(\"str=\" + str);\nmsg.payload = msg.payload.substring(n+1, n+6);\nreturn msg;","outputs":1,"noerr":0,"x":600,"y":260,"wires":[["5b7daebe.09d8f8"]]},{"id":"954daab5.89b158","type":"sqlite","z":"fa6ef639.63d4e8","mydb":"47186c71.905e34","sqlquery":"fixed","sql":"delete from firmware where platform = \"tasmota\";","name":"delete tasmota row","x":590,"y":140,"wires":[["a404b866.174848"]]},{"id":"5b7daebe.09d8f8","type":"change","z":"fa6ef639.63d4e8","name":"store global","rules":[{"t":"set","p":"ETDH.tasmota","pt":"global","to":"payload","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":810,"y":260,"wires":[[]]},{"id":"47186c71.905e34","type":"sqlitedb","z":"","db":"/home/pi/esp8266.db"}]

```

[Next page](https://discourse.nodered.org/t/web-scraping-issue/37945.md?page=2)