# Help with a webscrap

**URL:** <https://discourse.nodered.org/t/help-with-a-webscrap/47694>\
**Category:** General\
**Created:** [26 June 2021 09:18 UTC](https://discourse.nodered.org/t/help-with-a-webscrap/47694 "2021-06-26T09:18:27Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![leftymuller](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/leftymuller/32/102326_2.png) [@leftymuller](https://discourse.nodered.org/u/leftymuller)\
**Post date:** [26 June 2021 09:18 UTC](https://discourse.nodered.org/t/help-with-a-webscrap/47694/1 "2021-06-26T09:18:27Z")

</div>

Hello All!

Can somebody lend me a hand.. HTML is not my strong point

I want to get the oil price from this website [Crude Oil Prices - Brent And WTI Oil Price Per Barrel (oilcrudeprice.com)](https://www.oilcrudeprice.com/)

I can find the figure I need  
 ![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/7/d/7d9820309ead62257b2cc46bac1753f59083219f.png)

I did a flow and told it to get H3 and it gets this  
 ![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/9/6/96829781283acee081eb6276a979249d2c2f1bfe.png)

 ![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/0/c/0ca1ad440f96539c7a167dbe7f8cc79f4174f873.png)

Any hints or suggestions would be great appreciated!

---

<div class="post-metadata">

**Author:** ![Steve-Mcl](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/steve-mcl/32/4826_2.png) [@Steve-Mcl](https://discourse.nodered.org/u/Steve-Mcl)\
**Post date:** [26 June 2021 09:42 UTC](https://discourse.nodered.org/t/help-with-a-webscrap/47694/2 "2021-06-26T09:42:01Z")

</div>

Does this help...

```auto
[{"id":"0488087db4598ce8","type":"http request","z":"bec69dbd.8d622","name":"","method":"GET","ret":"txt","paytoqs":"ignore","url":"https://www.oilcrudeprice.com/","tls":"","persist":false,"proxy":"","authType":"","x":840,"y":1440,"wires":[["4645e9233d951348"]]},{"id":"4645e9233d951348","type":"html","z":"bec69dbd.8d622","name":"","property":"payload","outproperty":"payload","tag":".oilcov","ret":"html","as":"single","x":1010,"y":1440,"wires":[["d5ca7512d8d6dd76"]]},{"id":"b9667d098f36319d","type":"inject","z":"bec69dbd.8d622","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payloadType":"date","x":660,"y":1440,"wires":[["0488087db4598ce8"]]},{"id":"960445cec435fba3","type":"debug","z":"bec69dbd.8d622","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":1450,"y":1440,"wires":[]},{"id":"f7dfd696c7b9d979","type":"html","z":"bec69dbd.8d622","name":"","property":"payload","outproperty":"payload","tag":"p","ret":"text","as":"single","x":1290,"y":1440,"wires":[["960445cec435fba3"]]},{"id":"d5ca7512d8d6dd76","type":"split","z":"bec69dbd.8d622","name":"","splt":"\\n","spltType":"str","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"","x":1150,"y":1440,"wires":[["f7dfd696c7b9d979"]]}]

```

![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/e/4/e4ecb4cb15174d8cb8d334084c09245033b88f16.png)

Using a function node or JSONata in a change node you could easily extract the values from this ↑ output.

---

<div class="post-metadata">

**Author:** ![leftymuller](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/leftymuller/32/102326_2.png) [@leftymuller](https://discourse.nodered.org/u/leftymuller)\
**Post date:** [26 June 2021 09:47 UTC](https://discourse.nodered.org/t/help-with-a-webscrap/47694/3 "2021-06-26T09:47:26Z")

</div>

Absolutely! Well ive learned something today 🙂

Thanks mate.. appreciated

---

<div class="post-metadata">

**Author:** ![E1cid](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/e1cid/32/77971_2.png) [@E1cid](https://discourse.nodered.org/u/E1cid)\
**Post date:** [26 June 2021 09:48 UTC](https://discourse.nodered.org/t/help-with-a-webscrap/47694/4 "2021-06-26T09:48:19Z")

</div>

There is also a table at the bottom of the page which makes it easier to grab all the info  
e.g.

```auto
[{"id":"5096c6f0.4b0b88","type":"inject","z":"c74669a0.6a34f8","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":140,"y":1860,"wires":[["3ad9af0c.91263"]]},{"id":"3ad9af0c.91263","type":"http request","z":"c74669a0.6a34f8","name":"","method":"GET","ret":"txt","paytoqs":"ignore","url":"https://www.oilcrudeprice.com/","tls":"","persist":false,"proxy":"","authType":"","x":350,"y":1860,"wires":[["62932699.905cf"]]},{"id":"62932699.905cf","type":"html","z":"c74669a0.6a34f8","name":"","property":"payload","outproperty":"payload","tag":"tbody > tr > td","ret":"text","as":"single","x":560,"y":1860,"wires":[["896fbba5.052b98"]]},{"id":"896fbba5.052b98","type":"debug","z":"c74669a0.6a34f8","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":610,"y":1640,"wires":[]}]

```

returns

```auto
["Brent Oil","$76.18","+0.62","+0.82%","06/25/21","$75.61","$75.56","$74.95","$76.21","WTI Oil","$74.05","+0.75","+1.02%","06/25/21","$73.32","$73.3","$72.85","$74.18","Natural Gas","$3.520","+0.083","+2.41%","06/25/21","$3.439","$3.437","$3.43","$3.538","Gold","$1777.8","+1.1","+0.06%","06/25/21","$1775.7","$1776.7","$1773.6","$1791","Silver","$26.087","+0.037","+0.14%","06/25/21","$26.01","$26.05","$25.97","$26.38","Platinum","$1103.6","+9.7","+0.89%","06/25/21","$1093","$1093.9","$1091.3","$1111.2","Palladium","$2637.20","-6.70","-0.25%","06/25/21","$2634.5","$2643.9","$2612.5","$2660.5"]

```

---

<div class="post-metadata">

**Author:** ![leftymuller](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/leftymuller/32/102326_2.png) [@leftymuller](https://discourse.nodered.org/u/leftymuller)\
**Post date:** [11 July 2021 04:40 UTC](https://discourse.nodered.org/t/help-with-a-webscrap/47694/5 "2021-07-11T04:40:02Z")

</div>

So ive had another go at scraping

This is the data from the webbrowser  
 ![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/a/5/a590654f80d2eb352616eed8c1c54163247ae0a2.png)

with a HTML node with this

 ![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/3/a/3a6153c7f50a4e33aeabd5d507f63183aac2c2aa.png)

and i get  
 ![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/c/2/c24e259c4ba31d9eb28f691a1728fcaf8ec76df5.png)

where am I going wrong? Appreciate any help... im a HTML newb

here is the flow

```auto
[{"id":"eb3d7fa.90ab38","type":"tab","label":"test page","disabled":false,"info":""},{"id":"a2afc3f2.206ba","type":"http request","z":"eb3d7fa.90ab38","name":"","method":"GET","ret":"txt","paytoqs":"ignore","url":"https://bws.com.au/product/27390/jack-daniel-s-old-no-7-tennessee-whiskey-1l","tls":"8320344.08ef5c8","persist":false,"proxy":"","authType":"","x":270,"y":140,"wires":[["96680850.bb9468","f5bc8da1.dac12"]]},{"id":"9bfb874a.3124a8","type":"inject","z":"eb3d7fa.90ab38","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"780","crontab":"","once":true,"onceDelay":"5","topic":"","payload":"","payloadType":"date","x":100,"y":140,"wires":[["a2afc3f2.206ba"]]},{"id":"96680850.bb9468","type":"html","z":"eb3d7fa.90ab38","name":"","property":"payload","outproperty":"payload","tag":"div.product-detail_controls-col.ng-scope > h1","ret":"text","as":"single","x":410,"y":260,"wires":[["5a5860a4.7115f"]]},{"id":"6291cb18.38b9b4","type":"ui_media","z":"eb3d7fa.90ab38","group":"49798262.df9dec","name":"","width":1,"height":2,"order":2,"category":"Logo","file":"whiskey-bottle-png-11.png","layout":"adjust","showcontrols":true,"loop":true,"onstart":false,"scope":"local","tooltip":"","x":300,"y":60,"wires":[[]]},{"id":"5a5860a4.7115f","type":"debug","z":"eb3d7fa.90ab38","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":670,"y":140,"wires":[]},{"id":"f5bc8da1.dac12","type":"debug","z":"eb3d7fa.90ab38","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":680,"y":200,"wires":[]},{"id":"8320344.08ef5c8","type":"tls-config","name":"","cert":"","key":"","ca":"","certname":"","keyname":"","caname":"","servername":"","verifyservercert":false},{"id":"49798262.df9dec","type":"ui_group","name":"Oil Price","tab":"12e195b2.3de9fa","order":6,"disp":false,"width":4,"collapse":false},{"id":"12e195b2.3de9fa","type":"ui_tab","name":"Cameras","icon":"mi-videogame_asset","order":1,"disabled":false,"hidden":false}]

```

---

<div class="post-metadata">

**Author:** ![UnborN](https://avatars.discourse-cdn.com/v4/letter/u/4491bb/32.png) [@UnborN](https://discourse.nodered.org/u/UnborN)\
**Post date:** [11 July 2021 05:18 UTC](https://discourse.nodered.org/t/help-with-a-webscrap/47694/6 "2021-07-11T05:18:12Z")

</div>

Hi .. what information are you trying to get ? the price ?

if you follow the requests made by your browser using its _developer tools_, you will notice a request to an api that returns json with all the details of the product based on its id.

 ![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/2/5/25f3c07982f3471e8eb34789c0cb46e217a2d97b.png)

From there you can pick up any information you need from the product more easily.

**Test flow :**

```auto
[{"id":"a2afc3f2.206ba","type":"http request","z":"eb3d7fa.90ab38","name":"","method":"GET","ret":"obj","paytoqs":"ignore","url":"https://api.bws.com.au/apis/ui/Product/27390","tls":"","persist":false,"proxy":"","authType":"","x":490,"y":140,"wires":[["f5bc8da1.dac12"]]},{"id":"9bfb874a.3124a8","type":"inject","z":"eb3d7fa.90ab38","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":"5","topic":"","payloadType":"date","x":310,"y":140,"wires":[["a2afc3f2.206ba"]]},{"id":"f5bc8da1.dac12","type":"debug","z":"eb3d7fa.90ab38","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":710,"y":140,"wires":[]}]

```

---

<div class="post-metadata">

**Author:** ![leftymuller](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/leftymuller/32/102326_2.png) [@leftymuller](https://discourse.nodered.org/u/leftymuller)\
**Post date:** [11 July 2021 06:30 UTC](https://discourse.nodered.org/t/help-with-a-webscrap/47694/7 "2021-07-11T06:30:08Z")

</div>

Hi!  
Yep Price.

wow that makes it so much easier then webscraping!

Thanks so much... I have that working now 🙂

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/1X/d073cd938eafa2e558d7c2cd59003b3ef4963033.png) [@system](https://discourse.nodered.org/u/system)\
**Post date:** [25 July 2021 06:31 UTC](https://discourse.nodered.org/t/help-with-a-webscrap/47694/8 "2021-07-25T06:31:05Z")

</div>

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.