Help with a webscrap

Hello All!

Can somebody lend me a hand.. HTML is not my strong point

I want to get the oil price from this website Crude Oil Prices - Brent And WTI Oil Price Per Barrel (oilcrudeprice.com)

I can find the figure I need
image

I did a flow and told it to get H3 and it gets this
image

Any hints or suggestions would be great appreciated!

Does this help...

[{"id":"0488087db4598ce8","type":"http request","z":"bec69dbd.8d622","name":"","method":"GET","ret":"txt","paytoqs":"ignore","url":"https://www.oilcrudeprice.com/","tls":"","persist":false,"proxy":"","authType":"","x":840,"y":1440,"wires":[["4645e9233d951348"]]},{"id":"4645e9233d951348","type":"html","z":"bec69dbd.8d622","name":"","property":"payload","outproperty":"payload","tag":".oilcov","ret":"html","as":"single","x":1010,"y":1440,"wires":[["d5ca7512d8d6dd76"]]},{"id":"b9667d098f36319d","type":"inject","z":"bec69dbd.8d622","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payloadType":"date","x":660,"y":1440,"wires":[["0488087db4598ce8"]]},{"id":"960445cec435fba3","type":"debug","z":"bec69dbd.8d622","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":1450,"y":1440,"wires":[]},{"id":"f7dfd696c7b9d979","type":"html","z":"bec69dbd.8d622","name":"","property":"payload","outproperty":"payload","tag":"p","ret":"text","as":"single","x":1290,"y":1440,"wires":[["960445cec435fba3"]]},{"id":"d5ca7512d8d6dd76","type":"split","z":"bec69dbd.8d622","name":"","splt":"\\n","spltType":"str","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"","x":1150,"y":1440,"wires":[["f7dfd696c7b9d979"]]}]

image

Using a function node or JSONata in a change node you could easily extract the values from this ↑ output.

1 Like

Absolutely! Well ive learned something today :slight_smile:

Thanks mate.. appreciated

There is also a table at the bottom of the page which makes it easier to grab all the info
e.g.

[{"id":"5096c6f0.4b0b88","type":"inject","z":"c74669a0.6a34f8","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":140,"y":1860,"wires":[["3ad9af0c.91263"]]},{"id":"3ad9af0c.91263","type":"http request","z":"c74669a0.6a34f8","name":"","method":"GET","ret":"txt","paytoqs":"ignore","url":"https://www.oilcrudeprice.com/","tls":"","persist":false,"proxy":"","authType":"","x":350,"y":1860,"wires":[["62932699.905cf"]]},{"id":"62932699.905cf","type":"html","z":"c74669a0.6a34f8","name":"","property":"payload","outproperty":"payload","tag":"tbody > tr  > td","ret":"text","as":"single","x":560,"y":1860,"wires":[["896fbba5.052b98"]]},{"id":"896fbba5.052b98","type":"debug","z":"c74669a0.6a34f8","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":610,"y":1640,"wires":[]}]

returns

["Brent Oil","$76.18","+0.62","+0.82%","06/25/21","$75.61","$75.56","$74.95","$76.21","WTI Oil","$74.05","+0.75","+1.02%","06/25/21","$73.32","$73.3","$72.85","$74.18","Natural Gas","$3.520","+0.083","+2.41%","06/25/21","$3.439","$3.437","$3.43","$3.538","Gold","$1777.8","+1.1","+0.06%","06/25/21","$1775.7","$1776.7","$1773.6","$1791","Silver","$26.087","+0.037","+0.14%","06/25/21","$26.01","$26.05","$25.97","$26.38","Platinum","$1103.6","+9.7","+0.89%","06/25/21","$1093","$1093.9","$1091.3","$1111.2","Palladium","$2637.20","-6.70","-0.25%","06/25/21","$2634.5","$2643.9","$2612.5","$2660.5"]
1 Like

So ive had another go at scraping

This is the data from the webbrowser
image

with a HTML node with this

and i get
image

where am I going wrong? Appreciate any help... im a HTML newb

here is the flow

[{"id":"eb3d7fa.90ab38","type":"tab","label":"test page","disabled":false,"info":""},{"id":"a2afc3f2.206ba","type":"http request","z":"eb3d7fa.90ab38","name":"","method":"GET","ret":"txt","paytoqs":"ignore","url":"https://bws.com.au/product/27390/jack-daniel-s-old-no-7-tennessee-whiskey-1l","tls":"8320344.08ef5c8","persist":false,"proxy":"","authType":"","x":270,"y":140,"wires":[["96680850.bb9468","f5bc8da1.dac12"]]},{"id":"9bfb874a.3124a8","type":"inject","z":"eb3d7fa.90ab38","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"780","crontab":"","once":true,"onceDelay":"5","topic":"","payload":"","payloadType":"date","x":100,"y":140,"wires":[["a2afc3f2.206ba"]]},{"id":"96680850.bb9468","type":"html","z":"eb3d7fa.90ab38","name":"","property":"payload","outproperty":"payload","tag":"div.product-detail_controls-col.ng-scope > h1","ret":"text","as":"single","x":410,"y":260,"wires":[["5a5860a4.7115f"]]},{"id":"6291cb18.38b9b4","type":"ui_media","z":"eb3d7fa.90ab38","group":"49798262.df9dec","name":"","width":1,"height":2,"order":2,"category":"Logo","file":"whiskey-bottle-png-11.png","layout":"adjust","showcontrols":true,"loop":true,"onstart":false,"scope":"local","tooltip":"","x":300,"y":60,"wires":[[]]},{"id":"5a5860a4.7115f","type":"debug","z":"eb3d7fa.90ab38","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":670,"y":140,"wires":[]},{"id":"f5bc8da1.dac12","type":"debug","z":"eb3d7fa.90ab38","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":680,"y":200,"wires":[]},{"id":"8320344.08ef5c8","type":"tls-config","name":"","cert":"","key":"","ca":"","certname":"","keyname":"","caname":"","servername":"","verifyservercert":false},{"id":"49798262.df9dec","type":"ui_group","name":"Oil Price","tab":"12e195b2.3de9fa","order":6,"disp":false,"width":4,"collapse":false},{"id":"12e195b2.3de9fa","type":"ui_tab","name":"Cameras","icon":"mi-videogame_asset","order":1,"disabled":false,"hidden":false}]

Hi .. what information are you trying to get ? the price ?

if you follow the requests made by your browser using its developer tools, you will notice a request to an api that returns json with all the details of the product based on its id.

From there you can pick up any information you need from the product more easily.

Test flow :

[{"id":"a2afc3f2.206ba","type":"http request","z":"eb3d7fa.90ab38","name":"","method":"GET","ret":"obj","paytoqs":"ignore","url":"https://api.bws.com.au/apis/ui/Product/27390","tls":"","persist":false,"proxy":"","authType":"","x":490,"y":140,"wires":[["f5bc8da1.dac12"]]},{"id":"9bfb874a.3124a8","type":"inject","z":"eb3d7fa.90ab38","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":"5","topic":"","payloadType":"date","x":310,"y":140,"wires":[["a2afc3f2.206ba"]]},{"id":"f5bc8da1.dac12","type":"debug","z":"eb3d7fa.90ab38","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":710,"y":140,"wires":[]}]
1 Like

Hi!
Yep Price.

wow that makes it so much easier then webscraping!

Thanks so much... I have that working now :slight_smile:

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.