Help with a webscrap

[leftymuller]

Hello All!

Can somebody lend me a hand.. HTML is not my strong point

I want to get the oil price from this website Crude Oil Prices - Brent And WTI Oil Price Per Barrel (oilcrudeprice.com)

I can find the figure I need

I did a flow and told it to get H3 and it gets this

Any hints or suggestions would be great appreciated!

[Steve-Mcl]

Does this help...

[{"id":"0488087db4598ce8","type":"http request","z":"bec69dbd.8d622","name":"","method":"GET","ret":"txt","paytoqs":"ignore","url":"https://www.oilcrudeprice.com/","tls":"","persist":false,"proxy":"","authType":"","x":840,"y":1440,"wires":[["4645e9233d951348"]]},{"id":"4645e9233d951348","type":"html","z":"bec69dbd.8d622","name":"","property":"payload","outproperty":"payload","tag":".oilcov","ret":"html","as":"single","x":1010,"y":1440,"wires":[["d5ca7512d8d6dd76"]]},{"id":"b9667d098f36319d","type":"inject","z":"bec69dbd.8d622","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payloadType":"date","x":660,"y":1440,"wires":[["0488087db4598ce8"]]},{"id":"960445cec435fba3","type":"debug","z":"bec69dbd.8d622","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":1450,"y":1440,"wires":[]},{"id":"f7dfd696c7b9d979","type":"html","z":"bec69dbd.8d622","name":"","property":"payload","outproperty":"payload","tag":"p","ret":"text","as":"single","x":1290,"y":1440,"wires":[["960445cec435fba3"]]},{"id":"d5ca7512d8d6dd76","type":"split","z":"bec69dbd.8d622","name":"","splt":"\\n","spltType":"str","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"","x":1150,"y":1440,"wires":[["f7dfd696c7b9d979"]]}]

Using a function node or JSONata in a change node you could easily extract the values from this ↑ output.

[leftymuller]

Absolutely! Well ive learned something today

Thanks mate.. appreciated

[E1cid]

There is also a table at the bottom of the page which makes it easier to grab all the info
e.g.

[{"id":"5096c6f0.4b0b88","type":"inject","z":"c74669a0.6a34f8","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":140,"y":1860,"wires":[["3ad9af0c.91263"]]},{"id":"3ad9af0c.91263","type":"http request","z":"c74669a0.6a34f8","name":"","method":"GET","ret":"txt","paytoqs":"ignore","url":"https://www.oilcrudeprice.com/","tls":"","persist":false,"proxy":"","authType":"","x":350,"y":1860,"wires":[["62932699.905cf"]]},{"id":"62932699.905cf","type":"html","z":"c74669a0.6a34f8","name":"","property":"payload","outproperty":"payload","tag":"tbody > tr  > td","ret":"text","as":"single","x":560,"y":1860,"wires":[["896fbba5.052b98"]]},{"id":"896fbba5.052b98","type":"debug","z":"c74669a0.6a34f8","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":610,"y":1640,"wires":[]}]

returns

["Brent Oil","$76.18","+0.62","+0.82%","06/25/21","$75.61","$75.56","$74.95","$76.21","WTI Oil","$74.05","+0.75","+1.02%","06/25/21","$73.32","$73.3","$72.85","$74.18","Natural Gas","$3.520","+0.083","+2.41%","06/25/21","$3.439","$3.437","$3.43","$3.538","Gold","$1777.8","+1.1","+0.06%","06/25/21","$1775.7","$1776.7","$1773.6","$1791","Silver","$26.087","+0.037","+0.14%","06/25/21","$26.01","$26.05","$25.97","$26.38","Platinum","$1103.6","+9.7","+0.89%","06/25/21","$1093","$1093.9","$1091.3","$1111.2","Palladium","$2637.20","-6.70","-0.25%","06/25/21","$2634.5","$2643.9","$2612.5","$2660.5"]

[leftymuller]

So ive had another go at scraping

This is the data from the webbrowser

with a HTML node with this

and i get

where am I going wrong? Appreciate any help... im a HTML newb

here is the flow

[{"id":"eb3d7fa.90ab38","type":"tab","label":"test page","disabled":false,"info":""},{"id":"a2afc3f2.206ba","type":"http request","z":"eb3d7fa.90ab38","name":"","method":"GET","ret":"txt","paytoqs":"ignore","url":"https://bws.com.au/product/27390/jack-daniel-s-old-no-7-tennessee-whiskey-1l","tls":"8320344.08ef5c8","persist":false,"proxy":"","authType":"","x":270,"y":140,"wires":[["96680850.bb9468","f5bc8da1.dac12"]]},{"id":"9bfb874a.3124a8","type":"inject","z":"eb3d7fa.90ab38","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"780","crontab":"","once":true,"onceDelay":"5","topic":"","payload":"","payloadType":"date","x":100,"y":140,"wires":[["a2afc3f2.206ba"]]},{"id":"96680850.bb9468","type":"html","z":"eb3d7fa.90ab38","name":"","property":"payload","outproperty":"payload","tag":"div.product-detail_controls-col.ng-scope > h1","ret":"text","as":"single","x":410,"y":260,"wires":[["5a5860a4.7115f"]]},{"id":"6291cb18.38b9b4","type":"ui_media","z":"eb3d7fa.90ab38","group":"49798262.df9dec","name":"","width":1,"height":2,"order":2,"category":"Logo","file":"whiskey-bottle-png-11.png","layout":"adjust","showcontrols":true,"loop":true,"onstart":false,"scope":"local","tooltip":"","x":300,"y":60,"wires":[[]]},{"id":"5a5860a4.7115f","type":"debug","z":"eb3d7fa.90ab38","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":670,"y":140,"wires":[]},{"id":"f5bc8da1.dac12","type":"debug","z":"eb3d7fa.90ab38","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":680,"y":200,"wires":[]},{"id":"8320344.08ef5c8","type":"tls-config","name":"","cert":"","key":"","ca":"","certname":"","keyname":"","caname":"","servername":"","verifyservercert":false},{"id":"49798262.df9dec","type":"ui_group","name":"Oil Price","tab":"12e195b2.3de9fa","order":6,"disp":false,"width":4,"collapse":false},{"id":"12e195b2.3de9fa","type":"ui_tab","name":"Cameras","icon":"mi-videogame_asset","order":1,"disabled":false,"hidden":false}]

[UnborN]

Hi .. what information are you trying to get ? the price ?

if you follow the requests made by your browser using its developer tools, you will notice a request to an api that returns json with all the details of the product based on its id.

From there you can pick up any information you need from the product more easily.

Test flow :

[{"id":"a2afc3f2.206ba","type":"http request","z":"eb3d7fa.90ab38","name":"","method":"GET","ret":"obj","paytoqs":"ignore","url":"https://api.bws.com.au/apis/ui/Product/27390","tls":"","persist":false,"proxy":"","authType":"","x":490,"y":140,"wires":[["f5bc8da1.dac12"]]},{"id":"9bfb874a.3124a8","type":"inject","z":"eb3d7fa.90ab38","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":"5","topic":"","payloadType":"date","x":310,"y":140,"wires":[["a2afc3f2.206ba"]]},{"id":"f5bc8da1.dac12","type":"debug","z":"eb3d7fa.90ab38","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":710,"y":140,"wires":[]}]

[leftymuller]

Hi!
Yep Price.

wow that makes it so much easier then webscraping!

Thanks so much... I have that working now