Web scraping to grab an items price, but coming back empty

General
1

I'm trying to grab price data from this page via HTTP request. The price appears to be in <div class="sc-fznMnq biVOqy">$146.63 </div>. So, in the HTML Node, I set the Selector field to .sc-fznMnq biVOqy (following the example from the docs). But it keeps coming back empty - am I missing something here?

Here is my flow:

[{"id":"e5ce3a5d.b3d348","type":"inject","z":"f20f1b13.5e8608","name":"make request","repeat":"","crontab":"","once":false,"topic":"","payload":"","payloadType":"date","x":1310,"y":300,"wires":[["251988f5.6a3478"]]},{"id":"251988f5.6a3478","type":"http request","z":"f20f1b13.5e8608","name":"6x6x20","method":"GET","ret":"txt","paytoqs":"ignore","url":"https://www.lowes.com/pd/Common-6-in-x-6-in-x-20-ft-Actual-5-5-in-x-5-5-in-x-20-ft-2-Treated-Lumber/1000698158","tls":"","persist":false,"proxy":"","authType":"","x":1464.5,"y":300,"wires":[["15cd391f.bee797","ae2bd9d0.9ba898"]]},{"id":"bfbea6ff.1329b8","type":"debug","z":"f20f1b13.5e8608","name":"","active":true,"console":"false","complete":"false","x":1970,"y":300,"wires":[]},{"id":"15cd391f.bee797","type":"html","z":"f20f1b13.5e8608","name":"","property":"","outproperty":"","tag":".sc-fznMnq biVOqy","ret":"text","as":"single","x":1710,"y":300,"wires":[["bfbea6ff.1329b8"]]}]

2

It is likely the value is populated separate from the html (i.e. via JavaScript). In otherwords, the value is possibly not present in the html until some time after is arrives in the browser.

You might need to look at the network tab in dev tools to see if you can find the source of the data.

3

If the web page has frames... make sure you are getting the right frame. I wrote a cable modem status scraper in NR, and I was not getting the right results, realized looking at the HTML details, the status value was in a nested frame, which was not obvious. I just set the HTTP request to the right frame, and bingo, got the cable modem status as expected.

4

As i thought, value is NOT present in the HTML (it is updated in a later background call to another URL)

image
image1406×661 145 KB

The actual URL is https://www.lowes.com/pd/1000698158/productdetail/2209/Guest so do a request to that URL, set the request node to JSON then access the values from msg.payload.productDetails["1000698158"].price.itemPrice

5

@Steve-Mcl great - I can see the price value if I throw that URL in a browser - but price comes back as null in the JSON response via NR. Oddly, all the other data (that I don't care about!) is in the JSON response.

Via Browser:
image

Via NR:

image
image266×509 9.92 KB

6

At a guess, you need to replicate the headers Chrome uses.

Or set cookies.

7

Thanks @Steve-Mcl I'll look into that. Not something I am familiar with - so if you have any good resources to point me in the right direction, I'd be grateful. If not, thanks for the assistance so far - I'm closer than I was this morning!

8

Make a request to the original page - you will get cookie(s) then make 2nd request to the JSON URL with the cookies from the 1st response in the 2nd request.

Basically check using Devtools what is sent in the request for the JSON & try to replicate that in node-red.

9

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.