I think, this question is beyond NR, but asking anyway? So, trying to web page scrap dynamic content is just possible within NR? At least I don't see how a GET via http request/response can invoke dynamic content. Just like curl or wget can't as well, right?
I wanted to get a list of available versions of a given software solution as referenced above, that is only visible via a dynamic web page, based on javascript. I get the markup content only... which is expected once I realized the webpage was dynamic content.
As @Bobo has said there are a few topics on the forum covering this. I know I have went through it with the op. Search @Steve-Mcl scrape on this forum. Hopefully that will turn up the threads
Edit...
So this thread show how the value was obtained by monitoring the networks tab in devtools.
I absolutely agree to try NR options first ( I have no experience with NR web scraping), however, we needed a tactical soln scrape a providers web site. I used the google puppateer node.js module. It takes a bit to get your head around (probably less if you are a front end person) but is amazing.
Probably overkill for what you need here but if you get stuck then have a look. If you go down this path I guess in theory since it is an NPM module you could load into a function node but !!!! I think you would find a world of pain. I would create a separate node.js app that would
I searched a bit... but did not see any obvious example of how I might invoke the script logic to get the data it in turn generates. @Steve-Mcl, thanks for the pointer.
Unfortunately, or fortunately, I found a different solution... by using apt search to query the packages depo to find available versions of said package. That works for now. But it is not as accurate or elegant as being able to query say npm for available versions of a given module.
I am quite curious about this topic, now that I tripped over it. So day, some way, would be interesting if the http request/response nodes, or new nodes establish that could do this type of behind the curtain type of thing... thinking it might have a signficant use case? Or just wishful thinking on my part?
To get dynamic data from a website, you need to run a scraper capable of executing the front-end JavaScript. I am aware of only 1 node that will do that and it presents a full headless browser so is a massive installation. However, it should work.
