This question comes up regularly in the forum.
It usually starts with someone asking "how do I get XYZ data from this web page?". Where it turns out that the page is creating the required data dynamically using JavaScript.
There are two ways to resolve this issue. Which you use depends on how the page is working. So the first option is probably only really open to you if you can make sense of the code in your browser (view that by right-clicking on the page and selecting "Inspect" or by opening the browser's developer tools and going to the "Elements" tab.
- Find an API call on the page that returns the data you really want
- Use a "headless browser" to run the pages JavaScript and scrape the resulting data.
1) Find an API call on the page that returns the data you really want
To do this, you have to search through the web page's code or perhaps look at the network tab in the browser dev tools. Then once found, you need to see if you can call the API (which is another web endpoint) or whether it has other security that might be too hard to easily work out.
The advantage of this is that you are likely to get exactly the data you want in a form easily consumed in Node-RED (e.g. JSON or XML) and you won't have to pull the data out of some HTML.
2) Use a "headless browser" to run the pages JavaScript and scrape the resulting data.
There are a number of existing nodes that you may be able to use. Search for nodes that use nbrowser or puppeteer for example.
You could even run your own browser "headless" if you are running Node-RED on a server that has a Chromium or Firefox based browser installed. On Windows, for example, the following command line will grab a processed page using the Microsoft Edge Chromium-based browser:
"C:\Program Files (x86)\Microsoft\Edge\Application\msedge.exe" --headless --disable-gpu --enable-logging --dump-dom https://nodered.org
