Website opens in chrome but gives error message when using node red http node

General
1

If i open the following link in chrome it works fine and displays all data,

http://www.bom.gov.au/fwo/IDN60801/IDN60801.94767.json

but if i use same in node red http request node I get the following in debug.

"

Your access is blocked due to the detection of a potential automated access request. The Bureau of Meteorology website does not support web scraping: if you are trying to access Bureau data through automated means, you should stop. You may like to consider the following options:

If you still nee..."

can anyone help, thanks

2

It requires headers to be set.

Can you try with this http request:

[
    {
        "id": "3a0493d9fee81171",
        "type": "http request",
        "z": "92d9e3127ad54058",
        "name": "",
        "method": "GET",
        "ret": "obj",
        "paytoqs": "ignore",
        "url": "http://www.bom.gov.au/fwo/IDN60801/IDN60801.94767.json",
        "tls": "",
        "persist": false,
        "proxy": "",
        "insecureHTTPParser": false,
        "authType": "",
        "senderr": false,
        "headers": [
            {
                "keyType": "other",
                "keyValue": "User-Agent",
                "valueType": "other",
                "valueValue": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.0 Safari/605.1.15"
            },
            {
                "keyType": "other",
                "keyValue": "Accept",
                "valueType": "other",
                "valueValue": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"
            },
            {
                "keyType": "other",
                "keyValue": "Upgrade-Insecure-Requests",
                "valueType": "other",
                "valueValue": "1"
            },
            {
                "keyType": "other",
                "keyValue": "Connection",
                "valueType": "other",
                "valueValue": "keep-alive"
            }
        ],
        "credentials": {},
        "x": 450,
        "y": 280,
        "wires": [
            [
                "09dab6285439ee8f"
            ]
        ]
    }
]
3

Are you asking for help with breaking the bureau's terms of use?

4

Is opening the site with node Red scraping ????

5

Yes, unless, having read it, you are just going to display it in the dashboard, or similar. In which case you would be better to use an iframe.

6

Yes, I only want to view the data on my dashboard. Would that be termed scraping???

I could display it from the info that I get when I open the page in chrome. I am struggling to see what is different to opening directly in node Red. Morally/ legally/ or any other code I mean.

7

Hi Colin, please advise what is iframe that you mention and how is it used for getting and displaying the json data from this site???
thankyou

8

google what is iframe and you will find lots of information.

9

Also search the forum for
dashboard iframe
and you will find numerous posts about doing it.

10

if i copy this script above and paste into http request the node indicates that there is an error showing the red triangle.

11

thanks Colin, i can see the json data in my dashboard using iframe node. Is it possible to extract specific pieces of the json data from the iframe node to then display using graph or gauge in dashboard?????

12

No it isn't possible. Normally the iframe would be used to show the webpage, not the json.

13

The code IS the http request, tested it, it works.

Import the code into node red and a request node with the correct properties will be added to the flow.

14

Hello from Australia.

I get the same error using your URL. Something not right about that endpoint. Where did you get that from?

Try this:

http://reg.bom.gov.au/fwo/IDV60901/IDV60901.95936.json

I note the URL has reg.bom not www.bom.

That's the weather for Melbourne. Works great.

if that works for you I can point you to the page where you get these endpoints.

15

Thankyou Andrew, this link works perfectly. I adjusted the IDV stuff and now can see my local weather.

Again, thanks for you assistance

16

I imported into node red, but I still get the errors about scraping as previously

17

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.