Web Scraping with Empty results

General
1

Hello, I'm new to Node Red so I'm not quite sure how to extract correctly the data from web sites. I tried to retrieve data from http://www.bom.gov.au/nt/forecasts/darwin.shtml, but I got empty result. Could you please help me with this?

I used html selector as div#content>div.day.main>h2

This is my flow:

[
    {
        "id": "894c52497e20f993",
        "type": "inject",
        "z": "780c98709eb50e75",
        "name": "",
        "props": [
            {
                "p": "payload"
            },
            {
                "p": "topic",
                "vt": "str"
            }
        ],
        "repeat": "",
        "crontab": "",
        "once": false,
        "onceDelay": 0.1,
        "topic": "",
        "payload": "",
        "payloadType": "date",
        "x": 80,
        "y": 140,
        "wires": [
            [
                "d4b4012dd4ceaf6b"
            ]
        ]
    },
    {
        "id": "d4b4012dd4ceaf6b",
        "type": "http request",
        "z": "780c98709eb50e75",
        "name": "Weather ",
        "method": "GET",
        "ret": "txt",
        "paytoqs": "ignore",
        "url": "http://www.bom.gov.au/nt/forecasts/darwin.shtml",
        "tls": "",
        "persist": false,
        "proxy": "",
        "authType": "",
        "x": 260,
        "y": 160,
        "wires": [
            [
                "f38681f6d1ed532a"
            ]
        ]
    },
    {
        "id": "f38681f6d1ed532a",
        "type": "html",
        "z": "780c98709eb50e75",
        "name": "Today date",
        "property": "payload",
        "outproperty": "payload",
        "tag": "div#content>div.day.main>h2",
        "ret": "html",
        "as": "single",
        "x": 470,
        "y": 160,
        "wires": [
            [
                "72bf1c6b458cefff"
            ]
        ]
    },
    {
        "id": "72bf1c6b458cefff",
        "type": "function",
        "z": "780c98709eb50e75",
        "name": "Set date",
        "func": "flow.set(\"today\",msg.payload)\nreturn msg;",
        "outputs": 1,
        "noerr": 0,
        "initialize": "",
        "finalize": "",
        "libs": [],
        "x": 650,
        "y": 160,
        "wires": [
            [
                "4dc6bef1bb065db3"
            ]
        ]
    },
    {
        "id": "4dc6bef1bb065db3",
        "type": "function",
        "z": "780c98709eb50e75",
        "name": "Combining 2",
        "func": "var today = flow.get(\"today\");\nvar i;\nvar output = \"\";\nfor (i = 0; i < today.length; i++)\n{\n    output += today[i];\n}\nmsg.payload = output;\nreturn msg;",
        "outputs": 1,
        "noerr": 0,
        "initialize": "",
        "finalize": "",
        "libs": [],
        "x": 800,
        "y": 240,
        "wires": [
            [
                "4d4836bd2cec7c76"
            ]
        ]
    },
    {
        "id": "4d4836bd2cec7c76",
        "type": "debug",
        "z": "780c98709eb50e75",
        "name": "",
        "active": true,
        "tosidebar": true,
        "console": false,
        "tostatus": false,
        "complete": "false",
        "statusVal": "",
        "statusType": "auto",
        "x": 910,
        "y": 320,
        "wires": []
    }
]

Thanks

2

have you seen this msg (by wiring a debug node after the http-request) ?

image

I think its better if you use some other API for weather data

3

No I did not see that message. It showed: “msg.payload: string[0] “ and empty double quotation marks as an output.

5

Hi Erin,

I'm from Adelaide, and i met with the same issues with the BOM. i did send webreg@bom.go.au an email explaining my use (home automation for personal use) and the gave me a link to a fttp site which then allowed me to have local weather information on my dashboard.

Its well worth the effort, overseas information is not as accurate.

Kind Regards

Rick

1 Like
6

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.