# Web Scraping with Empty results

**URL:** https://discourse.nodered.org/t/web-scraping-with-empty-results/52587
**Category:** General
**Tags:** http-request
**Created:** [22 October 2021 18:08 UTC](https://discourse.nodered.org/t/web-scraping-with-empty-results/52587 "2021-10-22T18:08:09Z")
**Posts on this page:** 5
**Page:** 1

<div class="post-metadata">

### Author: ![Erin](https://avatars.discourse-cdn.com/v4/letter/e/67e7ee/32.png) [@Erin](https://discourse.nodered.org/u/Erin)
#### Post date: [22 October 2021 18:08 UTC](https://discourse.nodered.org/t/web-scraping-with-empty-results/52587/1 "2021-10-22T18:08:09Z")

</div>

Hello, I'm new to Node Red so I'm not quite sure how to extract correctly the data from web sites. I tried to retrieve data from [http://www.bom.gov.au/nt/forecasts/darwin.shtml](http://www.bom.gov.au/nt/forecasts/darwin.shtml), but I got empty result. Could you please help me with this?

I used html selector as div#content\>div.day.main\>h2

This is my flow:

```auto
[
    {
        "id": "894c52497e20f993",
        "type": "inject",
        "z": "780c98709eb50e75",
        "name": "",
        "props": [
            {
                "p": "payload"
            },
            {
                "p": "topic",
                "vt": "str"
            }
        ],
        "repeat": "",
        "crontab": "",
        "once": false,
        "onceDelay": 0.1,
        "topic": "",
        "payload": "",
        "payloadType": "date",
        "x": 80,
        "y": 140,
        "wires": [
            [
                "d4b4012dd4ceaf6b"
            ]
        ]
    },
    {
        "id": "d4b4012dd4ceaf6b",
        "type": "http request",
        "z": "780c98709eb50e75",
        "name": "Weather ",
        "method": "GET",
        "ret": "txt",
        "paytoqs": "ignore",
        "url": "http://www.bom.gov.au/nt/forecasts/darwin.shtml",
        "tls": "",
        "persist": false,
        "proxy": "",
        "authType": "",
        "x": 260,
        "y": 160,
        "wires": [
            [
                "f38681f6d1ed532a"
            ]
        ]
    },
    {
        "id": "f38681f6d1ed532a",
        "type": "html",
        "z": "780c98709eb50e75",
        "name": "Today date",
        "property": "payload",
        "outproperty": "payload",
        "tag": "div#content>div.day.main>h2",
        "ret": "html",
        "as": "single",
        "x": 470,
        "y": 160,
        "wires": [
            [
                "72bf1c6b458cefff"
            ]
        ]
    },
    {
        "id": "72bf1c6b458cefff",
        "type": "function",
        "z": "780c98709eb50e75",
        "name": "Set date",
        "func": "flow.set(\"today\",msg.payload)\nreturn msg;",
        "outputs": 1,
        "noerr": 0,
        "initialize": "",
        "finalize": "",
        "libs": [],
        "x": 650,
        "y": 160,
        "wires": [
            [
                "4dc6bef1bb065db3"
            ]
        ]
    },
    {
        "id": "4dc6bef1bb065db3",
        "type": "function",
        "z": "780c98709eb50e75",
        "name": "Combining 2",
        "func": "var today = flow.get(\"today\");\nvar i;\nvar output = \"\";\nfor (i = 0; i < today.length; i++)\n{\n output += today[i];\n}\nmsg.payload = output;\nreturn msg;",
        "outputs": 1,
        "noerr": 0,
        "initialize": "",
        "finalize": "",
        "libs": [],
        "x": 800,
        "y": 240,
        "wires": [
            [
                "4d4836bd2cec7c76"
            ]
        ]
    },
    {
        "id": "4d4836bd2cec7c76",
        "type": "debug",
        "z": "780c98709eb50e75",
        "name": "",
        "active": true,
        "tosidebar": true,
        "console": false,
        "tostatus": false,
        "complete": "false",
        "statusVal": "",
        "statusType": "auto",
        "x": 910,
        "y": 320,
        "wires": []
    }
]

```

Thanks

---

<div class="post-metadata">

### Author: ![UnborN](https://avatars.discourse-cdn.com/v4/letter/u/4491bb/32.png) [@UnborN](https://discourse.nodered.org/u/UnborN)
#### Post date: [22 October 2021 18:47 UTC](https://discourse.nodered.org/t/web-scraping-with-empty-results/52587/2 "2021-10-22T18:47:39Z")

</div>

have you seen this msg (by wiring a debug node after the http-request) ?

![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/0/7/0729263670114a472e114516c8032ed304d7ed1b.png)

I think its better if you use some other API for weather data

---

<div class="post-metadata">

### Author: ![Erin](https://avatars.discourse-cdn.com/v4/letter/e/67e7ee/32.png) [@Erin](https://discourse.nodered.org/u/Erin)
#### Post date: [22 October 2021 19:02 UTC](https://discourse.nodered.org/t/web-scraping-with-empty-results/52587/3 "2021-10-22T19:02:25Z")

</div>

No I did not see that message. It showed: “msg.payload: string[0] “ and empty double quotation marks as an output.

---

<div class="post-metadata">

### Author: ![Bigbloke](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/bigbloke/32/40969_2.png) [@Bigbloke](https://discourse.nodered.org/u/Bigbloke)
#### Post date: [28 November 2021 22:43 UTC](https://discourse.nodered.org/t/web-scraping-with-empty-results/52587/5 "2021-11-28T22:43:32Z")

</div>

Hi Erin,

I'm from Adelaide, and i met with the same issues with the BOM. i did send webreg@bom.go.au an email explaining my use (home automation for personal use) and the gave me a link to a fttp site which then allowed me to have local weather information on my dashboard.

Its well worth the effort, overseas information is not as accurate.

Kind Regards

Rick

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/1X/d073cd938eafa2e558d7c2cd59003b3ef4963033.png) [@system](https://discourse.nodered.org/u/system)
#### Post date: [27 January 2022 22:44 UTC](https://discourse.nodered.org/t/web-scraping-with-empty-results/52587/6 "2022-01-27T22:44:02Z")

</div>

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.