Extracting data from a webpage

General
1

Hi ,
I am trying get a cell data from a webpage but gives me an error "Json parse error".

    {
        "id": "8021ac643abe9bd7",
        "type": "inject",
        "z": "5c0984f6408e957c",
        "name": "",
        "props": [
            {
                "p": "payload"
            },
            {
                "p": "topic",
                "vt": "str"
            }
        ],
        "repeat": "",
        "crontab": "",
        "once": false,
        "onceDelay": 0.1,
        "topic": "",
        "payload": "",
        "payloadType": "date",
        "x": 180,
        "y": 140,
        "wires": [
            [
                "68304d98c7cee3f9"
            ]
        ]
    },
    {
        "id": "68304d98c7cee3f9",
        "type": "http request",
        "z": "5c0984f6408e957c",
        "name": "",
        "method": "GET",
        "ret": "obj",
        "paytoqs": "ignore",
        "url": "https://cloud.suryalog.com",
        "tls": "",
        "persist": false,
        "proxy": "",
        "insecureHTTPParser": false,
        "authType": "",
        "senderr": false,
        "headers": [],
        "x": 350,
        "y": 140,
        "wires": [
            [
                "4e7b2fc1bdf24494"
            ]
        ]
    },
    {
        "id": "398a530d4521d4cc",
        "type": "debug",
        "z": "5c0984f6408e957c",
        "name": "debug 2",
        "active": true,
        "tosidebar": true,
        "console": false,
        "tostatus": false,
        "complete": "false",
        "statusVal": "",
        "statusType": "auto",
        "x": 760,
        "y": 120,
        "wires": []
    },
    {
        "id": "4e7b2fc1bdf24494",
        "type": "html",
        "z": "5c0984f6408e957c",
        "name": "",
        "property": "payload",
        "outproperty": "payload",
        "tag": ".tabulator-cell 120,98 x30",
        "ret": "html",
        "as": "single",
        "chr": "_",
        "x": 550,
        "y": 140,
        "wires": [
            [
                "398a530d4521d4cc"
            ]
        ]
    }
]```

I am trying to get data as given below
2

image
image1366×768 192 KB

3

This tag doesn't look right. That is not a valid selector.

You need to first extract the whole table. Then convert the table to JSON and then you can pick out the correct entry.

4

image
image1366×768 169 KB

I tried this ...result the same...

5

You really need to share your configuration when saying things like this.

Are you saying that you got the content of the page using an http-request node, then used an html node to extract the table using the CSS selector #source_table_day and then pass that to a JSON node?

image
image829×107 7.73 KB

image
image898×699 29.4 KB

6

You are asking the http request node to return a parsed JSON object, but the page is returning a html string. change the requested nodes return type a utf8 string. Then the JSON parse error will disappear.

7

That's right...Now after changing the return message to "UTF8-String" in http node, the JSON parse error has disappeared but the node returns "empty" message.

I login in to this webpage with my credentials.Do I need put those details somewhere in the nodes?

8

ok. Now have some response but not sure how to get the figures.
Response received : Expected name, found .#source_table_day

image
image1366×768 78.5 KB

9

Try it without the .
ie. #source_table_day

10

Without the "." it returns an empty array

11

Fwiw, it is very unlikely any dynamic data will be returned in the HTML. It's more likely there is an API call to get the data and JavaScript populates that table.

See this recent similar post: How to deconstruct web pages with http-request - #2 by Steve-Mcl