Get data from a website using the Http request node

General
1

Hi, I am trying to pull data from a website using the http request node and a HTML node like they do in this example Simple GET request : Node-RED, However my data is in a jstcache variable instead of a class as shown in the picture below:
image
The data I am trying to pull is jstcache = 320, but whenever I pull it now it just gives me an empty array. Is there a way to access this data in node red?

I am new with anything html so any help would be appreciated,
Thanks in advance!

2

The selector would be span[jstcache=320] untested as you supplied no link or copyable data.

3

Sorry I didn't provide much info, the original had a bit of personal information, but here is the flow and a link of a more general test.

Site URL:
"Google Maps"

Flow:

[
    {
        "id": "c1d14628c2369d80",
        "type": "inject",
        "z": "235adc4f290fa86f",
        "name": "make request",
        "props": [
            {
                "p": "payload",
                "v": "",
                "vt": "date"
            },
            {
                "p": "topic",
                "v": "",
                "vt": "str"
            }
        ],
        "repeat": "",
        "crontab": "",
        "once": false,
        "topic": "",
        "payload": "",
        "payloadType": "date",
        "x": 270,
        "y": 160,
        "wires": [
            [
                "90f9e6f8756f49cb"
            ]
        ]
    },
    {
        "id": "90f9e6f8756f49cb",
        "type": "http request",
        "z": "235adc4f290fa86f",
        "name": "",
        "method": "GET",
        "ret": "txt",
        "paytoqs": "ignore",
        "url": "https://www.google.ca/maps/dir/Laval/Montreal,+QC/@45.5590971,-73.8519568,11z/data=!4m14!4m13!1m5!1m1!1s0x4cc92107b4dfda6b:0x2eb4b26fe333c419!2m2!1d-73.712409!2d45.6066487!1m5!1m1!1s0x4cc91a541c64b70d:0x654e3138211fefef!2m2!1d-73.5673919!2d45.5018869!3e3?hl=en",
        "tls": "",
        "persist": false,
        "proxy": "",
        "authType": "",
        "senderr": false,
        "x": 450,
        "y": 160,
        "wires": [
            [
                "c847c70b23acc5a8"
            ]
        ]
    },
    {
        "id": "9a5642cb78d66133",
        "type": "debug",
        "z": "235adc4f290fa86f",
        "name": "",
        "active": true,
        "console": "false",
        "complete": "false",
        "x": 830,
        "y": 160,
        "wires": []
    },
    {
        "id": "c847c70b23acc5a8",
        "type": "html",
        "z": "235adc4f290fa86f",
        "name": "",
        "property": "",
        "outproperty": "",
        "tag": "span[jstcache=\"1492\"]",
        "ret": "text",
        "as": "single",
        "x": 640,
        "y": 160,
        "wires": [
            [
                "9a5642cb78d66133"
            ]
        ]
    }
]

I am still encountering the issue where the debug will be an empty array.
image

The info I am trying to access of this page is the highlighted code below:

image
image1775×163 66.2 KB

4

The issue is Google map page is generated with Javascript, in the browser that is fine, unfortunately the http request node does not run the javascript, so using the http request node is not going to work.

You may be lucky and find some link in the developer console of your browser that may load the data, but I doubt it.

You could try some contrib nodes that can make a http request and execute the scripts, to get the final loaded page. Search for Nbrowser, Puppeteer or broswer.

5

Okay, thanks

6

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.