Html node: Maximum call stack size exceeded

General
1

Hi

I've been successfully using the html node to filter a table. The file grows over time. I'm not sure when it stopped working, but it seems the amount of tags is too high for the node (currently 130900). Is there anything I can do to increase the maximum call stack size? I'm not sure how to approach this problem since it seems to be the node itself that's causing it.

NR: v2.0.6

Thanks

[
    {
        "id": "07c880a888a7a77e",
        "type": "http request",
        "z": "b8d32947.47d458",
        "name": "",
        "method": "GET",
        "ret": "bin",
        "paytoqs": "ignore",
        "url": "",
        "tls": "",
        "persist": false,
        "proxy": "",
        "authType": "",
        "x": 410,
        "y": 300,
        "wires": [
            [
                "86793ba9672a4618"
            ]
        ]
    },
    {
        "id": "074c9da903370f34",
        "type": "html",
        "z": "b8d32947.47d458",
        "name": "",
        "property": "payload[0].payload",
        "outproperty": "payload",
        "tag": "th",
        "ret": "text",
        "as": "single",
        "x": 860,
        "y": 300,
        "wires": [
            [
                "3bb2d8cffde10fa0"
            ]
        ]
    },
    {
        "id": "9c70d98883298d3d",
        "type": "html",
        "z": "b8d32947.47d458",
        "name": "",
        "property": "payload[0].payload",
        "outproperty": "payload",
        "tag": "td",
        "ret": "text",
        "as": "single",
        "x": 860,
        "y": 340,
        "wires": [
            [
                "8693ef9d4d024a57"
            ]
        ]
    },
    {
        "id": "86793ba9672a4618",
        "type": "switch",
        "z": "b8d32947.47d458",
        "name": "",
        "property": "statusCode",
        "propertyType": "msg",
        "rules": [
            {
                "t": "eq",
                "v": "200",
                "vt": "num"
            },
            {
                "t": "eq",
                "v": "404",
                "vt": "num"
            },
            {
                "t": "else"
            }
        ],
        "checkall": "true",
        "repair": false,
        "outputs": 3,
        "x": 550,
        "y": 300,
        "wires": [
            [
                "70ee34a914d9bf7f"
            ],
            [],
            []
        ]
    },
    {
        "id": "70ee34a914d9bf7f",
        "type": "zip",
        "z": "b8d32947.47d458",
        "name": "",
        "mode": "decompress",
        "filename": "",
        "compressionlevel": 6,
        "outasstring": true,
        "x": 710,
        "y": 300,
        "wires": [
            [
                "074c9da903370f34",
                "9c70d98883298d3d",
                "efe9f1abd4037791"
            ]
        ]
    },
    {
        "id": "efe9f1abd4037791",
        "type": "debug",
        "z": "b8d32947.47d458",
        "name": "",
        "active": true,
        "tosidebar": true,
        "console": false,
        "tostatus": false,
        "complete": "true",
        "targetType": "full",
        "statusVal": "",
        "statusType": "auto",
        "x": 990,
        "y": 260,
        "wires": []
    },
    {
        "id": "a008527ca52af59c",
        "type": "inject",
        "z": "b8d32947.47d458",
        "name": "url",
        "props": [
            {
                "p": "url",
                "v": "http://www.statssa.gov.za/wp-content/../timeseriesdata/Excel/P0141 - CPI(COICOP) from Jan 2008 (202108).zip",
                "vt": "str"
            }
        ],
        "repeat": "",
        "crontab": "",
        "once": false,
        "onceDelay": 0.1,
        "topic": "",
        "x": 270,
        "y": 300,
        "wires": [
            [
                "07c880a888a7a77e"
            ]
        ]
    },
    {
        "id": "3bb2d8cffde10fa0",
        "type": "debug",
        "z": "b8d32947.47d458",
        "name": "",
        "active": true,
        "tosidebar": true,
        "console": false,
        "tostatus": false,
        "complete": "true",
        "targetType": "full",
        "statusVal": "",
        "statusType": "auto",
        "x": 990,
        "y": 300,
        "wires": []
    },
    {
        "id": "8693ef9d4d024a57",
        "type": "debug",
        "z": "b8d32947.47d458",
        "name": "",
        "active": true,
        "tosidebar": true,
        "console": false,
        "tostatus": false,
        "complete": "true",
        "targetType": "full",
        "statusVal": "",
        "statusType": "auto",
        "x": 990,
        "y": 340,
        "wires": []
    }
]
2

Maybe try the contributed Cherio node:

node-red-contrib-cheerio (node) - Node-RED (nodered.org)

3

Thanks for the suggestion. Unfortunately it has the exact same error.

Using the Cheerio node:

[
    {
        "id": "07c880a888a7a77e",
        "type": "http request",
        "z": "b8d32947.47d458",
        "name": "",
        "method": "GET",
        "ret": "bin",
        "paytoqs": "ignore",
        "url": "",
        "tls": "",
        "persist": false,
        "proxy": "",
        "authType": "",
        "x": 410,
        "y": 300,
        "wires": [
            [
                "86793ba9672a4618"
            ]
        ]
    },
    {
        "id": "074c9da903370f34",
        "type": "html",
        "z": "b8d32947.47d458",
        "name": "",
        "property": "payload[0].payload",
        "outproperty": "payload",
        "tag": "th",
        "ret": "text",
        "as": "single",
        "x": 890,
        "y": 300,
        "wires": [
            [
                "3bb2d8cffde10fa0"
            ]
        ]
    },
    {
        "id": "9c70d98883298d3d",
        "type": "html",
        "z": "b8d32947.47d458",
        "d": true,
        "name": "",
        "property": "payload[0].payload",
        "outproperty": "payload",
        "tag": "td",
        "ret": "text",
        "as": "single",
        "x": 890,
        "y": 340,
        "wires": [
            [
                "8693ef9d4d024a57"
            ]
        ]
    },
    {
        "id": "86793ba9672a4618",
        "type": "switch",
        "z": "b8d32947.47d458",
        "name": "",
        "property": "statusCode",
        "propertyType": "msg",
        "rules": [
            {
                "t": "eq",
                "v": "200",
                "vt": "num"
            },
            {
                "t": "eq",
                "v": "404",
                "vt": "num"
            },
            {
                "t": "else"
            }
        ],
        "checkall": "true",
        "repair": false,
        "outputs": 3,
        "x": 550,
        "y": 300,
        "wires": [
            [
                "70ee34a914d9bf7f"
            ],
            [],
            []
        ]
    },
    {
        "id": "70ee34a914d9bf7f",
        "type": "zip",
        "z": "b8d32947.47d458",
        "name": "",
        "mode": "decompress",
        "filename": "",
        "compressionlevel": 6,
        "outasstring": true,
        "x": 710,
        "y": 300,
        "wires": [
            [
                "074c9da903370f34",
                "9c70d98883298d3d",
                "efe9f1abd4037791",
                "0a4fb8f488530839"
            ]
        ]
    },
    {
        "id": "efe9f1abd4037791",
        "type": "debug",
        "z": "b8d32947.47d458",
        "name": "",
        "active": true,
        "tosidebar": true,
        "console": false,
        "tostatus": false,
        "complete": "true",
        "targetType": "full",
        "statusVal": "",
        "statusType": "auto",
        "x": 1020,
        "y": 260,
        "wires": []
    },
    {
        "id": "a008527ca52af59c",
        "type": "inject",
        "z": "b8d32947.47d458",
        "name": "url",
        "props": [
            {
                "p": "url",
                "v": "http://www.statssa.gov.za/wp-content/../timeseriesdata/Excel/P0141 - CPI(COICOP) from Jan 2008 (202108).zip",
                "vt": "str"
            }
        ],
        "repeat": "",
        "crontab": "",
        "once": false,
        "onceDelay": 0.1,
        "topic": "",
        "x": 270,
        "y": 300,
        "wires": [
            [
                "07c880a888a7a77e"
            ]
        ]
    },
    {
        "id": "3bb2d8cffde10fa0",
        "type": "debug",
        "z": "b8d32947.47d458",
        "name": "",
        "active": true,
        "tosidebar": true,
        "console": false,
        "tostatus": false,
        "complete": "true",
        "targetType": "full",
        "statusVal": "",
        "statusType": "auto",
        "x": 1020,
        "y": 300,
        "wires": []
    },
    {
        "id": "8693ef9d4d024a57",
        "type": "debug",
        "z": "b8d32947.47d458",
        "name": "",
        "active": true,
        "tosidebar": true,
        "console": false,
        "tostatus": false,
        "complete": "true",
        "targetType": "full",
        "statusVal": "",
        "statusType": "auto",
        "x": 1020,
        "y": 340,
        "wires": []
    },
    {
        "id": "73d93cc76885cf9c",
        "type": "cheerio",
        "z": "b8d32947.47d458",
        "name": "",
        "tag": "td",
        "ret": "text",
        "as": "single",
        "map": [
            {
                "search": "",
                "ret": "html",
                "replace": "",
                "attr": ""
            }
        ],
        "xmlMode": false,
        "decodeEntities": false,
        "lowerCaseTags": false,
        "lowerCaseAttributeNames": false,
        "recognizeCDATA": false,
        "recognizeSelfClosing": false,
        "x": 890,
        "y": 380,
        "wires": [
            [
                "8693ef9d4d024a57"
            ]
        ]
    },
    {
        "id": "0a4fb8f488530839",
        "type": "change",
        "z": "b8d32947.47d458",
        "name": "",
        "rules": [
            {
                "t": "set",
                "p": "payload",
                "pt": "msg",
                "to": "payload[0].payload",
                "tot": "msg"
            }
        ],
        "action": "",
        "property": "",
        "from": "",
        "to": "",
        "reg": false,
        "x": 700,
        "y": 380,
        "wires": [
            [
                "73d93cc76885cf9c"
            ]
        ]
    }
]
4

It looks like most of these types of nodes are based on cheeriojs, and it seems the developers know about this bug but are not going to solve it:

5

you do have to wonder if html is the best way to serve that data - 130,000 tags is quite a large page...

6

Well, that isn't an HTML file is it - it is a zipped Excel file - something very different indeed and perfectly capable of handling 130000 lines of data (unless you use an old version of Excel :cough: PHE handling COVID data!).

7

It's not actually an Excel file - Excel just happens to open it with no problem. If one changes the extension to html it opens in a browser. Opening it with a text editor shows this as well.

I have had this working successfully for just less than a year. It only broke with the latest update to the file (which updates ~monthly).

I have not found a solution to the "Maximum call stack size exceeded" html parsing issue - however, I have found the same dataset in a text format. I will try processing that instead.

Thank you for your help - I genuinely appreciate all you do. Learning to use UIbuilder as we speak.

8

If I worked at StatsSA I would have pushed for an API instead of this. The file is definitely too big.

1 Like
9

Who the heck stores an html file in a file-name ending in .xls and then zips it!! That is crazy. And yes, I confirmed that it is indeed an html file.

I imported the file fine into Excel, around 750 lines, 175 columns.

What I would do would probably be to automate this in Excel rather than Node-RED - but that's just me.

If you want to use Node-RED, I suggest grabbing and unpacking the file then using text processing to strip everything but the actual table. If that is still too much to process with Cheerio, do it the old-school way, breaking the file down into a JavaScript object.

10

Instead of processing those html nodes at the same time, can you try to select the elements in sequence ?
and instead of selecting the td select the tr and split it with javascript

image
image1011Ă—170 10.4 KB

Example ( used a File read node - didnt have the zip node )

[{"id":"efe9f1abd4037791","type":"debug","z":"5847b7aa62131d37","name":"2","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":630,"y":1580,"wires":[]},{"id":"3bb2d8cffde10fa0","type":"debug","z":"5847b7aa62131d37","name":"3","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":990,"y":1660,"wires":[]},{"id":"074c9da903370f34","type":"html","z":"5847b7aa62131d37","name":"","property":"payload","outproperty":"headers","tag":".header","ret":"text","as":"single","x":500,"y":1660,"wires":[["efe9f1abd4037791","6926418f3f603813"]]},{"id":"396fbd4c01e6c35e","type":"file in","z":"5847b7aa62131d37","name":"read file","filename":"c:\\share\\Excel - CPI (COICOP) from January 2008 (202108).xls","format":"utf8","chunk":false,"sendError":false,"encoding":"none","allProps":false,"x":330,"y":1660,"wires":[["257483723eec8e17","074c9da903370f34"]]},{"id":"95e2bec020123878","type":"inject","z":"5847b7aa62131d37","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":160,"y":1660,"wires":[["396fbd4c01e6c35e"]]},{"id":"257483723eec8e17","type":"debug","z":"5847b7aa62131d37","name":"1","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":430,"y":1580,"wires":[]},{"id":"a6f3982a843f1c10","type":"function","z":"5847b7aa62131d37","name":"","func":"msg.data = msg.data.map(el => {\n    return el.trim().split(\"\\n\")\n})\n\n\n\nreturn {\n    payload: {\n        headers: msg.headers,\n        data: msg.data\n    }\n};","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":820,"y":1660,"wires":[["3bb2d8cffde10fa0"]]},{"id":"6926418f3f603813","type":"html","z":"5847b7aa62131d37","name":"","property":"payload","outproperty":"data","tag":"tr","ret":"text","as":"single","x":670,"y":1660,"wires":[["a6f3982a843f1c10"]]}]
2 Likes
11

Thank you!

This works like a charm and now I don't have to redo the whole process using the other file type.

:grinning_face_with_smiling_eyes:

12

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.