Output msg size limitation in Dashboard-2 ui-template nodes

Dashboard
1

I have been stress-testing dashboard-2 nodes with big data messages, as discussed in Msg size limit in Dashboard 2.0 (messages beyond a certain size break the client connection).

I have now found out that the issue is not with the input msg size, but rather with the msg size of the node's response, emitted by this.send() (once I limit the response size, I am able to send big data arrays, with hundreds of thousands of elements, in a single message).

If my measurements are correct, responses bigger than ~825KB fail and break the client connection. Increasing apiMaxLength in settings.js, has no effect.

Am I missing any other setting here?

The issue is consistent and easy to simulate.
image

The template receives array dimensions, creates an array of objects, stringifies it (to enable measuring its size) and sends it using this.send(). For objects with 5 properties, an array up to 6980 elements (825KB) will go through, a bigger array crashes the socket connection (with no browser or server exception, other than the disconnect notification).

[
    {
        "id": "44b742873403c498",
        "type": "inject",
        "z": "f6d5b8d748838f68",
        "name": "",
        "props": [
            {
                "p": "maxRows",
                "v": "6980",
                "vt": "num"
            },
            {
                "p": "maxProps",
                "v": "5",
                "vt": "num"
            }
        ],
        "repeat": "",
        "crontab": "",
        "once": false,
        "onceDelay": 0.1,
        "topic": "",
        "x": 910,
        "y": 580,
        "wires": [
            [
                "e32e14cab9556887"
            ]
        ]
    },
    {
        "id": "e32e14cab9556887",
        "type": "ui-template",
        "z": "f6d5b8d748838f68",
        "group": "2bc9d878f96a3d0e",
        "page": "",
        "ui": "",
        "name": "Create big response",
        "order": 5,
        "width": "8",
        "height": "3",
        "head": "",
        "format": "<template>\n    <div>\n        Columns: {{msg.maxCols}}<br>\n        Rows: {{msg.maxPros}}\n    </div>\n</template>\n\n<script>\nconst $scope = this;\n\n// Socket listener\n$scope.$socket.on('msg-input:' + this.id, function(msg)\n{\n    const rows = [];\n    for (let i=0; i < msg.maxRows; i++)\n    {\n        const row = {id:i};\n        for (let j = 1 ; j <= msg.maxProps ; j++)\n            row[\"Prop\"+j] =\"DATA-\" + (i) + \"-\" + j;\n        rows.push(row);\n    }\n    msg.payload=JSON.stringify(rows);\n    msg.responseLength = \"\"+(msg.payload.length/1024).toFixed(2)+\"KB\";\n    $scope.send(msg);\n});\n\n</script>\n",
        "storeOutMessages": true,
        "passthru": false,
        "resendOnRefresh": true,
        "templateScope": "local",
        "className": "",
        "x": 1100,
        "y": 580,
        "wires": [
            [
                "aa240d6ff17944eb"
            ]
        ]
    },
    {
        "id": "aa240d6ff17944eb",
        "type": "debug",
        "z": "f6d5b8d748838f68",
        "name": "debug 126",
        "active": true,
        "tosidebar": true,
        "console": false,
        "tostatus": false,
        "complete": "true",
        "targetType": "full",
        "statusVal": "",
        "statusType": "auto",
        "x": 1245,
        "y": 580,
        "wires": [],
        "l": false
    }
]
2

That is close enough to 1MB for me to believe you are hitting the 1MB default limit (rough calc, + TCP packet overhead etc)

For SocketIO, when a message is larger than the set limit, the connection will be killed.

This number has no affect on the SocketIO connection.

Dashboard docs: Settings | Node-RED Dashboard 2.0

3

IMO, rather than trying to send such large data, you would be wise to chuck it and join/append the messages in Node-RED.

4

Thanks, Steve.

  1. Is there a way to configure the SocketIO max msg size? I believe Joe said something about this in the past
  2. Is there a different mechanism for input & output? I am able to load huge amounts of data in a single msg, while output is limited to 1MB
5

:point_down: Did you read this? :point_down:

6

Thumb rule for the BE <-> FE application over socket is to use no more than 25% of available msg size. It works for almost every case and leaves enough headroom which almost always is needed to keep the application performant.

Pushing the limits is reasonable only if it cannot be done otherwise. Which is rare. And even then, the performance optimizations are unavoidable. Zipping and other such options clearly come into play.

7

How did I ever miss this?
Thanks Steve!

8

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.