Performance of function vs. contrib node

General
1

Hello,
In a test app (checking data transfer performance), I need to generate a random string of 50000000 characters (the number may vary, but, it is of this kind of magnitude).
I found this contrib node node-red-contrib-random-string (node) - Node-RED that does (almost) what I need except the size of the string is in the config of the node and not as an input (eg. msg.size).
I add a look at the github repository of this contrib and looked at the code here https://github.com/Wavebreakers/node-red-contrib-random-string/blob/main/random-string.js

As the code is fairly simple I have used the random generation part in a function node. It works.
However, I noticed (for this kind of size) a fairly large difference in the execution time.
The contrib node being more than twice as fast as the function, even if the js in the code is exactly the same.

Here the two flows creating a 50M string along with flow-timers.

[
    {
        "id": "9a693308.7ebaf",
        "type": "subflow",
        "name": "flow-timer",
        "info": "",
        "category": "",
        "in": [
            {
                "x": 80,
                "y": 100,
                "wires": [
                    {
                        "id": "7fc82258.93e36c"
                    }
                ]
            }
        ],
        "out": [
            {
                "x": 440,
                "y": 100,
                "wires": [
                    {
                        "id": "7fc82258.93e36c",
                        "port": 0
                    }
                ]
            }
        ],
        "env": [
            {
                "name": "name",
                "type": "str",
                "value": "measure",
                "ui": {
                    "icon": "font-awesome/fa-tag",
                    "label": {
                        "en-US": "Timer Name"
                    },
                    "type": "input",
                    "opts": {
                        "types": [
                            "str",
                            "env"
                        ]
                    }
                }
            },
            {
                "name": "operation",
                "type": "str",
                "value": "start",
                "ui": {
                    "icon": "font-awesome/fa-cog",
                    "label": {
                        "en-US": "Operation"
                    },
                    "type": "select",
                    "opts": {
                        "opts": [
                            {
                                "l": {
                                    "en-US": "start"
                                },
                                "v": "start"
                            },
                            {
                                "l": {
                                    "en-US": "stop"
                                },
                                "v": "stop"
                            },
                            {
                                "l": {
                                    "en-US": "msg.topic"
                                },
                                "v": "msg.topic"
                            },
                            {
                                "l": {
                                    "en-US": "msg.operation"
                                },
                                "v": "msg.operation"
                            },
                            {
                                "l": {
                                    "en-US": "msg.payload"
                                },
                                "v": "msg.payload"
                            }
                        ]
                    }
                }
            }
        ],
        "meta": {
            "module": "node-red-contrib-flow-performance",
            "type": "flow-performance",
            "version": "1.0.1",
            "author": "Steve-Mcl",
            "desc": "Inline flow performance measure node",
            "keywords": "node-red performance",
            "license": "MIT"
        },
        "color": "#DAEAAA",
        "icon": "node-red/timer.svg",
        "status": {
            "x": 280,
            "y": 160,
            "wires": [
                {
                    "id": "7fc82258.93e36c",
                    "port": 1
                }
            ]
        }
    },
    {
        "id": "7fc82258.93e36c",
        "type": "function",
        "z": "9a693308.7ebaf",
        "name": "do operation",
        "func": "// @ts-ignore\nvar name = msg.perfName || env.get(\"name\");\n// @ts-ignore\nvar operation = msg.perfOperation || env.get(\"operation\");\nvar measures = global.get(\"flow_timers\") || {};\nvar measure = measures[name] || {};\n\nfunction doOp(measure, op){\n    if(operation === \"start\"){\n        measure.start = Date.now();//change to process.hrtime\n        measure.stop = null;\n        measure.durationMs = null;\n    } else if(operation === \"stop\") {\n        measure.stop = Date.now();//change to process.hrtime\n        measure.durationMs = measure.start ? measure.stop - measure.start : null;\n        msg._performance = measure;\n    }\n}\n\n\nif(operation === \"start\"){\n    doOp(measure, operation);\n} else if(operation === \"stop\") {\n    doOp(measure, operation);\n    node.send([null, { payload: { text: name + \": \" + measure.durationMs + \"ms\" }}]);\n} else if(operation === \"msg.topic\") {\n    operation = msg.topic;\n    doOp(measure, operation);\n} else if(operation === \"msg.operation\") {\n    operation = msg.operation;\n    doOp(measure, operation);\n} else if(operation === \"msg.payload\") {\n    operation = msg.payload;\n    doOp(measure, operation);\n} else {\n    return [msg, null];\n}\nmeasures[name] = measure;\nglobal.set(\"flow_timers\", measures);\n\nreturn [msg, null];",
        "outputs": 2,
        "noerr": 0,
        "initialize": "",
        "finalize": "",
        "libs": [],
        "x": 250,
        "y": 100,
        "wires": [
            [],
            []
        ]
    },
    {
        "id": "d82e74922cabc567",
        "type": "random-string",
        "z": "47900301b5f7a8d3",
        "size": "50000000",
        "characters": "",
        "property": "payload",
        "x": 580,
        "y": 300,
        "wires": [
            [
                "f6181e74.1732a"
            ]
        ]
    },
    {
        "id": "23b867b438bef31b",
        "type": "inject",
        "z": "47900301b5f7a8d3",
        "name": "",
        "props": [
            {
                "p": "payload"
            },
            {
                "p": "topic",
                "vt": "str"
            }
        ],
        "repeat": "",
        "crontab": "",
        "once": false,
        "onceDelay": 0.1,
        "topic": "",
        "payload": "",
        "payloadType": "date",
        "x": 260,
        "y": 300,
        "wires": [
            [
                "80b9b0b2.1dd84"
            ]
        ]
    },
    {
        "id": "db0c9a1d5590c769",
        "type": "debug",
        "z": "47900301b5f7a8d3",
        "name": "debug 1",
        "active": true,
        "tosidebar": true,
        "console": false,
        "tostatus": false,
        "complete": "false",
        "statusVal": "",
        "statusType": "auto",
        "x": 920,
        "y": 300,
        "wires": []
    },
    {
        "id": "80b9b0b2.1dd84",
        "type": "subflow:9a693308.7ebaf",
        "z": "47900301b5f7a8d3",
        "name": "",
        "env": [
            {
                "name": "name",
                "value": "function",
                "type": "str"
            }
        ],
        "x": 400,
        "y": 300,
        "wires": [
            [
                "d82e74922cabc567"
            ]
        ]
    },
    {
        "id": "f6181e74.1732a",
        "type": "subflow:9a693308.7ebaf",
        "z": "47900301b5f7a8d3",
        "name": "",
        "env": [
            {
                "name": "name",
                "value": "function",
                "type": "str"
            },
            {
                "name": "operation",
                "value": "stop",
                "type": "str"
            }
        ],
        "x": 760,
        "y": 300,
        "wires": [
            [
                "db0c9a1d5590c769"
            ]
        ]
    },
    {
        "id": "307e766c6c9a6a58",
        "type": "function",
        "z": "47900301b5f7a8d3",
        "name": "function 1",
        "func": "let characters = \"0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ\";\nlet size = msg.payload;\nlet payload = '';\n\t\t\tfor (let i = 0; i < size; i++) {\n\t\t\t\tpayload += characters.charAt(Math.floor(Math.random() * characters.length));\n\t\t\t}\nmsg.payload=payload;\nreturn msg;",
        "outputs": 1,
        "noerr": 0,
        "initialize": "",
        "finalize": "",
        "libs": [],
        "x": 560,
        "y": 380,
        "wires": [
            [
                "0853a821e09eee06"
            ]
        ]
    },
    {
        "id": "288b528a6b71c1dc",
        "type": "inject",
        "z": "47900301b5f7a8d3",
        "name": "",
        "props": [
            {
                "p": "payload"
            },
            {
                "p": "topic",
                "vt": "str"
            }
        ],
        "repeat": "",
        "crontab": "",
        "once": false,
        "onceDelay": 0.1,
        "topic": "",
        "payload": "50000000",
        "payloadType": "num",
        "x": 260,
        "y": 380,
        "wires": [
            [
                "e469935b15a999d2"
            ]
        ]
    },
    {
        "id": "e469935b15a999d2",
        "type": "subflow:9a693308.7ebaf",
        "z": "47900301b5f7a8d3",
        "name": "",
        "env": [
            {
                "name": "name",
                "value": "function",
                "type": "str"
            }
        ],
        "x": 400,
        "y": 380,
        "wires": [
            [
                "307e766c6c9a6a58"
            ]
        ]
    },
    {
        "id": "9aad27f0e84edf0e",
        "type": "debug",
        "z": "47900301b5f7a8d3",
        "name": "debug 2",
        "active": true,
        "tosidebar": true,
        "console": false,
        "tostatus": false,
        "complete": "false",
        "statusVal": "",
        "statusType": "auto",
        "x": 900,
        "y": 380,
        "wires": []
    },
    {
        "id": "0853a821e09eee06",
        "type": "subflow:9a693308.7ebaf",
        "z": "47900301b5f7a8d3",
        "name": "",
        "env": [
            {
                "name": "name",
                "value": "function",
                "type": "str"
            },
            {
                "name": "operation",
                "value": "stop",
                "type": "str"
            }
        ],
        "x": 740,
        "y": 380,
        "wires": [
            [
                "9aad27f0e84edf0e"
            ]
        ]
    }
]

How come the "same" code can be much faster (roughly twice) in the contrib node compare to the function ?

I have tried a significant number of time to run both. And the result is always the same.

What would be the most efficient way to create a very large random string / buffer ?

2

There is overhead in a function node as it is effectively a VM (sandbox) - then there is code implementation to consider. For example, strings are immutable - meaning everytime you do str = str + 'something you are creating new strings and destroying old strings over and over.

If you really want top speed then avoid string until the last moment & install "unsafe-function"

5000000

image
image1066Ă—495 39.5 KB

50000000

image
image1058Ă—482 39 KB

For reference - this is my-version - it was a quick modification of your-version (there are still improvements to be had)

const characters = "0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ";
const size = msg.payload;
const stringBuilder = []
for (let i = 0; i < size; i++) {
    stringBuilder.push(characters.charAt(Math.floor(Math.random() * characters.length)))
}
msg.payload = stringBuilder.join('')
return msg;
4 Likes
3

is unsafe function stable now? The project has not been updated for 4 years

4

No idea, not my node - but it seems to work.

For reference, I am currently using Node16

5

Thanks. I have tried the unsafe-function with your improved code, it is much faster.
And thank you for the explanation.
If I understand the logic of the unsafe-function node, it is safe (!) to use for this kind of basic functions, however, for more complex ones, it might be risky.

6

it is just named "unsafe" as it uses the real node environment (i.e. is not sandboxed) and so you could crash node-red if you change something in global scope (or mess with prototypes etc)

If you are just doing stuff like this (building strings etc) and nothing funky with object protos or proess/os stuff, then have at it :slight_smile:

7

I wonder if creating the array at the correct size up front is faster?

const stringBuilder = Array.from({ length: 50000000 })
for (let i = 0; i < stringBuilder.length; i++) {
    stringBuilder[i] = characters.charAt(Math.floor(Math.random() * characters.length))
}
8

According to this: Let’s get those Javascript Arrays to work fast | gamealchemist

This way of proceeding is ***10 times **** faster then a push() loop. …
(http://jsperf.com/push-allocated-vs-dynamic)

but it really depends on the JS engine and the optimisations it has made around array resizing.

2 Likes
9

The only way is play. :grin:

1 Like
10

I have tried it.
I am running on my mac, in a docker container.
For 50M buffer target, the option with const stringBuilder = Array.from({ length: 50000000 }) is almost three time slower than const stringBuilder = []

Don't ask me why :slight_smile:

11

Something to do with memory and garbage collection I suspect. But I don't really know the internals of node.js well enough to say for sure.

12

I suspect that that syntax creates an empty sparse array, rather than a chunk of memory ready to receive the data, in which case it won't help, and may make it worse.

13

I believe that the from with the length object is supposed to create an actual array. There is another approach that doesn't I believe. Anyway, as indicated, we have to test rather than assume - but obviously recognising that optimisation is a rabbit hole with diminishing returns.

14

You may well be right.

15

Does a 2x increase warrant a bit of play time :thinking:

Using crypto to generate random numbers.

image
image1154Ă—865 91.6 KB

Function Code (needs you to add crypto to the setup tab imports:

function randomString(length, charSet) {
    charSet = charSet || '0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ';
    const charSetLen = charSet.length
    const randomBytes = crypto.randomBytes(length);
    let result = []
    for (let i = 0; i < length; i++) {
        result[i] = charSet[randomBytes[i] % charSetLen];
    }
    return result.join("");
}
msg.payload = randomString(msg.payload)
return msg;
16

Depends whether it is 2ms to 1ms :blush: or more importantly, it depends on whether it matters to your workflow!

17

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.