Handling large message payloads and memory efficiency in custom Function nodes

General
1

Hi everyone,

I am working on a Node-RED automation flow that processes high-frequency data streams and running into a performance bottleneck within custom JavaScript Function nodes.

Current Setup:

  • Node-RED running on an edge server instance.

  • Flows ingesting large JSON payloads and transforming nested array structures via JavaScript inside standard Function nodes.

The Bottleneck: During peak data bursts, memory usage spikes noticeably as messages queue up, occasionally causing message delays or sluggish dashboard responsiveness.

While reviewing system optimization strategies and code efficiency benchmarks, I referenced a utility and troubleshooting guide over at https://cloudstream.pk

, but I wanted to see what architectural tips or performance tuning strategies this community recommends for optimizing memory management and payload processing inside Node-RED flows—specifically regarding message cloning, local context variables, or splitting heavy tasks across sub-flows.

Would love to hear your insights from your production environments! Thanks!

2

Hi There,

I did some work around the NodeJS Stream API and did a writeup on creating a node package for using that API as nodes inside Node-RED.

As the package says, it's work in progress/proof of concept - but does work and I did create a series of examples in a single flow example.

TL;DR: Use the stream API to break up large datasets (CSV and JSONL) into single messages before passing them through Node-RED.

3

Node-RED isn't really all that geared towards handling streams in my view. My first question would be - what are you aiming to get out of passing a stream of data through Node-RED?

Only by understanding that can we give you realistic advice on how to optimise things otherwise we are just flying blind.

If Node-RED is getting overwhelmed, consider moving the stream into a stream-oriented DB engine. Which might (I'm not an expert in that area) be able to handle the transforms more efficiently. Or "simply" create a separate node.js (or even Python if that's your thing) service dedicated to handling and transforming the stream. Maybe outputting whatever data you actually want to MQTT so that Node-RED can simply subscribe to the simpler values.

Certainly, absolutely avoid cloning fast-moving, complex objects but if you absolutely have to, make sure you are using the latest LTS version of node.js and native cloning, not the JSON library.

Node-RED function nodes are set up in their own node.js VM and broken down for each received message. This generally makes them a poor choice for handling large-volumes of complex messages. Though still better than a change node using JSONata.

Depending on what data you are handling and what you want to get out of it, you might also consider dropping messages - but without knowing any details, it isn't possible to say whether that is useful in this case.

These can be useful, especially for disaggregating flows - e.g. one flow to consolidate data, another to process it. But don't forget that they live in memory, all of them.

There is no point. Sub-flows are duplicated on each inclusion in a flow, they can make matters worse not better. Given what you've said, it sounds like you would be cloning messages to split them? If so, I think you might be multiplying the memory and CPU required?

Bottom line is that Node-RED, while great, can't be all things to all requirements. Processing of high-speed, complex data streams may well need a dedicated process to work along side node-red.

4

One important technique when handling messages containing large amounts of data is, where possible, to make sure each node only has one output wire connected. In particular remove any debug nodes connected to outputs that pass those messages. The reason is that if a node has more than one wire connected then the message is cloned before being passed to each wire other than the first one. Where there is only one output, and in the case of function nodes the msg object passed in is passed on using return msg for example, then any data objects contained will be passed by reference so there is no copying of the data.

5

A few Node-red versions back, the default behavior for handling flow messages has been changed so that they are now replicated, and not passed by reference (to avoid situations where modification of a message somewhere down the flow impacts the original).
There is an extra parameter in the send command which avoids the cloning and passes the original reference. In case of huge messages, this can reduce memory consumption & processing dramatically.

6

I am pretty sure that is not correct for the first output wire. Can you link to the docs for that?

[Edit] That is provided you do not use send() explicitly, but rely on return msg

7
8

I believe that is only talking about the use of node.send(). It does not mention what happens when return msg is used.

9

Hopefully one of the core devs will give us the definitive answer on this (whether messages are cloned in a function node with one wire connected to the output, where return msg is used to pass the message on).

10

I think that this flow shows that, with a function node that sends its message using return msg, the message sent on the first wire connected is the original message but if another wire is connected then the message is cloned before sending down the second wire.

image
image833Ă—175 15.4 KB

[{"id":"d4a5fd76abbd8807","type":"inject","z":"bdd7be38.d3b55","name":"Inject msg","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":200,"y":4960,"wires":[["e3d96cbb9bb7e449"]]},{"id":"e3d96cbb9bb7e449","type":"function","z":"bdd7be38.d3b55","name":"keep ref to msg in flow.msg and check identical","func":"flow.set('msg', msg)\nlet flowMsg = flow.get('msg')\nif (msg == flowMsg) {\n    node.status('msg == flowMsg')\n} else {\n    node.status('msg != flowMsg')\n}\nreturn msg;","outputs":1,"timeout":0,"noerr":0,"initialize":"","finalize":"","libs":[],"x":480,"y":4960,"wires":[["76e79a8b24450286","7237f23a1d2488b9"]]},{"id":"76e79a8b24450286","type":"function","z":"bdd7be38.d3b55","name":"Test msg on 1st wire","func":"let flowMsg = flow.get('msg')\nif (msg == flowMsg) {\n    node.status('msg == flowMsg')\n} else {\n    node.status('msg != flowMsg')\n}\nreturn msg;","outputs":1,"timeout":0,"noerr":0,"initialize":"","finalize":"","libs":[],"x":820,"y":4960,"wires":[[]]},{"id":"7237f23a1d2488b9","type":"function","z":"bdd7be38.d3b55","name":"Test msg on 2nd wire","func":"let flowMsg = flow.get('msg')\nif (msg == flowMsg) {\n    node.status('msg == flowMsg')\n} else {\n    node.status('msg != flowMsg')\n}\nreturn msg;","outputs":1,"timeout":0,"noerr":0,"initialize":"","finalize":"","libs":[],"x":820,"y":5040,"wires":[[]]}]
11

On that the point of cloning, that's a learning I made by starting the Erlang-Red project. Erlang-Red does not have this cloning issue. Erlang uses immutable data structures and thus messages sent, regardless of how many, all have the same reference to the data "object".

If a node wants to modify the data in the data object, it then "clones" the data object (how that is done is up to Erlang/BEAM). This is a fundamental part of Erlang that I didn't invent but that is extremely useful when doing this flow based programming.

This approach is far better because it saves unnecessarily cloning complex data structures when, for example, a message is sent to a debug node - which is essentially a "read-only" node. Read-only nodes could actually be sent an uncloned reference to the data structure but that is unknown at the point of cloning the message.

12

It is certainly possible to make JavaScript objects immutable (Object.freeze()). But it wasn't when Node-RED was created (freeze was baselined in 2015). Making that change now would doubtless break a lot of stuff. Also note that Object.freeze does a shallow freeze so for really complex objects, it might be better to use a proxy instead (baselined in 2016). In fact, I think Node-RED does use proxies for message objects now? If so, that might be an easier route for an option to make messages immutable. It would likely still break a lot of stuff of course.

Node-RED's cloning action came about due to the way that JavaScript passes complex objects by reference but does not block changes and because of the way it has developed async processing over the years to deal with its single-threaded operation. JavaScript is, of course, rather a mongrel language! I doubt anyone would choose to use it if it weren't for the fact that it became the de facto language of the web.

I would say that Node-RED's default message cloning may well be it's biggest "gotchya". It is the best general approach but can really catch you out when dealing with high volumes of complex messages.

13

Think this one is an Ai post, as they posted a second nearly identical question with an unrelated link, i hid that post.

14

Yes, but it led to an interesting conversation.

15

A thought, what happens if a junction is used on the output of the function node instead of the 2 wires coming direct from the function?

16

I believe that any output port having >1 connection will clone the message.

BTW, the previous comment about node.send - has anyone actually tested that? Does it actually prevent the message being cloned if 2 wires are connected to the output port?

17

According to the documentation

The Function node will clone every message object you pass to node.send to ensure there is no unintended modification of message objects that get reused in the function. Before Node-RED 1.0, the Function node would not clone the first message passed to node.send, but would clone the rest.

The Function can request the runtime to not clone the first message passed to node.send by passing in false as a second argument to the function. It would do this if the message contains something that is not otherwise cloneable, or for performance reasons to minimise the overhead of sending messages:

But nothing about what happens with 2 outputs

18

Thanks for checking that Jeff.

So it implies it makes no difference to the cloning that happens from 2 wires connected to the port. That makes more sense.

Of course, a function node can, these days, have >1 output port so this is less of an issue for it. Still, worth trying to remember that node.send will clone regardless unless you pass the optional parameter.

19

Having tested this with an output with two wires connected, node.send(msg) and node.send(msg, true) clone the msg on both wires, whereas node.send(msg, false) does not clone on the first wire, but does on the second.

As expected, it is cloned for the second wire.