JavaScript heap out of memory after message processing ends

[KronwarsCZ]

Hi everyone!
I am facing a weird issue recently after (I believe) a message processing ends. After about 30 seconds after a message is processed, I get a Javascript heap out-of-memory error and my whole docker container hangs. While testing, I can see a sudden jump in CPU usage to 100% and memory jump from normal 950MB to 1.2 GB, followed by a crash.

Right now I am stuck, as I don't know how to debug it more. As far as I can tell, the message is processed and no loop is happening, and yet I can't seem to find out where the issue lies.

Here are docker container logs I can see:

node-red-1  | 2025-06-24T12:22:00.591392037Z 24 Jun 14:22:00 - [info] [debug:No new messages] No new messages.
node-red-1  | 2025-06-24T12:22:00.638293206Z 24 Jun 14:22:00 - [info] [debug:No new messages] No new messages.
node-red-1  | 2025-06-24T12:22:23.253549851Z 24 Jun 14:22:23 - [info] [debug:Store log to system console] info - db86819d-0919-46c1-a581-15463b0b6422 - Message is TEST, continue with processing.
node-red-1  | 2025-06-24T12:22:23.257691358Z 24 Jun 14:22:23 - [info] [debug:Store log to system console] info - db86819d-0919-46c1-a581-15463b0b6422 - MS Graph: Moving message to folder DONE.
node-red-1  | 2025-06-24T12:22:23.694448125Z 24 Jun 14:22:23 - [info] [debug:Store log to system console] info - db86819d-0919-46c1-a581-15463b0b6422 - MS Graph: Message was successfully moved to folder DONE.
node-red-1  | 2025-06-24T12:22:30.742263897Z 24 Jun 14:22:30 - [info] [debug:No new messages] No new messages.
node-red-1  | 2025-06-24T12:22:35.626520064Z 24 Jun 14:22:35 - [info] [debug:No new messages] No new messages.
node-red-1  | 2025-06-24T12:22:37.295473978Z 24 Jun 14:22:37 - [info] [debug:No new messages] No new messages.
node-red-1  | 2025-06-24T12:23:11.407285854Z
node-red-1  | 2025-06-24T12:23:11.407391845Z <--- Last few GCs --->
node-red-1  | 2025-06-24T12:23:11.407415207Z
node-red-1  | 2025-06-24T12:23:11.407425389Z [7:0x7fee4a091650]   533729 ms: Mark-Compact 967.6 (1002.5) -> 963.8 (1003.8) MB, 1670.52 / 0.00 ms  (average mu = 0.287, current mu = 0.020) allocation failure; scavenge might not succeed
node-red-1  | 2025-06-24T12:23:11.407436961Z [7:0x7fee4a091650]   535699 ms: Mark-Compact 971.6 (1003.8) -> 967.8 (1006.3) MB, 1935.83 / 0.00 ms  (average mu = 0.169, current mu = 0.018) allocation failure; scavenge might not succeed
node-red-1  | 2025-06-24T12:23:11.407465871Z
node-red-1  | 2025-06-24T12:23:11.407476816Z
node-red-1  | 2025-06-24T12:23:11.407485761Z <--- JS stacktrace --->
node-red-1  | 2025-06-24T12:23:11.407494772Z
node-red-1  | 2025-06-24T12:23:11.407503561Z FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
node-red-1  | 2025-06-24T12:23:11.407512493Z ----- Native stack trace -----
node-red-1  | 2025-06-24T12:23:11.407521262Z
node-red-1  | 2025-06-24T12:28:04.990361675Z ./entrypoint.sh: line 14:     7 Aborted                 (core dumped) /usr/local/bin/node $NODE_OPTIONS node_modules/node-red/red.js --userDir /data $FLOWS "${@}"
node-red-1 exited with code 0

Do you have any ideas as to why this might be happening and where could I look to investigate more?

Running in docker, image tag nodered:4.0.9
Node-RED version: v4.0.9
Node.js  version: v20.19.0
Linux 5.14.0-570.22.1.el9_6.x86_64 x64 LE
The machine this is running on has 2GB of RAM and 2GB of swap

[Steve-Mcl]

Hi, welcome to the forum.

@KronwarsCZ

While testing, I can see a sudden jump in CPU usage to 100% and memory jump from normal 950MB to 1.2 GB, followed by a crash.

@KronwarsCZ

As far as I can tell, the message is processed and no loop is happening

I know, but really does smell like an uncontrolled loop (or recursion)

@KronwarsCZ

Do you have any ideas as to why this might be happening

There are many threads on the forum for this topic: Search results for '"JavaScript heap out of memory" order:latest' - Node-RED Forum

This one was a little different.

@Steve-Mcl

The error get thrown in a variety of situations, here are a few:

• When attempting to process large amounts of data or execute complex algorithms that require a lot of memory.
• When running a Node.js server with a large number of connections or concurrent requests.
• When running a Node.js application in a container or virtual machine with limited memory resources.

which was resolved like this:

@Steve-Mcl

we do add --max_old_space_size=512. What that means is when this heap fills up, Node.js needs to perform a garbage collection (GC) cycle. Coupled with your comment about "now not able to modify flows" (my mind reads "you have recently modified flows") from that, I dont think it is a stretch to conclude something has changed and the GC is now unable able to keep up with memory allocations.

Are you able to try something out? If possible, edit the file lib/launcher.js in your device agent installation and change the line to something like '--max_old_space_size=1024' (or remove it altogether)

Where you set max_old_space_size will depend on how you are running node-red (do a search)

but the 100% CPU spike really really smells like a hard, blocking loop - so perhaps a buggy contrib? or some recursive (self calling) function node code? or an invisible loop caused by link nodes or MQTT?

[KronwarsCZ]

Hi!
Thanks for the quick reply. I was able to find out with an inspection node and chrome devtools debugger (which thankfully paused execution just before node-red crashed) that the issue was in JSON.stringify where it tried to stringify a very large (1GB) json object in global storage for whatever reason.

This also leads me to another question, does global.get() return by value or by a reference?
Here is a function node that parses all logs created by different nodes/subflows during message processing and sends them to Slack.

const allLogs = global.get("logs");
const logObject = allLogs[msg.log_id];
let logs = logObject[logs]

...some processing on logs...

delete allLogs[msg.log_id]
global.set("logs", allLogs);

return xxx;

There may be a race condition happening at the delete and global.set which causes the whole log object to grow larger and larger? The log object is supposed to hold log messages only during processing and delete them after a message has been processed.

We have recently upgraded from Node-RED 2.2.3 to Node-RED 4.0.9 and this didn't happen before.

[Colin]

@KronwarsCZ

global.get() return by value or by a reference

Reference.

As for the race condition the best thing is to avoid context. Pass the data in messages instead.