Out of memory crash on Windows 10 PC

General
1

I run Node-RED on a Windows 10 (Home) PC with 32 gigs of ram. Typically, the PC runs with around 9-10 gigs in use.
I am running with Node.js version 14.18.1 and Node-RED version 2.1.3

For the past many years, the odd crash has been acceptable, but as my flows grow in complexity and the number of users has grown dramatically, its more important to keep the system running as much as I can.

To this end, I started using an application to monitor and restart Node-RED on a crash. (Called 'Restart on Crash'). This has the unfortunate side effect of closing the cmd window that Node-RED was running in, and we lose the reason why - But it does keep things running while I get a few hours sleep...

The past few days, I have been manually starting Node-RED in a cmd window and the crash reason has been the same each time:

<--- Last few GCs --->

[11856:000001F83FB74F40] 84155306 ms: Mark-sweep 4071.3 (4121.3) -> 4055.1 (4121.8) MB, 1649.1 / 0.6 ms  (average mu = 0.163, current mu = 0.073) allocation failure scavenge might not succeed
[11856:000001F83FB74F40] 84157107 ms: Mark-sweep 4071.7 (4122.0) -> 4055.4 (4122.5) MB, 1658.7 / 0.4 ms  (average mu = 0.123, current mu = 0.079) allocation failure scavenge might not succeed


<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 00007FF7A0FB412F napi_wrap+133311
 2: 00007FF7A0F4DD06 SSL_get_quiet_shutdown+63062
 3: 00007FF7A0F4EB9D node::OnFatalError+301
 4: 00007FF7A18319CE v8::Isolate::ReportExternalAllocationLimitReached+94
 5: 00007FF7A18167BD v8::SharedArrayBuffer::Externalize+781
 6: 00007FF7A16BFFCC v8::internal::Heap::EphemeronKeyWriteBarrierFromCode+1516

Looking at the forums I see a lot of options and support for running on Linux (and more so on a Pi), but very little on how to allocate Node.js more ram on Windows.
Turning to Google and StackOverflow etc, I see some options.

I first tired:

C:\Users\tbg>node-red --max-old-space-size=8192

That did not make many difference to the time between crashes (20 to 30 hours).

Digging more shows this is worth a try:

C:\Users\tbg>node --max-old-space-size=81928 red.js --userDir C:\Users\tbg\AppData\Roaming\npm\node_modules\node-red\red.js

This does not run and throws the following error:

internal/modules/cjs/loader.js:905
  throw err;
  ^

Error: Cannot find module 'C:\Users\tbg\red.js'
?[90m    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:902:15)?[39m
?[90m    at Function.Module._load (internal/modules/cjs/loader.js:746:27)?[39m
?[90m    at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:76:12)?[39m
?[90m    at internal/main/run_main_module.js:17:47?[39m {
  code: ?[32m'MODULE_NOT_FOUND'?[39m,
  requireStack: []
}

red.js is in that path I used and I don't know why the error does not use the path I supply?

Long story long.
How do I start Node-RED in Node.js with more RAM on a Windows PC?

Thanks.

2

The short answer is that you don't allocate more memory to Node.js on any platform, from Node.js v12 onwards, the heap is allocated according to the available memory. If you are using a version less than that, you need to upgrade.

What you need to do is find out what is using up all the heap. Typically this would be a memory leak unless you are trying to run Windows on a device with far too little memory.

The other question is how are you running Node.js? Windows has its scheduler which will keep a process running so if you need to run Node-RED as a service, that may be the best way. You can redirect the logs to a file.

Alternatively you can use PM2 to manage running it. It has its own log management. However, it is quite chunky in its own right so maybe not the best if you are already short on memory.

3

The PC has 32 gig of RAM with around 9 or 10 used by different apps (browser etc). So I am not even close to resource bound at all.

I am running Node-RED by clicking on Windows Start, typing in cmd, in the dos box that opens, I type node-red and press enter.

I do it this way so I can see any errors and so I can see the crash reason(s).
I tried writing the log to a file, but cant get it to work and am told via the Internet that method wont write the crash reason the file.

4

Then you have something that is using the heap faster than the Node.js process loop can clear it. As I say, this would often be a memory leak from some code. In node-red, this is typically a contrib node that isn't clearing up for itself. Maybe check through your installed nodes for somthing that hasn't been updated in a while, that's a good place to start.

Not a good way to run a node.js app if you need to rely on it. Maybe try something like node-windows - npm (npmjs.com). This creates a service wrapper, intelligently does restarts for you and handles logging. If you want to keep something more interactive, try PM2.

BTW, you can see what Heap size you have by putting this at the start of your settings.js file, it will output to console, change that to a file if you need to.

/** Optionally display the Node.js/v8 engine heap size on startup */
const v8 = require('v8')
console.info(`V8 Total Heap Size: ${(v8.getHeapStatistics().total_available_size / 1024 / 1024).toFixed(2)} MB`)
let mem = process.memoryUsage()
const formatMem = (m) => ( m/1048576 ).toFixed(2)
console.info(`Initial Memory Use (MB): RSS=${formatMem(mem.rss)}. Heap: Used=${formatMem(mem.heapUsed)}, Tot=${formatMem(mem.heapTotal)}. Ext C++=${formatMem(mem.external)}`)

On my (currently 16GB dev PC), it shows:

0|Node-RED  | V8 Total Heap Size: 4123.72 MB
0|Node-RED  | Initial Memory Use (MB): RSS=48.45. Heap: Used=19.31, Tot=30.82. Ext C++=1.72
0|Node-RED  | 18 Nov 17:31:50 - [info]
0|Node-RED  | Welcome to Node-RED
0|Node-RED  | ===================

(Running under PM2 manually). You can see that the heap is around 4GB. You can see though that the actual memory in use is far lower than the total heap available.

I also have the memory use line embedded in my uibuilder dev version so that I can make sure I'm not leaking memory.

5

Thanks very much for your time to write a detailed reply.

I'd really like to try this vs PM2 which sounds a bit heavy, but it all seems well above my pay grade, I am not sure what I need to tweak in each of the 19 code snippets on that that page I need to change to run and log Node-RED.

On top of that, it will not address my core issue. Very confused here.....

6

Do you have any thoughts on how I might do that?
All I have to go on is the crash log?
There are no node names mentioned in there.
I have been very careful to keep everything as up to date as I can.
Is there another layer of debug info I can add to find out what is causing the issue?

I was eyes on the computer today when it crashed and it still had over 16gigs of RAM to go, so its just not using the ram. Its like Node.js / Node-RED is using kbytes of the RAM, not the gigs I'd like it to use.

7

As I said, it isn't running out of memory per-se, it is running out of heap. This means that memory is being allocated/used faster than it can be cleared. There are plenty of generic guides out there that give ideas about how to reduce the issue - of course, these don't always directly help with Node-RED which is hiding the complexities of Node.js from you.

But as I also said, in Node-RED, the starting place is to look at what contrib nodes you are using. Firstly make sure that they are current. 2nd would be to check here and on GitHub for issues with the nodes you are using. 3rd would be to see if there are alternative nodes.

It is possible to run node-red in a debug environment which would potentially let you get into the weeds with what is using up memory. However, this is a really complex task and requires a fair bit of knowledge to set up as well as to make sense of.

I looked again at your other issue about manual start. You've mixed up the order of the parameters:

node C:\Users\tbg\AppData\Roaming\npm\node_modules\node-red\red.js --max-old-space-size=81928 red.js

That should work and will use the default ~/.node-red folder for userDir. This will likely delay the problem but at the cost of longer pauses in your flows when GC does actually kick in.

Actually it is very easy. You have enough memory not to worry about its overheads.

https://www.google.com/search?q=running+node+red+under+pm2

8

If you use node-red-mode-mysql 0.3.0, it can cause memory leak (see here).
You can use node-red-contrib-gc to check if memory is leaking in flow like that:

изображение
изображение795×133 9.26 KB 

[{"id":"693c807aba34e169","type":"inject","z":"170b10c08a03ffc3","name":"1 min","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"60","crontab":"","once":false,"onceDelay":0.1,"topic":"","payloadType":"date","x":330,"y":960,"wires":[["7f26bf7872b32147"]]},{"id":"7f26bf7872b32147","type":"function","z":"170b10c08a03ffc3","name":"Show Memory Heap Usage","func":"var heap_usage = flow.get(\"heap_usage\", msg.payload.after);\nmsg = undefined;\nreturn [{topic : \"HeapTotal\", payload : heap_usage.totalHeapSize/1024/1024}, {topic : \"HeapUsed\", payload : heap_usage.usedHeapSize/1024/1024}];","outputs":2,"noerr":0,"initialize":"","finalize":"","libs":[],"x":620,"y":960,"wires":[["f744096.dfe3af8"],["f744096.dfe3af8"]]},{"id":"f744096.dfe3af8","type":"ui_chart","z":"170b10c08a03ffc3","name":"Memory Heap Usage","group":"1ff820d1b48f0454","order":0,"width":0,"height":0,"label":"","chartType":"line","legend":"true","xformat":"HH:mm","interpolate":"linear","nodata":"Garbage collections","dot":false,"ymin":"","ymax":"","removeOlder":"24","removeOlderPoints":"","removeOlderUnit":"3600","cutout":0,"useOneColor":false,"useUTC":false,"colors":["#1f77b4","#aec7e8","#ff7f0e","#2ca02c","#98df8a","#d62728","#ff9896","#9467bd","#c5b0d5"],"outputs":1,"useDifferentColor":false,"className":"","x":900,"y":960,"wires":[[]]},{"id":"c79dbd16.6f388","type":"function","z":"170b10c08a03ffc3","name":"SaveMemoryHeapUsage","func":"flow.set(\"heap_usage\", msg.payload.after);\nreturn null;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":910,"y":1020,"wires":[[]]},{"id":"d4273bb3.2956f8","type":"gc","z":"170b10c08a03ffc3","minor":true,"major":true,"incremental":true,"weak":true,"all":true,"name":"","x":310,"y":1020,"wires":[["c79dbd16.6f388"]]},{"id":"1ff820d1b48f0454","type":"ui_group","name":"Memory Heap Usage, MBytes","tab":"89b9dc8710eeab40","order":6,"disp":true,"width":"10","collapse":false,"className":""},{"id":"89b9dc8710eeab40","type":"ui_tab","name":"System","icon":"dashboard","order":4,"disabled":false,"hidden":false}]

Also I guess you may have some possible memory leak warning shortly after Node-Red restart in some of Windows logs and/or Node-Red log.

9

The site crashed again just now, so I was able to make some changes.

@TotallyInformation Thanks for the two tips.

Here is what my heap looks like after adding your small stub to the front of the settings.js file:
V8 Total Heap Size: 4127.68 MB Initial Memory Use (MB): RSS=41.23. Heap: Used=15.72, Tot=29.05. Ext C++=1.83

When I started like this, it started a blank project, none of my flows were there. Is there a way I can use this command with my exiting project as I really like the idea of kicking the GC in and will need to see what sort of delays you describe actually look like - they might be totally acceptable vs the crashes.

@ls819011

I really want to use this node and graph, but just for those following along, this node has the most extreme requirements of any node I have ever used... It requires not just Python 3, but also Studio Code with desktop C++ support. All up its about 10gb of drive space.
I have not yet got this node to install, its still throwing errors, but I hope its worth it to see what's going on with the heap garbage collection.

10

I have node-red-node-mysql@next in beta right now - so feel free to install and try it... there are some possible changes as we use a different underlying library (mysql2 vs mysql) - the main breaking one being the timezone MUST be specified (if a t all using +/-hh:mm format and not a name like GMT.
(aka - hence why it's a beta and we are trying to find out anything it breaks... - but at least it doesn't seem to leak memory). To try it...

cd ~/.node-red
npm i node-red-node-mysql@next

then restart

11

Yes, that's about what I'd suspect. And shows, as we already surmised, that you have loads of free heap at the start. It confirms as well that you have 4GB of heap available to use.

Please read the docs about starting Node-RED from the command line. But without a --userDir parameter, it should have used the default ~/.node-red/ folder which is where your flow files, settings.js, etc would all be with a standard install.

When Node-RED starts, it prints to the log (to the console if starting manually) what folders it is using. So it would be helpful to see that startup output.

Yes, as the docs state, use the --userDir parameter if you are using something other than the default folder.

There are articles online that indicate what the GC speed is for a given memory size. It can cause a 1+ second pause with larger heap sizes.

It is a very easy setting though so I recommend just trying it.

It isn't as bad as it seems. As you are running on Windows, all that is happening is that you are having to install some C/C++ compiler tools. That requires Python and the small part of Visual Studio that contains the c++ command line compiler, it isn't the whole of VS. You don't notice this on Linux typically because there is already Python and the gcc compiler already installed. Without these tools nd libraries installed, npm cannot compile C++ libraries for your platform.

Actually, I just tried to install the gc node for myself. It is dependent on gc-stats which is the culpret here. For some reason, it is downloading and installing a load of C++ library source code and then trying to compile it. However, instead of relying on the version of the Windows C++ compiler that used to be installed with the windows developer build package but now comes natively installed with Node.js for Windows (v16+ I think), it is trying to force you to install a full copy of VS - madness.

You don't need that for the basic stats anyway. If you just want to monitor the heap usage, Try sticking my code into a function node and running it periodically. You won't get the timing data but you will get the memory utilisation.

12

At this point I would just like to figure out how to double the 4gig heap to 8, but I guess that is only going to push my crashes from every ~24 hours to ~48.

Starting Node-RED in Linux prints the user dir, but not in Windows.
Here is my start up output:

C:\Users\tbg>node-red
V8 Total Heap Size: 4127.68 MB
Initial Memory Use (MB): RSS=41.23. Heap: Used=15.72, Tot=29.05. Ext C++=1.83
23 Nov 13:58:05 - [error] [function:oceanic box] SyntaxError: Unexpected token 'return' (body:line 36)
23 Nov 13:58:05 - [error] [function:Extract weather data] SyntaxError: Unexpected token '.' (body:line 22)
(node:26408) Warning: Setting the NODE_TLS_REJECT_UNAUTHORIZED environment variable to '0' makes TLS connections and HTTPS requests insecure by disabling certificate verification.
(Use `node --trace-warnings ...` to show where the warning was created)

That's the start up, then it just logs things as they happen.

I did not think that ~/.node-red/ worked under Windows?
Much to learn I have it seems.

Just doing a full backup now so I can test some more options / paths to try and start it the way you are suggesting, but some how find my current flows.

Thanks for testing that node. I will give up trying to get it installed and do as you suggest, I will run your code every 15 minutes and see what I can see.

13

I took the site down for about 15 minutes and tested a lot of different --userDir options from the docs and none of them gave me my flows. I got blank flows and 'Welcome to Node-RED 2.0' intro splash screens, but nothing with my flows.
Would like to figure out how to start Node-RED with my flows and --max-old-space-size=81928

I next tried the new node-red-node-mysql@next and had the same insert issues as last time I tried to use a different version of the MySQL node.

Learning from that last attempt, I had a full .node-red back up so just copied all the nodes back over and I am back up and running for the next 24 hour crash cycle.

Next job is to put the GC script into a function block and graph it to see whats going on.

Grateful for all the help, not a chance I would digging into this with out the support.

14

Looks like you have lots of other problems you need to sort out.

It's a shorthand. It works under PowerShell, not under CMD prompt. Windows has its own ways of identifying your user root folder. Or just use the complete path. Something like c:\users\<username>\.node-red.

1 Like
15

Could you spare a moment to help me with this? I get errors about require is not defined and I am not sure how to remove the stuff about the console.info.

I just did as you suggested and put the code in a function block with an inject and debug either side.

Thanks.

16

Do you want to start a new thread on this?
If you get a version that fixes the insert issue, I am happy to try it out.

17

First you need to add this to your settings.js:

    /** Allow external modules to be specified in function nodes
     * See https://nodered.org/docs/user-guide/writing-functions#using-the-functionexternalmodules-option
     */
    functionExternalModules: true,

Then you can use the following flow:

[{"id":"7d0ac0eb47daffe0","type":"function","z":"d0860be6.7951b8","name":"Memory Utilisation","func":"//const process = require('process')\nconst formatMem = (m) => (m / 1048576).toFixed(2)\n\n// You don't need this as it doesn't change - leave it in the start of your settings.js file\n//const v8 = require('v8')\n//msg.payload.V8_Total_Heap_Size = (v8.getHeapStatistics().total_available_size / 1024 / 1024).toFixed(2)\n\nlet mem = process.memoryUsage()\n\nmsg.payload = {\n    RSS: formatMem(mem.rss),\n    Heap: {\n        Used: formatMem(mem.heapUsed),\n        Tot: formatMem(mem.heapTotal),\n        Ext_C: formatMem(mem.external)\n    }\n}\n\nreturn msg","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[{"var":"process","module":"process"}],"x":950,"y":60,"wires":[["a7475d77d2e17aaa"]]},{"id":"a56607d761ee32e4","type":"inject","z":"d0860be6.7951b8","name":"","props":[{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"Memory Utilisation","x":825,"y":60,"wires":[["7d0ac0eb47daffe0"]],"l":false},{"id":"a7475d77d2e17aaa","type":"debug","z":"d0860be6.7951b8","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":1075,"y":60,"wires":[],"l":false}]

Which produces the following:
image

1 Like
18

Reading that old thread I wasn't able to work out what the exact issue was. Do you have any new debug of the insert so we can see what is the latest problem? At some point we need to fix your insert syntax as the latest version does seem to fix the memory leak (for others) so catch 22 really.

19

Uh, yeah, that's where I am at.
The catch node does not catch the failed inserts and only some fail, so putting a node.warn it does not work.
I insert around 1 to 1.5 million records a day, so the debug tab fills up pretty quick.
I simply don't know how to catch the required debug information.

All inserts work with the older version node, badly need to solve this memory crash issue, but cant find which inserts are breaking the new version.

Catch 22.

20

the node should either output to a catch node - or it's output. If it's not causing an error then it must be sending an output - but it may need to be decoded. usually it will say something like number rows affected or the insertId or warningStatus etc - in the failure case then one of those must be "not normal" in some way - either not present or strange value.

You can of course send the debug node output also to the console - and indeed pipe that to a file - (by adding > mydebug.log to the command to run Node-RED. - If it fails as quickly as you said last night the file shouldn't grow too large - and can then be edited down - or just sent to me direct t have a look at...

Thanks