Nod red keep rebooting due FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory

General
1

Here is my current problem.
Running Node red (3.0.2 / nodejs 18.16.0) locally on Ubuntu 18.04 LTS with 16 GB memory (Ryzen 5 machine) . Heap memory allocation is 4 GB. Started to experience reboots on node red a week ago. No new nodes were introduced at that time. InfluxDB and Grafana about week before seeing this problem.
Started node red in --safe mode an noticed couple node's were giving error so cleaned them up (node-red-contrib-arp and node-red-contrib-tradfri). Now node-red --safe opens nicely with no errors (couple warnings as some tuya devices are not yet fully online at boot). If I deploy same nodes from the safe mode everything works (all tabs and data collections and influxdb and grafana injects). If I try to open node red directly I get into the constant reboot cycle on node red.
This is taken from syslog when error / reboot occurred. When looking through Webmin the memory usage never goes above 50% when reboots happen.

Apr 17 17:51:31 SAMONE influxd-systemd-start.sh[1519]: [httpd] 127.0.0.1 - root [17/Apr/2023:17:51:31 +0300] "POST /write?db=IELEC&p=%5BREDACTED%5D&precision=n&rp=&u=root HTTP/1.1 " 204 0 "-" "-" 575bf107-dd2f-11ed-bbfa-04421a20a486 4517
Apr 17 17:51:31 SAMONE influxd-systemd-start.sh[1519]: [httpd] 127.0.0.1 - root [17/Apr/2023:17:51:31 +0300] "POST /write?db=IELEC&p=%5BREDACTED%5D&precision=n&rp=&u=root HTTP/1.1 " 204 0 "-" "-" 5777900c-dd2f-11ed-bbfb-04421a20a486 3350
Apr 17 17:51:31 SAMONE influxd-systemd-start.sh[1519]: [httpd] 127.0.0.1 - root [17/Apr/2023:17:51:31 +0300] "POST /write?db=IELEC&p=%5BREDACTED%5D&precision=n&rp=&u=root HTTP/1.1 " 204 0 "-" "-" 5777aef4-dd2f-11ed-bbfc-04421a20a486 5557
Apr 17 17:51:32 SAMONE systemd[1]: Stopping Node-RED graphical event wiring tool...
Apr 17 17:51:32 SAMONE Node-RED[178965]: <--- Last few GCs --->
Apr 17 17:51:32 SAMONE Node-RED[178965]: [178965:0x6bb43b0]     5666 ms: Scavenge 243.0 (259.5) -> 242.9 (260.5) MB, 2.4 / 0.0 ms  (average mu = 0.874, current mu = 0.576) allocation failure;
Apr 17 17:51:32 SAMONE Node-RED[178965]: [178965:0x6bb43b0]     5671 ms: Scavenge 243.9 (260.5) -> 243.8 (265.5) MB, 2.2 / 0.0 ms  (average mu = 0.874, current mu = 0.576) allocation failure;
Apr 17 17:51:32 SAMONE Node-RED[178965]: [178965:0x6bb43b0]     5736 ms: Mark-sweep 246.7 (265.5) -> 245.5 (269.0) MB, 43.9 / 0.0 ms  (average mu = 0.790, current mu = 0.454) allocation failure; scavenge might not succeed
Apr 17 17:51:32 SAMONE Node-RED[178965]: <--- JS stacktrace --->
Apr 17 17:51:32 SAMONE Node-RED[178965]: FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
Apr 17 17:51:32 SAMONE Node-RED[178965]:  1: 0xb7a940 node::Abort() [node-red]
Apr 17 17:51:32 SAMONE Node-RED[178965]:  2: 0xa8e823  [node-red]
Apr 17 17:51:32 SAMONE Node-RED[178965]:  3: 0xd5c940 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node-red]
Apr 17 17:51:32 SAMONE Node-RED[178965]:  4: 0xd5cce7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node-red]
Apr 17 17:51:32 SAMONE Node-RED[178965]:  5: 0xf3a3e5  [node-red]
Apr 17 17:51:32 SAMONE Node-RED[178965]:  6: 0xf4c8cd v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node-red]
Apr 17 17:51:32 SAMONE Node-RED[178965]:  7: 0xf26fce v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node-red]
Apr 17 17:51:32 SAMONE Node-RED[178965]:  8: 0xf28397 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node-red]
Apr 17 17:51:32 SAMONE Node-RED[178965]:  9: 0xf088e0 v8::internal::Factory::AllocateRaw(int, v8::internal::AllocationType, v8::internal::AllocationAlignment) [node-red]
Apr 17 17:51:32 SAMONE Node-RED[178965]: 10: 0xf00354 v8::internal::FactoryBase<v8::internal::Factory>::AllocateRawWithImmortalMap(int, v8::internal::AllocationType, v8::internal::Map, v8::internal::AllocationAlignment) [node-red]
Apr 17 17:51:32 SAMONE Node-RED[178965]: 11: 0xf026e8 v8::internal::FactoryBase<v8::internal::Factory>::NewRawTwoByteString(int, v8::internal::AllocationType) [node-red]
Apr 17 17:51:32 SAMONE Node-RED[178965]: 12: 0x1333f8c v8::internal::IncrementalStringBuilder::Extend() [node-red]
Apr 17 17:51:32 SAMONE Node-RED[178965]: 13: 0x1048668 v8::internal::JsonStringifier::SerializeDouble(double) [node-red]
Apr 17 17:51:32 SAMONE Node-RED[178965]: 14: 0x104ee6f v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_<false>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [node-red]
Apr 17 17:51:32 SAMONE Node-RED[178965]: 15: 0x104eb23 v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_<false>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [node-red]
Apr 17 17:51:32 SAMONE Node-RED[178965]: 16: 0x104c2ca v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_<true>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [node-red]
Apr 17 17:51:32 SAMONE Node-RED[178965]: 17: 0x104c60f v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_<true>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [node-red]
Apr 17 17:51:32 SAMONE Node-RED[178965]: 18: 0x104ee6f v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_<false>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [node-red]
Apr 17 17:51:32 SAMONE Node-RED[178965]: 19: 0x104c2ca v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_<true>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [node-red]
Apr 17 17:51:32 SAMONE Node-RED[178965]: 20: 0x104ee6f v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_<false>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [node-red]
Apr 17 17:51:32 SAMONE Node-RED[178965]: 21: 0x104eb23 v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_<false>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [node-red]
Apr 17 17:51:32 SAMONE Node-RED[178965]: 22: 0x104fbbf v8::internal::JsonStringify(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>) [node-red]
Apr 17 17:51:32 SAMONE Node-RED[178965]: 23: 0xde28f7 v8::internal::Builtin_JsonStringify(int, unsigned long*, v8::internal::Isolate*) [node-red]
Apr 17 17:51:32 SAMONE Node-RED[178965]: 24: 0x16fb7b9  [node-red]
Apr 17 17:51:32 SAMONE systemd[1]: nodered.service: Main process exited, code=dumped, status=6/ABRT
Apr 17 17:51:32 SAMONE systemd[1]: nodered.service: Failed with result 'core-dump'.
Apr 17 17:51:32 SAMONE systemd[1]: Stopped Node-RED graphical event wiring tool.

Any guidance how to solve? (please no remove everything and start from zero :slight_smile: )

2

Start by checking and sharing your OS memory allocations. Are you seeing swap happening?

Also, how large have you allowed your InfluxDB tables to get?

Also check any context variables in Node-RED - how large have they got? Are there any flows that are flinging very large data objects around? Do you have any looping flows?

3

These are my memory allocations. I do not use swap at all.

security@SAMONE:~$ free -h
              total        used        free      shared  buff/cache   available
Mem:           14Gi       6.2Gi       502Mi       245Mi       8.3Gi       8.5Gi
Swap:            0B          0B          0B
security@SAMONE:~$ java -XX:+PrintFlagsFinal -version | grep -iE 'HeapSize|PermSize|ThreadStackSize'
     intx CompilerThreadStackSize                   = 0                                   {pd product}
    uintx ErgoHeapSizeLimit                         = 0                                   {product}
    uintx HeapSizePerGCThread                       = 87241520                            {product}
    uintx InitialHeapSize                          := 251658240                           {product}
    uintx LargePageHeapSizeThreshold                = 134217728                           {product}
    uintx MaxHeapSize                              := 4024434688                          {product}
     intx ThreadStackSize                           = 1024                                {pd product}
     intx VMThreadStackSize                         = 1024                                {pd product}
openjdk version "1.8.0_362"
OpenJDK Runtime Environment (build 1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09)
OpenJDK 64-Bit Server VM (build 25.362-b09, mixed mode)
security@SAMONE:~$

Currently the influxDB is only 16Mb of size

security@SAMONE:~$ sudo du -sh /var/lib/influxdb/data/IELEC
[sudo] password for security: 
16M	/var/lib/influxdb/data/IELEC

Context node (stored in localfile) are not big either only 17 Mb

security@SAMONE:~$ sudo du -sh ~/.node-red/context
17M	/home/security/.node-red/context

Or did you have any specific command you want me to run to give more insight?

No large nodes etc nor any looping.

As said now running in --safe mode and everything seems normal. but if I run it as directly as node red service (sudo systemctl start nodered.service) it will start with reboot cycle.

4

Are you using that for something?

Everything else looks normal.

That actually seems rather large. Mine is only 2.4M and I have some pretty large objects. Also, I think that only measures things you've set to use the filesystem store, so what is in memory will potentially be much larger.

It would be good if you could measure the resource uses of Node-RED when it crashes if you can. On my 6GB RAM i5, Node-RED is using 11.0G of virtual memory, 206M resident, 40993b of shared (2.7% of memory) according to htop.

Clearly something in Node-RED is sucking up all of the available heap if it won't even start. That implies that you have something using that heap almost as soon as it starts. That might not be something actually using all the memory, it could be that node.js is trying to copy/move something in memory that is too large and so it can't find a big enough slot to handle it.

One other thing you can do is turn the Node-RED log level to max so that you can see more clearly when it crashes and what it was doing just before.

5

I've seen this issue when a user doesn't have any swap. Did you disable swap?

6

In that case it must be something different about the environment in those two cases.
Start node-red using node-red --safe, then deploy. Wait till it stabilises then copy/paste the node-red startup log here.
Do the same, but using node-red-start to start it, wait till it fails then again post the full startup log.

Also post the systemd script for node red, nodered.service, please.

7

Thank you to all that helped me on this.

I have now removed all context data that I used to restore some charts data and that seemed to have done the trick. Now system starts also in normal mode.
I belive some of the chart data had grown through the months and final combined with others had start eaten so much memory in startup (restoring charts) that the heap limit (as error said) had been reached. Majority of them were anyways now redundant as influxdb and grafana had taken their responsibility.

Thanks again to all @TotallyInformation @Steve-Mcl @Colin

8

I've seen this issue when a user doesn't have any swap. Did you disable swap?
[/quote]

Yes this was by design as though that 16 GB should be more than enough

9

"Should" is the operative word here.

As you have seen, disabling swap can have side effects.

You would be better off leaving swap enabled but adjust the swappiness

The Linux kernel provides a tweakable setting that controls how often the swap file is used, called swappiness

A swappiness setting of zero means that the disk will be avoided unless absolutely necessary (you run out of memory), while a swappiness setting of 100 means that programs will be swapped to disk almost instantly.

Ubuntu system comes with a default of 60, meaning that the swap file will be used fairly often if the memory usage is around half of my RAM. You can check your own system's swappiness value by running:

one@onezero:~$ cat /proc/sys/vm/swappiness
60
10

Adjusting that is very much a Linux black art!

What Is Swappiness on Linux? (and How to Change It) (howtogeek.com)

It requires some experimentation.

11

sorcery indeed... returned to 4GB

security@SAMONE:~$ free -m
              total        used        free      shared  buff/cache   available
Mem:          15350        5414        7440         136        2494        9486
Swap:          4095           0        4095
security@SAMONE:~$
12

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.