Random reboots due FATAL ERROR; Reached Heap limit Allocation

General
1

Original post related to my issue was reboot looping of node-red. System seems to stabilize when removing majority of contect data and restoring swap memory for my system.

https://discourse.nodered.org/t/nod-red-keep-rebooting-due-fatal-error-reached-heap-limit-allocation-failed-javascript-heap-out-of-memory/77665/4

But I do still get random reboots from time to time (once every 1-3 days) and still the error log indicated running out of heap memory.

Attached lates error log from the time of reboot and also pictures of whoopsie.

error log:

May  6 01:03:52 SAMONE Node-RED[336912]: <--- Last few GCs --->
May  6 01:03:52 SAMONE Node-RED[336912]: [336912:0x4cc3d00] 195307453 ms: Scavenge 249.4 (261.8) -> 249.3 (266.8) MB, 2.4 / 0.0 ms  (average mu = 0.962, current mu = 0.847) allocation failure
May  6 01:03:52 SAMONE Node-RED[336912]: [336912:0x4cc3d00] 195307528 ms: Mark-sweep (reduce) 251.5 (266.8) -> 251.3 (262.8) MB, 10.0 / 0.1 ms  (+ 10.3 ms in 34 steps since start of marking, biggest step 4.0 ms, walltime since start of marking 75 ms) (average mu = 0.948, current mu = 0.848) fina
May  6 01:03:52 SAMONE Node-RED[336912]: <--- JS stacktrace --->
May  6 01:03:52 SAMONE Node-RED[336912]: FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
May  6 01:03:52 SAMONE Node-RED[336912]:  1: 0xb090e0 node::Abort() [node-red]
May  6 01:03:52 SAMONE Node-RED[336912]:  2: 0xa1b70e  [node-red]
May  6 01:03:52 SAMONE Node-RED[336912]:  3: 0xce19d0 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node-red]
May  6 01:03:52 SAMONE Node-RED[336912]:  4: 0xce1d77 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node-red]
May  6 01:03:52 SAMONE Node-RED[336912]:  5: 0xe993e5  [node-red]
May  6 01:03:52 SAMONE Node-RED[336912]:  6: 0xea90ad v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node-red]
May  6 01:03:52 SAMONE Node-RED[336912]:  7: 0xeabd25 v8::internal::Heap::HandleGCRequest() [node-red]
May  6 01:03:52 SAMONE Node-RED[336912]:  8: 0xe393c7 v8::internal::StackGuard::HandleInterrupts() [node-red]
May  6 01:03:52 SAMONE Node-RED[336912]:  9: 0xf92b85 v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_<true>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [node-red]
May  6 01:03:52 SAMONE Node-RED[336912]: 10: 0xf97471 v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_<false>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [node-red]
May  6 01:03:52 SAMONE Node-RED[336912]: 11: 0xf97dad v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_<false>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [node-red]
May  6 01:03:52 SAMONE Node-RED[336912]: 12: 0xf953c0 v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_<true>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [node-red]
May  6 01:03:52 SAMONE Node-RED[336912]: 13: 0xf947f5 v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_<true>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [node-red]
May  6 01:03:52 SAMONE Node-RED[336912]: 14: 0xf97471 v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_<false>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [node-red]
May  6 01:03:52 SAMONE Node-RED[336912]: 15: 0xf953c0 v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_<true>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [node-red]
May  6 01:03:52 SAMONE Node-RED[336912]: 16: 0xf97471 v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_<false>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [node-red]
May  6 01:03:52 SAMONE Node-RED[336912]: 17: 0xf97dad v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_<false>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [node-red]
May  6 01:03:52 SAMONE Node-RED[336912]: 18: 0xf98d8f v8::internal::JsonStringify(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>) [node-red]
May  6 01:03:52 SAMONE Node-RED[336912]: 19: 0xd63c37 v8::internal::Builtin_JsonStringify(int, unsigned long*, v8::internal::Isolate*) [node-red]
May  6 01:03:52 SAMONE Node-RED[336912]: 20: 0x15d9ef9  [node-red]
May  6 01:03:52 SAMONE systemd[2242]: Starting Notification regarding a crash report...
May  6 01:03:52 SAMONE update-notifier-crash[560580]: /usr/bin/whoopsie
May  6 01:03:52 SAMONE systemd[2242]: update-notifier-crash.service: Succeeded.
May  6 01:03:52 SAMONE systemd[2242]: Finished Notification regarding a crash report.
May  6 01:03:58 SAMONE systemd[2242]: Starting Notification regarding a crash report...
May  6 01:03:58 SAMONE update-notifier-crash[560586]: /usr/bin/whoopsie
May  6 01:03:58 SAMONE systemd[1]: nodered.service: Main process exited, code=dumped, status=6/ABRT
May  6 01:03:58 SAMONE systemd[1]: nodered.service: Failed with result 'core-dump'.
May  6 01:03:58 SAMONE update-notifier-crash[560588]: node
May  6 01:03:58 SAMONE systemd[1]: nodered.service: Scheduled restart job, restart counter is at 2.
May  6 01:03:58 SAMONE systemd[1]: Stopped Node-RED graphical event wiring tool.
May  6 01:03:58 SAMONE systemd[1]: Started Node-RED graphical event wiring tool.
May  6 01:03:58 SAMONE apport-gtk[560600]: Couldn't register with accessibility bus: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken.
May  6 01:03:58 SAMONE Node-RED[560602]: 6 May 01:03:58 - [info]
May  6 01:03:58 SAMONE Node-RED[560602]: Welcome to Node-RED
May  6 01:03:58 SAMONE Node-RED[560602]: ===================
May  6 01:03:58 SAMONE Node-RED[560602]: 6 May 01:03:58 - [info] Node-RED version: v3.0.2
May  6 01:03:58 SAMONE Node-RED[560602]: 6 May 01:03:58 - [info] Node.js  version: v16.20.0
May  6 01:03:58 SAMONE Node-RED[560602]: 6 May 01:03:58 - [info] Linux 5.4.0-148-generic x64 LE
May  6 01:03:59 SAMONE Node-RED[560602]: 6 May 01:03:59 - [info] Loading palette nodes
May  6 01:04:02 SAMONE Node-RED[560602]: 6 May 01:04:02 - [info] Dashboard version 3.4.0 started at /ui
May  6 01:04:02 SAMONE Node-RED[560602]: 6 May 01:04:02 - [warn] -----------------------------------------

Whoopsie pictures:

image
image2447Ă—889 122 KB

image
image2459Ă—800 69.7 KB

Memeory output

security@SAMONE:~$ free -h
              total        used        free      shared  buff/cache   available
Mem:           14Gi       7.0Gi       180Mi       214Mi       7.8Gi       7.7Gi
Swap:         4.0Gi       3.0Mi       4.0Gi
security@SAMONE:~$

Swappines=60 ;Cache Pressure = 80

In between I have downgraded my Node js to 16.20 as I saw some posts were 18.16 had created some issues (and also whoopsies was saying it is the part that had crashed)

@TotallyInformation @Steve-Mcl ; Any ideas next ?

Thanks Sami

2

Are you handling any large lumps of data, such as files, images, video?

3

So the first thing to note is that you don't seem to have a system issue (though there is a tiny chance that you have some bad memory locations but that is very unlikely these days). What you have is a data issue I believe. So playing further with swap, etc wont help.

But looking at the Mark-sweep line in the log, it appears that Node.js is maxing out the heap size at 256MB which isn't correct for an OS with 16GB of RAM.

Have you started Node-RED with a --max-old-space-size=256 parameter by any chance? If so, get rid of that and let node.js handle memory itself.

4

Hi @Colin,

No main data is just temp, humidity and such small data.

5

@TotallyInformation , Not to my knowledge. I start the Node Red as service in reboot

sudo systemctl enable nodered.service

I did not find either anything added into settings.js in .node.red folder? Might it be somewhere else ?

I found that I had at one point added following in my .bashrc as last command

export NODE_OPTIONS=--max-old-space-size=4096

I have now removed it. Lets see if that has any difference.

6

What is in the service script? You can find where it is by using
systemctl status nodered

7

@Colin

 nodered.service - Node-RED graphical event wiring tool
     Loaded: loaded (/lib/systemd/system/nodered.service; enabled; vendor preset: enabled)
     Active: active (running) since Sat 2023-05-06 01:03:58 EEST; 14h ago
       Docs: http://nodered.org/docs/hardware/raspberrypi.html
   Main PID: 560602 (node-red)
      Tasks: 11 (limit: 18271)
     Memory: 318.2M
     CGroup: /system.slice/nodered.service
             └─560602 node-red

May 06 09:46:39 SAMONE Node-RED[560602]: 6 May 09:46:39 - [info] [tradfri-config:IKEA GW] Accessory updated: 65544 Kaukosäädin MH
May 06 10:05:24 SAMONE Node-RED[560602]: 6 May 10:05:24 - [info] [e-mail:soiski71@gmail.com] Message sent: 250 2.0.0 OK  1683356724 r3-20020a19ac43000000b004f155743899sm159007lfc.92 - gsmtp
May 06 10:11:58 SAMONE Node-RED[560602]: 6 May 10:11:58 - [info] [tradfri-config:IKEA GW] Accessory updated: 65546 lamppu 2 MH
May 06 10:11:58 SAMONE Node-RED[560602]: 6 May 10:11:58 - [info] [tradfri-config:IKEA GW] Accessory updated: 65548 lamppu 4 MH
May 06 10:11:58 SAMONE Node-RED[560602]: 6 May 10:11:58 - [info] [tradfri-config:IKEA GW] Accessory updated: 65545 lamppu 1 MH
May 06 10:11:58 SAMONE Node-RED[560602]: 6 May 10:11:58 - [info] [tradfri-config:IKEA GW] Accessory updated: 65547 lamppu 3 MH
May 06 10:11:58 SAMONE Node-RED[560602]: 6 May 10:11:58 - [info] [tradfri-state:11155e9f.1336f1] Retrieving current state of accessories: [65545,65545] and groups: [0]
May 06 12:02:29 SAMONE Node-RED[560602]: 6 May 12:02:29 - [info] [tradfri-config:IKEA GW] Accessory updated: 65544 Kaukosäädin MH
May 06 13:53:11 SAMONE Node-RED[560602]: 6 May 13:53:11 - [info] [tradfri-config:IKEA GW] Accessory updated: 65569 Kaukosäädin OH
May 06 14:05:18 SAMONE Node-RED[560602]: 6 May 14:05:18 - [info] [tradfri-config:IKEA GW] Accessory updated: 65560 LAURAN
~
~
~
~
8

That tells you where nodered.service is (at the start). We need to see what is in there.

9

@Colin Sorry, running slow today :slight_smile:

# systemd service file to start Node-RED

[Unit]
Description=Node-RED graphical event wiring tool
Wants=network.target
Documentation=http://nodered.org/docs/hardware/raspberrypi.html

[Service]
Type=simple
# Run as normal pi user - change to the user name you wish to run Node-RED as
User=security
Group=security
WorkingDirectory=/home/security

Nice=5
Environment="NODE_OPTIONS=--max_old_space_size=256"
# uncomment and edit next line if you need an http proxy
#Environment="HTTP_PROXY=my.httpproxy.server.address"
# uncomment the next line for a more verbose log output
#Environment="NODE_RED_OPTIONS=-v"
#ExecStart=/usr/bin/env node $NODE_OPTIONS red.js $NODE_RED_OPTIONS
ExecStart=/usr/bin/env node-red-pi $NODE_OPTIONS $NODE_RED_OPTIONS
# Use SIGINT to stop
KillSignal=SIGINT
# Auto restart on crash
Restart=on-failure
# Tag things in the log
SyslogIdentifier=Node-RED
#StandardOutput=syslog

[Install]
WantedBy=multi-user.target

@TotallyInformation

And there the culprit is running the 256 only. Excluded the line now !

Thank you both on helping me on this !!

10

And a lesson for everyone having memory problems to take note of.

It is VERY rare to ever need --max_old_space_size even on devices with constrained memory. Let Node.js manage things itself, it does a much better job typically.

11

Did you use the bash install script to install node-red? That should have put a sensible value in there.

12

@TotallyInformation @Colin Strangely enough that even I removed the line and rebooted the whole machine I got next night exactly the same error and it was showing again the 256 limit reached. I have now manually (sorry I know...) added the 4096 limit to the .nodered.service. Lets see if that does the trick.

I believe the original problem comes that I have just updated the distros always and really no clean install even though I have changed a lot hardware during the years. If my memory serves me correctly 1st machine was only with 4GB ram running on it and seems that Node-Red installation script for ubuntu has hardcoded that limit at that time (talking Node-red 1.xx version years ago).

1 Like
13

I don't suppose this is anything to do with your heap allocaton failure issue, but it may help others who make an error similar mine, causing my Node-red to crash out of memory.

To track it down, in settings.js I set the logging level to "trace".

Looking in the log file (/var/log/syslog) I found 8000 copies of the line

May 13 11:41:23 localhost Node-RED[2052]: 13 May 11:41:23 - [warn] [function:function 32] 1

In function 32 I found an infinite loop:

for (let i = columns.length -1 ; i>=0; i) {
    node.warn (i)
}
14

4 Days and running with out memory issues any more.

1 Like
15

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.