Node Red crashed - why?

Flixm · 7 March 2022 16:15

Hi,
the last time neary every day my Node Red restart by itself and i could not find out why.
I worked with a few charts the last time. Maybe that could be a reason ?

Attached the LOG of the "restart":

7 Mar 17:01:39 - [info] Geänderte Nodes sind gestoppt
7 Mar 17:01:40 - [info] Geänderte Nodes werden gestartet
7 Mar 17:01:40 - [info] Geänderte Nodes sind gestartet
<--- Last few GCs --->
[7674:0x40a1dc8] 18553927 ms: Scavenge 238.6 (258.4) -> 237.0 (258.4) MB, 2.1 / 0.0 ms (average mu = 0.985, current mu = 0.988) task
[7674:0x40a1dc8] 18554863 ms: Scavenge 238.7 (258.4) -> 237.1 (258.7) MB, 5.3 / 0.0 ms (average mu = 0.985, current mu = 0.988) task
[7674:0x40a1dc8] 18555337 ms: Mark-sweep 238.9 (258.7) -> 224.2 (258.7) MB, 434.7 / 0.1 ms (average mu = 0.989, current mu = 0.991) task scavenge might not succeed
<--- JS stacktrace --->
FATAL ERROR: MarkCompactCollector: young object promotion failed Allocation failed - JavaScript heap out of memory
<--- Last few GCs --->
[7674:0x40a1dc8] 18553927 ms: Scavenge 238.6 (258.4) -> 237.0 (258.4) MB, 2.1 / 0.0 ms (average mu = 0.985, current mu = 0.988) task
[7674:0x40a1dc8] 18554863 ms: Scavenge 238.7 (258.4) -> 237.1 (258.7) MB, 5.3 / 0.0 ms (average mu = 0.985, current mu = 0.988) task
[7674:0x40a1dc8] 18555337 ms: Mark-sweep 238.9 (258.7) -> 224.2 (258.7) MB, 434.7 / 0.1 ms (average mu = 0.989, current mu = 0.991) task scavenge might not succeed
<--- JS stacktrace --->
FATAL ERROR: MarkCompactCollector: young object promotion failed Allocation failed - JavaScript heap out of memory
nodered.service: Main process exited, code=killed, status=6/ABRT
nodered.service: Failed with result 'signal'.
nodered.service: Consumed 1h 24min 3.200s CPU time.
nodered.service: Scheduled restart job, restart counter is at 6.
Stopped Node-RED graphical event wiring tool.
nodered.service: Consumed 1h 24min 3.200s CPU time.
Started Node-RED graphical event wiring tool.
7 Mar 17:03:31 - [info]
Willkommen bei Node-RED!

The Node Red is running on a Pi 4 2GB

Maybe someone could help me run out of this problem

Colin · 7 March 2022 16:32

It could be the charts, or are you manipulating images or video or something that consumes large amounts of memory?

How many charts have you got, how many lines on each chart, at what rate is new data added to the charts and what is the time range specified for the charts?

Flixm · 8 March 2022 05:57

Hi,
Actually I have 2 chats. One with 4 lines every minute about 3 days, and another one with one line every minute.
What is the best setting for a chart?

Flixm · 8 March 2022 08:21

Is it the memory RAM ? Or the size of the SD-card from the pi?

Colin · 8 March 2022 09:54

Once a minute is 1440 per day, so if the chart is three days then each line will have 4320 points on it. You have a lotal of 5 lines across the two charts so that is a total of 21,600 points to be shuffled down the chart once a minute. That is quite a lot but will certainly not cause it to run out of memory.

Yes it is RAM that is running out.

Can you post more of the log please, from well before the crash to after the node-red startup messages please.

Flixm · 8 March 2022 18:51

There's not much more then before:
Mar 15:00:00 - [warn] [ui_level:37703d51fa457d11] msg.payload does not contain numeric value
8 Mar 15:00:00 - [warn] [ui_level:37703d51fa457d11] msg.payload does not contain numeric value
8 Mar 15:00:00 - [warn] [ui_level:37703d51fa457d11] msg.payload does not contain numeric value
8 Mar 15:00:00 - [warn] [ui_level:37703d51fa457d11] msg.payload does not contain numeric value
8 Mar 15:00:00 - [warn] [ui_level:37703d51fa457d11] msg.payload does not contain numeric value
8 Mar 15:00:00 - [warn] [ui_level:37703d51fa457d11] msg.payload does not contain numeric value
8 Mar 15:00:00 - [warn] [ui_level:37703d51fa457d11] msg.payload does not contain numeric value
8 Mar 15:00:00 - [warn] [ui_level:37703d51fa457d11] msg.payload does not contain numeric value
8 Mar 15:00:00 - [warn] [ui_level:37703d51fa457d11] msg.payload does not contain numeric value
8 Mar 15:00:00 - [info] [pushover api:Pushover] pushover POST succeeded:
"{"status":1,"request":"1def1905-179e-46f1-beeb-84a23cfe79dc"}"
8 Mar 15:00:06 - [info] nora: - synced 36 device(s)
8 Mar 16:37:24 - [info] [pushover api:Pushover] pushover POST succeeded:
"{"status":1,"request":"c11f6c30-ceae-4d3b-94ca-2154ab028cc9"}"
8 Mar 16:48:51 - [info] [pushover api:Pushover] pushover POST succeeded:
"{"status":1,"request":"3f992f99-a31f-4915-9aae-03a1dafe3885"}"
8 Mar 17:27:27 - [info] [pushover api:Pushover] pushover POST succeeded:
"{"status":1,"request":"a5abea1c-9e19-4120-b012-6d689a79defb"}"
<--- Last few GCs --->
[5876:0x5518dc8] 12583299 ms: Scavenge 225.0 (256.9) -> 223.4 (256.9) MB, 3.0 / 0.0 ms (average mu = 0.993, current mu = 0.993) allocation failure
[5876:0x5518dc8] 12584254 ms: Scavenge 225.1 (256.9) -> 223.5 (256.9) MB, 6.0 / 0.0 ms (average mu = 0.993, current mu = 0.993) allocation failure
[5876:0x5518dc8] 12584298 ms: Scavenge 225.1 (256.9) -> 223.6 (256.9) MB, 2.1 / 0.0 ms (average mu = 0.993, current mu = 0.993) task
<--- JS stacktrace --->
FATAL ERROR: MarkCompactCollector: young object promotion failed Allocation failed - JavaScript heap out of memory
nodered.service: Main process exited, code=killed, status=6/ABRT
nodered.service: Failed with result 'signal'.
nodered.service: Consumed 53min 57.762s CPU time.
nodered.service: Scheduled restart job, restart counter is at 12.
Stopped Node-RED graphical event wiring tool.
nodered.service: Consumed 53min 57.762s CPU time.
Started Node-RED graphical event wiring tool.
8 Mar 18:29:52 - [info]
Willkommen bei Node-RED!

... and then a regular startup without any abnormalities.

I think there are two options:
1st -> i need a Pi 4 with 4GB Ram
2nd -> i send the values to Google Tabs and create a chart to check the last days (also more then 3).

For the next days i stoped the charts, cause I'm on a buisness trip.

I read something about --max-old-space-size can i do something with this information in my case?

Colin · 9 March 2022 09:20

Did you use the recommended script to install node-red?

Please show us the section of the startup log that you snipped as the "regular startup" log.

Also please post the file /lib/systemd/system/nodered.service

Flixm · 11 March 2022 16:34

... I think I did it correct. I did it as written in the description.

3 days ago I change the timing to every 15 minutes.

Since then no crash

Tassidar · 22 March 2022 13:58

Hi Everyone,

this is my first post. I've been using Node Red for a while and I'm quite happy with it.

Since it was lots of fun and no real problems, I didn't need to ask for help... until now

Today I accidentally started an infinite recursion of an external Python-Script. (via exec-Node...)

Well, infinite was a little too much for my poor ol' Pi4 with just 2 GB, and so I crashed Node-Red.

I figured, that when I move the Script somewhere else and restart my PI, that everything would come back to life, but no.

Do you have any Idea how to revive it?

Here is a snippet of log, that node-red gives me, when I run it manually (did do a proper "sudo service nodered stop" before...

Welcome to Node-RED
===================

22 Mar 14:41:19 - [info] Node-RED version: v2.2.2
22 Mar 14:41:19 - [info] Node.js  version: v14.19.1
22 Mar 14:41:19 - [info] Linux 5.10.103-v8+ arm64 LE
22 Mar 14:41:20 - [info] Loading palette nodes
22 Mar 14:41:21 - [info] Dashboard version 3.1.6 started at /ui
22 Mar 14:41:22 - [info] Settings file  : /home/pi/.node-red/settings.js
22 Mar 14:41:22 - [info] Context store  : 'default' [module=memory]
22 Mar 14:41:22 - [info] User directory : /home/pi/.node-red
22 Mar 14:41:22 - [info] Projects directory: /home/pi/.node-red/projects
22 Mar 14:41:22 - [info] Server now running at http://127.0.0.1:1880/
22 Mar 14:41:22 - [info] Active project : Sonne
22 Mar 14:41:22 - [info] Flows file     : /home/pi/.node-red/projects/Sonne/flow.json
22 Mar 14:41:22 - [info] Starting flows
22 Mar 14:41:22 - [info] Started flows
22 Mar 14:41:22 - [info] [mqtt-broker:9ccb1d4ebcf2234b] Connection failed to broker: mqtt://localhost:1883

<--- Last few GCs --->

[1180:0x17438910]    47202 ms: Scavenge (reduce) 922.8 (928.2) -> 922.2 (929.0) MB, 4.9 / 0.0 ms  (average mu = 0.213, current mu = 0.153) allocation failure 
[1180:0x17438910]    52460 ms: Mark-sweep (reduce) 923.0 (926.0) -> 922.1 (926.7) MB, 5249.6 / 0.1 ms  (+ 2.2 ms in 265 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 5507 ms) (average mu = 0.123, current mu = 0.072) al

<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0xa2f1a8 node::Abort() [node-red]
 2: 0x975a58 node::FatalError(char const*, char const*) [node-red]
 3: 0xb947e0 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node-red]
 4: 0xb9496c v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node-red]
 5: 0xd3d088 v8::internal::Heap::EnsureFromSpaceIsCommitted() [node-red]
 6: 0xd3daf0  [node-red]
 7: 0xd4b7b4 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node-red]
 8: 0xd4ec84 v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node-red]
 9: 0xd20f74 v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType, v8::internal::AllocationOrigin) [node-red]
10: 0x103c1ec v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [node-red]
11: 0x139e98c  [node-red]
Aborted

Thanks alot.

All the best

Thorsten

Colin · 22 March 2022 14:20

Welcome to the forum @Tassidar

Generally it is best to start a new thread rather than replying to an old one, unless you believe that the cause is the same as the previous thread. I will ask some questions, if you could start a new thread, and provide the answers to those questions, as well as the information you posted, then that would be best.

Does it fail very soon after the mqtt broker message or is it some time later?

Was it working ok and then failed? If so then did you change anything?

Should it connect to the mqtt broker ok?

dceejay · 22 March 2022 17:43

you can possibly start node-red with the --safe command line option to start but not run. You can then tidy up whatever is causing the issue and deploy.

Tassidar · 22 March 2022 18:30

Thank you. Yes, that did solve my problem.

Happy to be up-and-running again

system · 5 April 2022 18:31

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Node-red-node-mysql 0.3.0 is causing a memory leak General database	17	1260	4 December 2021
Out of memory crash on Windows 10 PC General	25	3987	24 December 2021
Node-red-node-google - node-red crash General	4	1169	18 April 2019
[SOLVED FOR NOW] Out Of Memory exception interrupts my Flows General	5	548	28 August 2022
Node-red restarts General	7	728	14 November 2021

Node Red crashed - why?

Related Topics