I have found myself in an odd situation. Node-red crashes after starting up every minute. The only change I made was copy/paste all nodes from one flow to a new flow and deploy. I was able to delete the flow<>.json file and the server comes up fine. I moved all json data from a backup (2 days old) and restart the server and everything comes up fine. when I tried to C/P again and deploy the node-red crashes. In the end, I deleted some unused flows then tried C/P and everything is working. No issues with disk space, and plenty of CPU and Mem. Is there any setting in settings.json that I need to look at? Flow_<>.json file is about 2.3MB (yes I have very large flows) Also, it worked fine if using node-red start (with all my flow data in startup.json).
What is in there Node-RED log when it crashes?
I was not able to find the log file 
looked everywhere for it. I am running on centos.
How are you starting Node-RED? And how did you install it?
Try running node-red-log
Node-Red starts with the OS from this install:
Logs don't go back that far.
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
Stop node red, then start it again using
node-red-start
Post the full log from startup to the crash. When posting the log use this technique: How to share code or flow json
27 Oct 09:48:03 - [info] [cronplus:First Day of the Month] createTask - index: 0, static: true, opt: {"name":"schedule1","topic":"schedule1","payloadType":"default","payload":"","expressionType":"cron","expression":"0 0 12 1 * *","location":"","offset":"0","solarType":"all","solarEvents":"sunrise,sunset"}
27 Oct 09:48:03 - [info] Started flows
<--- Last few GCs --->
[59627:0x523b7b0] 11130 ms: Mark-sweep 485.0 (521.0) -> 484.8 (521.0) MB, 484.6 / 0.1 ms (average mu = 0.149, current mu = 0.116) deserialize GC in old space requested
[59627:0x523b7b0] 11623 ms: Mark-sweep 485.0 (521.0) -> 484.9 (521.0) MB, 486.6 / 0.1 ms (average mu = 0.080, current mu = 0.014) deserialize GC in old space requested
<--- JS stacktrace --->
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
1: 0xa389b0 node::Abort() [node-red]
2: 0x96e0af node::FatalError(char const*, char const*) [node-red]
3: 0xbb7a4e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node-red]
4: 0xbb7dc7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node-red]
5: 0xd73fd5 [node-red]
6: 0xd74b5f [node-red]
7: 0xd8299b v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node-red]
8: 0xd85e1f v8::internal::Heap::ReserveSpace(std::vector<v8::internal::Heap::Chunk, std::allocator<v8::internal::Heap::Chunk> >*, std::vector<unsigned long, std::allocator<unsigned long> >*) [node-red]
9: 0x110a9e2 v8::internal::DeserializerAllocator::ReserveSpace() [node-red]
10: 0x11084e6 v8::internal::ContextDeserializer::Deserialize(v8::internal::Isolate*, v8::internal::Handle<v8::internal::JSGlobalProxy>, v8::DeserializeInternalFieldsCallback) [node-red]
11: 0x1108d84 v8::internal::ContextDeserializer::DeserializeContext(v8::internal::Isolate*, v8::internal::SnapshotData const*, bool, v8::internal::Handle<v8::internal::JSGlobalProxy>, v8::DeserializeInternalFieldsCallback) [node-red]
12: 0x112f17a v8::internal::Snapshot::NewContextFromSnapshot(v8::internal::Isolate*, v8::internal::Handle<v8::internal::JSGlobalProxy>, unsigned long, v8::DeserializeInternalFieldsCallback) [node-red]
13: 0xe30388 v8::internal::Genesis::Genesis(v8::internal::Isolate*, v8::internal::MaybeHandle<v8::internal::JSGlobalProxy>, v8::Local<v8::ObjectTemplate>, unsigned long, v8::DeserializeInternalFieldsCallback, v8::MicrotaskQueue*) [node-red]
14: 0xe30738 v8::internal::Bootstrapper::CreateEnvironment(v8::internal::MaybeHandle<v8::internal::JSGlobalProxy>, v8::Local<v8::ObjectTemplate>, v8::ExtensionConfiguration*, unsigned long, v8::DeserializeInternalFieldsCallback, v8::MicrotaskQueue*) [node-red]
15: 0xbccf42 v8::NewContext(v8::Isolate*, v8::ExtensionConfiguration*, v8::MaybeLocal<v8::ObjectTemplate>, v8::MaybeLocal<v8::Value>, unsigned long, v8::DeserializeInternalFieldsCallback, v8::MicrotaskQueue*) [node-red]
16: 0xbcd694 v8::Context::New(v8::Isolate*, v8::ExtensionConfiguration*, v8::MaybeLocal<v8::ObjectTemplate>, v8::MaybeLocal<v8::Value>, v8::DeserializeInternalFieldsCallback, v8::MicrotaskQueue*) [node-red]
17: 0xa2953e node::contextify::ContextifyContext::CreateV8Context(node::Environment*, v8::Local<v8::Object>, node::contextify::ContextOptions const&) [node-red]
18: 0xa29b45 node::contextify::ContextifyContext::ContextifyContext(node::Environment*, v8::Local<v8::Object>, node::contextify::ContextOptions const&) [node-red]
19: 0xa29f26 node::contextify::ContextifyContext::MakeContext(v8::FunctionCallbackInfo<v8::Value> const&) [node-red]
20: 0x13ddced [node-red]
Process 59627 (node-red) of user 0 dumped core.
nodered.service: Main process exited, code=dumped, status=6/ABRT
nodered.service: Failed with result 'core-dump'.
top - 10:04:32 up 1 min, 1 user, load average: 1.23, 0.52, 0.19
Tasks: 331 total, 3 running, 328 sleeping, 0 stopped, 0 zombie
%Cpu(s): 4.7 us, 0.6 sy, 0.0 ni, 94.6 id, 0.0 wa, 0.1 hi, 0.0 si, 0.0 st
MiB Mem : 64150.2 total, 63286.3 free, 502.5 used, 361.3 buff/cache
MiB Swap: 8100.0 total, 8100.0 free, 0.0 used. 63041.5 avail Mem
As I asked, please post the full startup log, from the start.
Is this better?
Stopped Node-RED graphical event wiring tool.
Started Node-RED graphical event wiring tool.
27 Oct 10:19:30 - [info]
Welcome to Node-RED
===================
27 Oct 10:19:30 - [info] Node-RED version: v2.1.3
27 Oct 10:19:30 - [info] Node.js version: v14.18.1
27 Oct 10:19:30 - [info] Linux 4.18.0-348.el8.x86_64 x64 LE
27 Oct 10:19:30 - [info] Loading palette nodes
27 Oct 10:19:31 - [info] Dashboard version 3.1.0 started at /ui
27 Oct 10:19:32 - [info] Settings file : /root/.node-red/settings.js
27 Oct 10:19:32 - [info] Context store : 'default' [module=memory]
27 Oct 10:19:32 - [info] User directory : /root/.node-red
27 Oct 10:19:32 - [warn] Projects disabled : editorTheme.projects.enabled=false
27 Oct 10:19:32 - [info] Flows file : /root/.node-red/flows_DFW-01.json
27 Oct 10:19:32 - [info] Server now running at http://127.0.0.1:1880/
27 Oct 10:19:32 - [warn]
---------------------------------------------------------------------
Your flow credentials file is encrypted using a system-generated key.
If the system-generated key is lost for any reason, your credentials
file will not be recoverable, you will have to delete it and re-enter
your credentials.
You should set your own key using the 'credentialSecret' option in
your settings file. Node-RED will then re-encrypt your credentials
file using your chosen key the next time you deploy a change.
---------------------------------------------------------------------
27 Oct 10:19:32 - [info] Starting flows
27 Oct 10:19:34 - [info] [cronplus:First Day of the Month] createTask - index: 0, static: true, opt: {"name":"schedule1","topic":"schedule1","payloadType":"default","payload":"","expressionType":"cron","expression":"0 0 12 1 * *","location":"","offset":"0","solarType":"all","solarEvents":"sunrise,sunset"}
27 Oct 10:19:34 - [info] Started flows
<--- Last few GCs --->
[3258:0x4e187b0] 10729 ms: Mark-sweep 486.0 (529.3) -> 485.7 (529.5) MB, 486.5 / 0.1 ms (average mu = 0.113, current mu = 0.015) deserialize GC in old space requested
[3258:0x4e187b0] 11225 ms: Mark-sweep 486.0 (529.5) -> 485.9 (529.8) MB, 488.6 / 0.1 ms (average mu = 0.065, current mu = 0.015) deserialize GC in old space requested
<--- JS stacktrace --->
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
1: 0xa389b0 node::Abort() [node-red]
2: 0x96e0af node::FatalError(char const*, char const*) [node-red]
3: 0xbb7a4e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node-red]
4: 0xbb7dc7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node-red]
5: 0xd73fd5 [node-red]
6: 0xd74b5f [node-red]
7: 0xd8299b v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node-red]
8: 0xd85de7 v8::internal::Heap::ReserveSpace(std::vector<v8::internal::Heap::Chunk, std::allocator<v8::internal::Heap::Chunk> >*, std::vector<unsigned long, std::allocator<unsigned long> >*) [node-red]
9: 0x110a9e2 v8::internal::DeserializerAllocator::ReserveSpace() [node-red]
10: 0x11084e6 v8::internal::ContextDeserializer::Deserialize(v8::internal::Isolate*, v8::internal::Handle<v8::internal::JSGlobalProxy>, v8::DeserializeInternalFieldsCallback) [node-red]
11: 0x1108d84 v8::internal::ContextDeserializer::DeserializeContext(v8::internal::Isolate*, v8::internal::SnapshotData const*, bool, v8::internal::Handle<v8::internal::JSGlobalProxy>, v8::DeserializeInternalFieldsCallback) [node-red]
12: 0x112f17a v8::internal::Snapshot::NewContextFromSnapshot(v8::internal::Isolate*, v8::internal::Handle<v8::internal::JSGlobalProxy>, unsigned long, v8::DeserializeInternalFieldsCallback) [node-red]
13: 0xe30388 v8::internal::Genesis::Genesis(v8::internal::Isolate*, v8::internal::MaybeHandle<v8::internal::JSGlobalProxy>, v8::Local<v8::ObjectTemplate>, unsigned long, v8::DeserializeInternalFieldsCallback, v8::MicrotaskQueue*) [node-red]
14: 0xe30738 v8::internal::Bootstrapper::CreateEnvironment(v8::internal::MaybeHandle<v8::internal::JSGlobalProxy>, v8::Local<v8::ObjectTemplate>, v8::ExtensionConfiguration*, unsigned long, v8::DeserializeInternalFieldsCallback, v8::MicrotaskQueue*) [node-red]
15: 0xbccf42 v8::NewContext(v8::Isolate*, v8::ExtensionConfiguration*, v8::MaybeLocal<v8::ObjectTemplate>, v8::MaybeLocal<v8::Value>, unsigned long, v8::DeserializeInternalFieldsCallback, v8::MicrotaskQueue*) [node-red]
16: 0xbcd694 v8::Context::New(v8::Isolate*, v8::ExtensionConfiguration*, v8::MaybeLocal<v8::ObjectTemplate>, v8::MaybeLocal<v8::Value>, v8::DeserializeInternalFieldsCallback, v8::MicrotaskQueue*) [node-red]
17: 0xa2953e node::contextify::ContextifyContext::CreateV8Context(node::Environment*, v8::Local<v8::Object>, node::contextify::ContextOptions const&) [node-red]
18: 0xa29b45 node::contextify::ContextifyContext::ContextifyContext(node::Environment*, v8::Local<v8::Object>, node::contextify::ContextOptions const&) [node-red]
19: 0xa29f26 node::contextify::ContextifyContext::MakeContext(v8::FunctionCallbackInfo<v8::Value> const&) [node-red]
20: 0x13ddced [node-red]
Process 3258 (node-red) of user 0 dumped core.
nodered.service: Main process exited, code=dumped, status=6/ABRT
nodered.service: Failed with result 'core-dump'.
OK, so nodejs is running out of memory. Are you doing something with large buffers or large files, images, video etc?
During the minute before it crashes is node red using a lot of CPU? If so then likely you have some sort of loop somewhere, which is allocating memory and never releasing it.
No, I don't use large files, videos, or large buffers. Each flow does check an email account every 10 minutes for new emails, but the emails are HTML/text only. CPU does go crazy right before crashing. What I don't understand is why it starts fine if I remove one or two of the flows.
All,
I was able to start Node-Red with this:
node-red-start --node-args="--max-old-space-size=3072" -- -v
Now, my next question is how to make this permanent when the server reboots and auto-starts Node-Red.
Probably you have a loop somewhere that gets triggered occasionally. It must be to do with the flows you are adding. Add some debug about and watch for it anything strange.
I suspect that you are just hiding the issue temporarily and that it will recur.
You can edit the startup in the systemd script. this command will show you where it is (at least it does on Debian systems, I don't know whether yours is the same)
systemctl status nodered
I am back in business! Thanks all!!
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.
