Node-red crashes after a while and no html view -- please help -- SOLVED

#1

I am using Node-red for some time, recently on Linux Mint.
A few days ago it turned out that, while node-red was running and I did a deploy, rode-ned went into a kind of loop (deploy never finished), and didn't respond anymore.
It turned out that free disk space of the home directory was running low and eventually zero.
After allocating more space and restarting bode-red again, I got the same problem again: 1) browser page was empty/not fed by node-red anymore and the background job crashed again after a while.
The messages I got in the terminal were more or less the same as during the first crash. I tried again several times, but no change, same symptoms and messages in the terminal.
The last statement was that node-red aborted and a dump was made, but I don'd know where it was saved or what the file name is. Probably the dump could not be saved as the disk space was 0.

Here is a copy of the background activities/messages until the 1st crash.

================= start of terminal output ============================

Welcome to Node-RED

5 Jan 20:51:32 - [info] Node-RED version: v0.19.5
5 Jan 20:51:32 - [info] Node.js version: v8.14.0
5 Jan 20:51:32 - [info] Linux 4.15.0-43-generic x64 LE
5 Jan 20:51:32 - [info] Loading palette nodes
5 Jan 20:51:33 - [warn] rpi-gpio : Raspberry Pi specific node set inactive
5 Jan 20:51:33 - [warn] rpi-gpio : Cannot find Pi RPi.GPIO python library
5 Jan 20:51:34 - [info] Dashboard version 2.11.0 started at /ui
5 Jan 20:51:34 - [info] Settings file : /home/leo/.node-red/settings.js
5 Jan 20:51:34 - [info] Context store : 'default' [module=memory]
5 Jan 20:51:34 - [info] User directory : /home/leo/.node-red
5 Jan 20:51:34 - [info] Server now running at http://127.0.0.1:1880/
5 Jan 20:51:34 - [info] Active project : GateManager
5 Jan 20:51:34 - [info] Flows file : /home/leo/.node-red/projects/GateManager/flows_LT03X.json
5 Jan 20:51:34 - [info] Starting flows
5 Jan 20:51:34 - [info] Started flows

<--- Last few GCs --->

[2289:0x2cd06f0] 520160 ms: Mark-sweep 1350.5 (1413.9) -> 1350.5 (1413.9) MB, 1586.8 / 0.0 ms allocation failure GC in old space requested
[2289:0x2cd06f0] 522120 ms: Mark-sweep 1350.5 (1413.9) -> 1350.4 (1381.9) MB, 1960.5 / 0.0 ms last resort GC in old space requested
[2289:0x2cd06f0] 523365 ms: Mark-sweep 1350.4 (1381.9) -> 1350.4 (1381.9) MB, 1244.1 / 0.0 ms last resort GC in old space requested

<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0xebb0ffa58b9
1: /* anonymous */ [/usr/lib/node_modules/node-red/nodes/core/core/89-delay.js:~92] [pc=0xd1ea23d13a3](this=0x3a11873ccf09 ,msg=0x33145dba8d99 )
2: emitOne(aka emitOne) [events.js:~114] [pc=0xd1ea23db518](this=0x30e1abe022d1 ,handler=0x21925adff101 <JSFunction (sfi = 0x27e97d7a7831)>,isFn=0x30e1abe02371 ,self=0x3a...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
1: node::Abort() [node-red]
2: 0x8cce9c [node-red]
3: v8::Utils::ReportOOMFailure(char const*, bool) [node-red]
4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [node-red]
5: v8::internal::factory::NewUninitializedFixedArray(int) [node-red]
6: 0xd817a3 [node-red]
7: v8::internal::Runtime_GrowArrayElements(int, v8::internal::Object**, v8::internal::Isolate*) [node-red]
8: 0xd1ea22042fd
Aborted (core dumped)

==== end of terminal output ===============================

I am not a json / linux / node-red specialist and seeking desperately for help.
All suggestions/help are appreciated.

0 Likes

#2

first thing, lets look at your disk space.

Open a terminal window and type df -h the first two lines will look loki

Filesystem      Size  Used Avail Use% Mounted on
/dev/root        15G  5.9G  7.8G  43% /

please show the /dev/root line

next you say

It turned out that free disk space of the home directory was running low and eventually zero.
After allocating more space and restarting bode-red again,

How did you allocate more space?

0 Likes

#3

Hi zenofmud,

I am new on this forum and not clear if I should answer your email directly, or here on the forum. Anyway I do the latter, as it might help other people as well.

My system is Linux Mint 19 (Ubuntu). I don't get the /dev/root line with df -h. My linux file system is on /dev/sda5 and I get the following details:

Filesystem Size Used Avail Use% Mounted on
/dev/sda5 67G 63G 914M 99% /

I managed to free up some space by deleting some big files in /home which I don't need anymore.
Strangely after the delete free space was 900+ MB, then it went down next day ultimately to 375 MB and now another day it's 914MB again.
What I am planning now is to shrink the main Windows partition (80 GB) (using gparted) on the system and allocate it for a new Mint /home drive. Currently the /home drive is 25 GB, so in total I will get 105GB.
However I need some time to figure out how to do this. A simple copy or rsync terminal command failed due to files for which I have no permission to copy.

I am still wondering how I can restore the probably corrupted node-red/json files.
I can think of two options:

  1. restore the system to the status before the crash (using timeshift, which I have never done before), or
  2. use the backup json file

In the projects/GateManager directory I see (amonst others) the following files:

-rw-r--r-- 1 leo leo 34140 Jan 5 20:39 flows_LT03X.json
-rw-r--r-- 1 leo leo 34105 Jan 5 20:38 .flows_LT03X.json.backup

The time difference between the 2 flow files is only 1 second, so not clear if it makes sense.

Any suggestion or another approach?

0 Likes

#4

The flows you located are the last two you have. When you do a deploy a backup copy is made.

What are you doing in your flow? Is it possible you are doing something that ie eating up the avaliable memory?

0 Likes

#5

My flows are not very special and I don't see why they are eating up memory. During the design process I had sometimes an endless loop of 1 second repeating messages. I could break the loop only by cutting a link between 2 nodes and redeploy. However that was not the case at my last deploy, when it crashed.

Actually my flow is simulating a gate manager (opening/closing) for our gates, to replace the old electronics and giving more flexibility in configuring the dynamics of the open/close process.
If the simulator is working well, I will deploy the flow on a Pi, connect some relays and connect it as well to a sonoff switch for remote control.

0 Likes

#6

It took a couple of days, but I managed to solve the problem that occurred on Jan 5, though not clear what exactly the problem was (and while writing this update, discovered that the problem was not really solved).

The cause of the problem was likely one of the following or maybe both:

  1. Free disk space on my disk where node-red is installed went down to almost zero;
  2. Deploy problem due to an invalid/wrong/broken/etc node.

Today I got the UI flow editor up again. Since I am using the project feature, I can now look back at all changes made so far. However, I didn't see any date/time stamps in the project history database and hence I have no idea what the last changes were before node-red crashed on Jan 4.
Can anyone point me to documentation for using the project history database?
Unfortunately I ran today into a similar problem as before. After hitting the deploy key it grayed out and showed a wave indicating busy as on Jan 4.

Anyway, what I did until the moment node-red came up again today, was the following:
(BTW I regularly tested after a (set of) changes if node-red was working again.)

  1. Creating plenty of available free disk space, it took me more than a week, because I wanted to make some other configuration changes as well and implementing another backup method. However this didn't bring up node-red again. Starting Jan 12 I decided to have a more detailed look at node-red and did some tests.

  2. Copied .node-red folder to save it in case its getting more worse and data will be damaged/destroyed.

  3. Imported the project flow in my old node-red system (v0.18.4) on Windows10, which doesn't support projects. I was happily surprised that I could import the flows, as it showed that the flows are correctly configured and I have now a working copy of the flows.

  4. Checked the versions of my current node-red system on Linux:
    0 linux: 4.15.0-43
    0 node: v8.15.0
    0 npm: v6.4.1
    0 node-red: v0.19.5
    0 dashboard: v2.11.0

  5. Upgrading node-red, as follows:


sudo npm install -g --unsafe-perm node-red
[sudo] password for leo:
npm WARN deprecated mailparser@0.6.2: Mailparser versions older than v2.3.0 are deprecated
npm WARN deprecated nodemailer@1.11.0: All versions below 4.0.1 of Nodemailer are deprecated. See https://nodemailer.com/status/
npm WARN deprecated mimelib@0.3.1: This project is unmaintained
npm WARN deprecated mailcomposer@2.1.0: This project is unmaintained
npm WARN deprecated buildmail@2.0.0: This project is unmaintained
/usr/bin/node-red-pi -> /usr/lib/node_modules/node-red/bin/node-red-pi
/usr/bin/node-red -> /usr/lib/node_modules/node-red/red.js

  • node-red@0.19.5
    added 1 package from 1 contributor, removed 3 packages and updated 10 packages in 24.716s

node-red UI Still not working and finally crashes.

  1. Checked for 'outdated' nodes and upgraded them (3 nodes)

  2. Cleaning NPM cache, as follows:


sudo npm cache clean --force
[sudo] password for leo:
npm WARN using --force I sure hope you know what you are doing.


  1. NPM rebuild. Successful, 130 lines of output.

  2. Tried node-red start parameters, such as defining port number, node-red directory, etc. Nothing helped.

  3. NPM doctor. No problem reported, however suggesting higher versions of npm and node.

  4. Deleting nodes, that I am not using anymore, 8 in total.

At this point, the node-red editor came up again, however not all problems solved as turned out later. One node was still used (dead-man-switch), so I tried to reinstall, but it didn't work and decided to delete that node. Anyway after a while node-red crashed again, but the crash log differs from the ones before. See below copy.


node-red start
13 Jan 12:15:13 - [info]

Welcome to Node-RED

13 Jan 12:15:13 - [info] Node-RED version: v0.19.5
13 Jan 12:15:13 - [info] Node.js version: v8.15.0
13 Jan 12:15:13 - [info] Linux 4.15.0-43-generic x64 LE
13 Jan 12:15:13 - [info] Loading palette nodes
13 Jan 12:15:13 - [warn] rpi-gpio : Raspberry Pi specific node set inactive
13 Jan 12:15:13 - [warn] rpi-gpio : Cannot find Pi RPi.GPIO python library
13 Jan 12:15:14 - [info] Dashboard version 2.11.0 started at /ui
13 Jan 12:15:14 - [warn] Missing node modules:
13 Jan 12:15:14 - [warn] - node-red-contrib-bigtimer (2.0.8): bigtimer
13 Jan 12:15:14 - [warn] - node-red-contrib-blindcontroller (4.6.0): blindcontroller, multiblindcontroller
13 Jan 12:15:14 - [warn] - node-red-contrib-internetattv (0.1.6): internetAtTv
13 Jan 12:15:14 - [warn] - node-red-contrib-newsamsungtv (0.1.0): newSamsungTV, new-samsung-tv-isalive
13 Jan 12:15:14 - [warn] - node-red-contrib-deadmanswitch (1.0.1): dead-man-switch
13 Jan 12:15:14 - [warn] - node-red-contrib-message-counter (1.0.1): Message Counter
13 Jan 12:15:14 - [info] Removing modules from config
13 Jan 12:15:14 - [info] Settings file : /home/leo/.node-red/settings.js
13 Jan 12:15:14 - [info] Context store : 'default' [module=memory]
13 Jan 12:15:14 - [info] User directory : /home/leo/.node-red
13 Jan 12:15:14 - [info] Server now running at http://127.0.0.1:1880/
13 Jan 12:15:14 - [info] Active project : GateManager
13 Jan 12:15:14 - [info] Flows file : /home/leo/.node-red/projects/GateManager/flows_LT03X.json
13 Jan 12:15:14 - [info] Waiting for missing types to be registered:
13 Jan 12:15:14 - [info] - dead-man-switch
13 Jan 14:26:03 - [info] Stopping modified flows
13 Jan 14:26:03 - [info] Stopped modified flows
13 Jan 14:26:03 - [info] Waiting for missing types to be registered:
13 Jan 14:26:03 - [info] - dead-man-switch
13 Jan 14:27:20 - [info] Installing module: node-red-contrib-deadmanswitch, version: 1.0.1
13 Jan 14:27:26 - [info] Installed module: node-red-contrib-deadmanswitch
13 Jan 14:27:26 - [info] Missing type registered: dead-man-switch
13 Jan 14:27:26 - [info] Starting flows
13 Jan 14:27:26 - [info] Started flows

<--- Last few GCs --->

[2264:0x2ac2700] 8497851 ms: Mark-sweep 1446.5 (1507.2) -> 1446.5 (1476.2) MB, 948.0 / 0.0 ms (+ 0.0 ms in 0 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 948 ms) last resort GC in old space requested
[2264:0x2ac2700] 8499037 ms: Mark-sweep 1446.5 (1476.2) -> 1446.5 (1476.2) MB, 1185.6 / 0.0 ms last resort GC in old space requested

<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x33a2f4f258b9
0: builtin exit frame: stringify(this=0x33a2f4f18371 ,0x33a2f4f7c651 <String[1]: >,0x2f965abfef01 <JSFunction (sfi = 0x7c840df6c21)>,0x2f965abfeca1 )

1: encodeObject [/usr/lib/node_modules/node-red/red/runtime/util.js:~463] [pc=0x71b5db64018](this=0x1139b4173429 <Object map = 0x1ad59f1fcd81>,msg=0x213e1a5d8f09 <Object map = 0x37e...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
1: node::Abort() [node-red]
2: 0x8ccf9c [node-red]
3: v8::Utils::ReportOOMFailure(char const*, bool) [node-red]
4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [node-red]
5: v8::internal::factory::NewTransitionArray(int) [node-red]
6: v8::internal::TransitionsAccessor::Insert(v8::internal::Handlev8::internal::Name, v8::internal::Handlev8::internal::Map, v8::internal::SimpleTransitionFlag) [node-red]
7: v8::internal::map::ConnectTransition(v8::internal::Handlev8::internal::Map, v8::internal::Handlev8::internal::Map, v8::internal::Handlev8::internal::Name, v8::internal::SimpleTransitionFlag) [node-red]
8: v8::internal::map::CopyReplaceDescriptors(v8::internal::Handlev8::internal::Map, v8::internal::Handlev8::internal::DescriptorArray, v8::internal::Handlev8::internal::LayoutDescriptor, v8::internal::TransitionFlag, v8::internal::MaybeHandlev8::internal::Name, char const*, v8::internal::SimpleTransitionFlag) [node-red]
9: v8::internal::map::CopyAddDescriptor(v8::internal::Handlev8::internal::Map, v8::internal::Descriptor*, v8::internal::TransitionFlag) [node-red]
10: v8::internal::map::CopyWithField(v8::internal::Handlev8::internal::Map, v8::internal::Handlev8::internal::Name, v8::internal::Handlev8::internal::FieldType, v8::internal::PropertyAttributes, v8::internal::PropertyConstness, v8::internal::Representation, v8::internal::TransitionFlag) [node-red]
11: v8::internal::map::TransitionToDataProperty(v8::internal::Handlev8::internal::Map, v8::internal::Handlev8::internal::Name, v8::internal::Handlev8::internal::Object, v8::internal::PropertyAttributes, v8::internal::PropertyConstness, v8::internal::Object::StoreFromKeyed, bool*) [node-red]
12: v8::internal::LookupIterator::PrepareTransitionToDataProperty(v8::internal::Handlev8::internal::JSObject, v8::internal::Handlev8::internal::Object, v8::internal::PropertyAttributes, v8::internal::Object::StoreFromKeyed) [node-red]
13: v8::internal::Object::AddDataProperty(v8::internal::LookupIterator*, v8::internal::Handlev8::internal::Object, v8::internal::PropertyAttributes, v8::internal::Object::ShouldThrow, v8::internal::Object::StoreFromKeyed) [node-red]
14: v8::internal::JSObject::AddProperty(v8::internal::Handlev8::internal::JSObject, v8::internal::Handlev8::internal::Name, v8::internal::Handlev8::internal::Object, v8::internal::PropertyAttributes) [node-red]
15: v8::internal::JsonStringifier::ApplyReplacerFunction(v8::internal::Handlev8::internal::Object, v8::internal::Handlev8::internal::Object, v8::internal::Handlev8::internal::Object) [node-red]
16: v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_(v8::internal::Handlev8::internal::Object, bool, v8::internal::Handlev8::internal::Object) [node-red]
17: v8::internal::JsonStringifier::Stringify(v8::internal::Handlev8::internal::Object, v8::internal::Handlev8::internal::Object, v8::internal::Handlev8::internal::Object) [node-red]
18: v8::internal::Builtin_JsonStringify(int, v8::internal::Object**, v8::internal::Isolate*) [node-red]
19: 0x71b5d80697d
Aborted (core dumped)


So I am back at where I started on Jan 5.

What's new is the following message:
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory

What can be done to increase heap size or any other measure?

Anyone understands what's going on, how to solve this problem, or can tell me if it makes sense to remove all node-red and npm stuff and install from scratch?
Even removing all stuff seems not to be an easy job, I read in some posts!

Thanks, Leo

0 Likes

#7

Will Node-RED start with a blank flow ? eg - node-red myblankflow.json
Again the dump seems to indicate it's running out of memory... so something is chewing that up - maybe there is a loop in your flow - or yet another bad node.

0 Likes

#8

Can I just point out that that is not the correct command to start node red unless you want to run it with a flow file called start. I presume this works as it doesn't find that file and drops back to the default. Assuming you installed node-red on the pi using the recommended method then you can use node-red-stop and node-red-start.
It seems you still have the problem you had at the start, that something is using up all the memory.
Do you have any nodes that queue up messages, or are you using the Delay node in Rate Limit mode? If so then these can consume memory if messages are coming in at a greater rate than they are going out.
I see that the dead-man-switch node has not been updated in two years, which is not a good sign. From the limited information in the readme I suspect that you could use a Trigger node instead.
Another possibility is a loop, such as an MQTT loop. If you use MQTT put debug nodes showing what you are sending to MQTT and check that it is not sending thousands of messages.

0 Likes

#9

Looking at the code of dead-man-switch it don't think it is likely to be the cause of your problems, but as I thought the Trigger node will do exactly the same function for you.

0 Likes

#10

Thanks a lot, I believe we are coming close to the solution and cause.

By the way I am running this node-red on a linux Mint system, not on a pi.

As dceejay suggested, I tried to run "node-red myblankflow.json". This didn't work probably because I had enabled project mode, and in this mode I guess the flow parameter is discarded. So I set the project mode to disabled state in the settings.js file and now I get the flow editor with a blank flow, and could easily enter some dashboard nodes and got the dashboard page as well.

It might be that, as suggested, that there is a loop somewhere in my flow that is eating up storage.
So I will build up my flow again step by step and try to find out where and when it goes wrong.

I keep this issue open until I have some more proof of what happened and will report my findings.

0 Likes

#11

you should no be able to import your old flow - use and editor to copy it to the clipboard - then menu - import - from clipboard . but before you hit deploy... go and investigate / remove / disconnect anything you may think will have caused the issue... then deploy - test .. repeat as necessary :slight_smile:

0 Likes

#12

This may be a long-shot, but if this log line is referring to the JSON.stringify() function, could it be trying to serialize a JS object with a recursive object property, something that eventually includes itself?

1 Like

#13

A week ago I reported on this forum that my problem was more or less solved.
It still is working okay, and I am quite busy finalizing my "gate manager" project. That's why I didn't give you feedback on the reported issue the last couple of days.
It took me a while to reconfigure and recover my flows. Basically only one node function that I wrote before is still missing and that was probably the cause of the problems, so I couldn't replay the problem. Hence my first thought: "disk space shortage", was not really the cause of the problem, but probably the function node I wrote perhaps combined with a flow design/implementation that was not very handy and resulting in some kind of a loop while deploying the flow.

Meanwhile I have re-engineered the missing node function, this time by using standard nodes in a subflow, which is working quite well.

I would like to close this issue now, but not sure if required and if so how to do.

Thanks for your time and support, Leo

0 Likes