Diagnosing Node-RED crashes

While tinkering with Node-RED, I ended up in a situation where each time I start it, it now crashes. The log is as follows

14 Aug 20:40:59 - [info] Starting flows
14 Aug 20:40:59 - [info] Started flows
node-red[27912]: ../src/base_object-inl.h:140:static void node::BaseObject::InternalFieldSet(v8::Local<v8::String>, v8::Local<v8::Value>, const v8::PropertyCallbackInfo<void>&) [with int Field = 2; bool (v8::Value::* typecheck)() const = &v8::Value::IsFunction]: Assertion `((*value)->*typecheck)()' failed.
 1: 0x9aedf0 node::Abort() [node-red]
 2: 0x9aee77  [node-red]
 3: 0xa58502  [node-red]
 4: 0xe0d047 v8::internal::Object::SetPropertyWithAccessor(v8::internal::LookupIterator*, v8::internal::Handle<v8::internal::Object>, v8::Maybe<v8::internal::ShouldThrow>) [node-red]
 5: 0xe21cf3 v8::internal::Object::SetPropertyInternal(v8::internal::LookupIterator*, v8::internal::Handle<v8::internal::Object>, v8::Maybe<v8::internal::ShouldThrow>, v8::internal::StoreOrigin, bool*) [node-red]
 6: 0xe21e63 v8::internal::Object::SetProperty(v8::internal::LookupIterator*, v8::internal::Handle<v8::internal::Object>, v8::internal::StoreOrigin, v8::Maybe<v8::internal::ShouldThrow>) [node-red]
 7: 0xfabfc7 v8::internal::Runtime::SetObjectProperty(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, v8::internal::StoreOrigin, v8::Maybe<v8::internal::ShouldThrow>) [node-red]
 8: 0xfacf9a v8::internal::Runtime_SetKeyedProperty(int, unsigned long*, v8::internal::Isolate*) [node-red]
 9: 0x13162b9  [node-red]
Aborted

This is pretty severe and doesn't appear to give any indication as to what I might have broke. I can't seem to restart my Node-RED server as each time I do, I end up with the same crash.

node -v reports

v12.6.0

Umm, that's an assert deep in the native code of the node runtime... not sure where you'd begin tracking that down.

I note you are using Node 12. We still recommend Node 10 - we don't have much testing under our belt with Node 12 and there could well be issues with native code in third party modules... which could cause odd asserts just like this.

I backed down node to v10.16.0 and re-installed all my node_modules. This gave me a whole new stack trace:

14 Aug 20:56:39 - [info] Starting flows
14 Aug 20:56:40 - [info] Started flows
14 Aug 20:56:42 - [red] Uncaught Exception:
14 Aug 20:56:42 - TypeError: Cannot assign to read only property 'writeQueueSize' of object '#<TCP>'
    at _clone (/home/kolban/node-red/node_modules/clone/clone.js:162:16)
    at _clone (/home/kolban/node-red/node_modules/clone/clone.js:162:18)
    at _clone (/home/kolban/node-red/node_modules/clone/clone.js:162:18)
    at _clone (/home/kolban/node-red/node_modules/clone/clone.js:175:25)
    at _clone (/home/kolban/node-red/node_modules/clone/clone.js:162:18)
    at _clone (/home/kolban/node-red/node_modules/clone/clone.js:162:18)
    at _clone (/home/kolban/node-red/node_modules/clone/clone.js:162:18)
    at _clone (/home/kolban/node-red/node_modules/clone/clone.js:162:18)
    at /home/kolban/node-red/node_modules/clone/clone.js:141:24
    at Map.forEach (<anonymous>)

To try and get my system running again, I renamed my flows.json file and was able to restart Node-RED just fine. Of course, at this point I have none of my original flows. Since I am able to start Node-RED just fine with an empty flows.json my next trick will be to massage my copy of that file starting Node-RED one flow at a time until I find the one causing the issue. I'm wondering if there are tools, recipes or techniques that would make this easier since it appears (to me) that at this point I'm text twiddling JSON.

If you are on Node-RED 0.20, then you can start in safe mode by specifying --safe on the command-line.

That will start Node-RED without running your flows. You can then open them in the editor, edit and deploy to start the flows.

To be a bit more systematic, disable all your flows (dbl click on the tab in the editor, toggle their Status property to disabled) and then deploy. Then enable them one at a time, deploying after each to see which tab has the issue.

1 Like

To conclude .... as always @knolleary is THE BEST. The recipe worked perfectly. I was able to start Node-RED using --safe and disabled all my flows until I found the culprit. I then copied it to a new flow tab and deleted the nodes one by one until the error stopped. Re-adding the failing node with the identical configuration caused the whole flow to work just fine so something is badly fishy with that node. However, since I am the author of that node and it is being actively developed, its given me what I need to examine the puzzle. The key take-away from this thread is the fantastic --safe capability that has saved me bunches of work and time ... awesome stuff!!!

6 Likes

A salient lesson that teaches us to code as defensively as possible! I've been there myself plenty of times so I certainly know the feeling. :slight_smile:

I had something very similar a few months back when I started using node 12.10 with NR1.0
Seems it hit me again today.
Luckily within a few minutes, I remembered what caused the issue.
In my case, it was something in a message which was not easily cloneable (a function, or maybe a stream, in my case). I moved it into msg.req (since this does not get cloned!!!!), and issue went away.
node-clone is not great when it comes to cloning some stuff; like anything which involves native elements (e.g. opencv stuff). Personally, I use a modified node-clone which allows a little control when you know something downstream may want to clone an object, but you want it treated in a special way.

To be fair, I think this is why JavaScript doesn't have a built-in clone function. Because JavaScript allows circular referenced objects and cloning is therefore a hard problem.

You can code defensively to work around this issue, if you need to clone an object that might contain something overly complex, simply wrap the clone statement in a try and deal with the failure in a catch.

haha :). that worked before node 12, but crashes like the one from the OP are not caught - Effectively a bug in Node itself that it can and does crash in native code - at least at 12.10 :). I've not tried higher versions to see if they have fixed it....
Also, in this specific case, we're not in control of the clone call... it's the clone of the message in NR.
My crash yesterday also mentioned '../src/base_object-inl.h', but with a different error.

1 Like