'Illegal instruction' thrown when logging into editor - Node-RED crash

General
1

The website is down, so I'm a little frantic/panicked, but will try and take my time here to explain the sequence of events....

My site has been running since March 22nd 2024 mostly trouble free.

A few days ago I happened to notice this error one time in the debug. (I was monitoring some strings, nothing that could have caused this):

"Flushing file /root/.node-red/.sessions.json to disk failed : Error: ENOENT: no such file or directory, rename '/root/.node-red/.sessions.json.$$$' -> '/root/.node-red/.sessions.json'"

I made a mental note to Google it since I have (in 9 years) never seen anything like it, but everything kept working and of course I forgot.

Fast forward a few days and scores of deploy and work on the site later....

A few hours ago I got a site monitoring alert that the server was unresponsive (site down alarm).
If you want to know why I have a site monitoring service, this thread will help: Dashboard load times > 30 sec with more than 250 connections to website

Normally I just check pm2 monit and then do a pm2 restart of the main node-red and we are back up and running.... Not this time.
This time when I checked pm2, I saw the memory use had gone wildly out of control...

image
image850Ă—277 72.6 KB

I've never seen it so high. I've seen 3-5 Gb, but never 12.
When I restarted, I found node-red to be in a boot loop.
I then did a pm2 stop 0
Looking at the log shows this error:

image
image1347Ă—489 198 KB

I then tried making the file .sessions.json.$$$ but it still boot-loop.
I then tried removing all the .sessions files. Boot-loop.
I removed flows.json and get a worrying error....

< trim the last few lines with senstive stuff in them >

15 Feb 19:43:33 - [error] [sqlite:stdc] Error: SQLITE_ERROR: table allmsg already exists
15 Feb 19:43:34 - [error] [sqlite:mission] Error: SQLITE_ERROR: table missioncode already exists
Illegal instruction

At that point, node-red has crashed.
When I check dmesg I see:

[3550905.921067] traps: node-red[905034] trap invalid opcode ip:1d3a55d sp:7ffc76390548 error:0 in node[400000+4bf1000]
[3551003.054291] traps: node-red[940484] trap invalid opcode ip:1d3a55d sp:7ffdaa030208 error:0 in node[400000+4bf1000]
[3551252.395371] traps: node-red[909663] trap invalid opcode ip:1d3a55d sp:7ffc7dccfa08 error:0 in node[400000+4bf1000]

Tons of these, but those are the past few.

Big clue?
The Illegal instruction is thrown right when I click 'log in' on the editor credentials dialog.

Node-RED version: v4.0.9
15 Feb 20:10:55 - [info] Node.js  version: v18.20.4
15 Feb 20:10:55 - [info] Linux 5.15.0-126-generic x64 LE

I then did npm install -g --unsafe-perm node-red
Still crashes.
So, fresh flows.json, fresh install of node-red and Im dead in the water.
I cant log into the editor.

Any suggestions gratefully accepted.

2

You could try reinstalling nodejs. Otherwise run disk and memory checks. I suspect hardware. I suppose it could be a hack, but more likely hardware I think.

3

Re installing nodjs takes me to v18.20.6
Same error when I click log in.

Server is a VM in a data center... But I can try some command line tests (ie, I cant boot to any sort of bios or other diagnostics).

4

Found a hint.

If I comment out my admunAuth section from the settings.js, it allows me into the editor Ok.

 //adminAuth: {
 84     //    type: "credentials",
 85     //    users: [{
 86     //        username: "admin",
 87     //        password: "$2a$08$zZWtXTja0fB1pzD4sHCMyOCMYz2Z6dNbM6tl8sJogENOMcxWV9DN.",
 88     //        permissions: "*"
 89     //    }]
 90     //},

This bit here, but of course with my user/password.
I've not changed them for a few years.
So not sure why they are throwing the illegal instruction crash all of a sudden.

EDIT: Changed the httpAdminRoot directory name, the username and password and the illegal instruction error still crashes Node-RED.

Second EDIT: starting in safe mode does not make any diffrence.

your settings file. Node-RED will then re-encrypt your credentials
file using your chosen key the next time you deploy a change.
---------------------------------------------------------------------

15 Feb 23:15:17 - [warn] Encrypted credentials not found
15 Feb 23:15:17 - [info] *****************************************************************
15 Feb 23:15:17 - [info] Flows stopped in safe mode. Deploy to start.
15 Feb 23:15:17 - [info] *****************************************************************
Illegal instruction

As soon as I log in, crash.
If use the wrong creds, Node-RED keeps running, but of course I cant log in.

What runs straight after you log in?

Third Edit.
9 hours later and I am still the wiser.
As best as I can tell something broke the auth service that Node-RED uses.
I cant change any user/pass anywhere in the flow without causing the 'illegal instruction' error and the hard crash.
For example, I wanted to remove an MQTT broker as it was giving a connect error for some reason, if I make a new broker as soon as I put in the user/pass and deploy, it crashes. If I change the existing broker, I can deploy Ok.

5

Tends to be an indication of something getting corrupted, i.e. some dynamic library was upgraded and some application using that library fails with an illegal instruction because it was not recompiled or upgraded. It's a very low level error and would most probably have nothing to do with your authentication in the settings.json.

Did you upgrade any packages? Did you upgrade any software on the machine? Did you reinstall anything on the underlying machine? Are you running in a docker image or on bare metal? Have you tried reproducing the error on another machine using the exactly same setup?

Most things don't break unless something has changed.

Alternatively the disks are breaking "disk failed : Error: ENOENT: no such file or directory," is not healthy. If you can access the directory but Node-RED can't:

then there is some mismatch in location.

I would also say try NodeJS 20.x.y especially since your running NR 4.0.9. Perhaps some node package you're using requires NodeJS 20 ... who knows!

6

It would be a good idea to rebuild any nodejs modules that require compiling. To do this, first go into your .node-red directory and run
npm rebuild
That will rebuild any custom nodes you have installed.

Then run
npm list -g --depth 0
which will tell you where node-red is installed, probably usr/lib. Go into that folder and run
sudo npm rebuild
which will rebuild any core nodes that need it.
If you update nodejs then you must do the rebuild anyway, after upgrading nodejs.

7

Had a friend help me out.
He attached gdb (a debugger) and found this message when logging in:

Thread 1 "node-red" received signal SIGILL, Illegal instruction.
0x0000000001d48f5d in base64_stream_encode_avx512 ()
#0  0x0000000001d48f5d in base64_stream_encode_avx512 ()
No symbol table info available.
#1  0x0000000001d3c424 in base64_encode ()
No symbol table info available.
#2  0x0000000000c876b8 in node::StringBytes::Encode(v8::Isolate*, char const*, unsigned long, node::encoding, v8::Local<v8::Value>*) ()
No symbol table info available.
#3  0x0000000000b70b1d in void node::Buffer::(anonymous namespace)::StringSlice<(node::encoding)2>(v8::FunctionCallbackInfo<v8::Value> const&) ()
No symbol table info available.

Which led to this: NodeJS unconditionally uses AVX512 instructions even if they are disabled · Issue #53426 · nodejs/node · GitHub
Found the base64 node was an issue.
Upgrading to nodejs 20.x resolved the issue.

I honestly can not explain how a Node-RED restart caused the issue this time but not any of the other times with zero updates, installs or changes in the interim.

2 Likes