Locking up NR...and escape out from there

#1

I was in discussion with another NR user if NR could "by accident" be so "busy" so it lost responsiveness completely. Only way to "escape out" would be to kill the NR process

I thought I should try this and wrote the following code in a function node:

wait(30000);
node.warn('Waiting is over...');

function wait(ms){
    var start = new Date().getTime();
    var end = start;
    while(end < start + ms) {
        end = new Date().getTime();
    }
}

Initiating this with an inject node and during 30 seconds, yes, NR was totally busy, did not respond to anything else

The commands node-red-stop or sudo systemctl stop nodered.service does not work

In this case my loop ended after 30 seconds and NR was responding again. But if you by mistake have created "a loop forever", I think only chance is then to kill the NR process with pkill -f node-red or similar

Or is there another way (except being very careful when writing code in function nodes)?

0 Likes

#2

If you have created infinite loop in your function or wired nodes in the way that NR loses response, you can always kill NR and run it again in safe mode (flows not started in safe mode), fix you mistakes and then NR starts your flow(s) on first publish.
See https://nodered.org/docs/getting-started/running Command-line usage

1 Like

#3

Try a much longer loop time and try node-red-stop again, I think that eventually systemd should kill it, maybe after a couple of minutes, though I am not certain.

0 Likes

#4

Depends how you have setup the service - the default we use in nodered.service is
KillSignal=SIGINT
so it tries to kill it "nicely" which lets it exit cleanly, close connections, and so on... which is what you want most of the time... However if you are in a tight loop - as node.js is single threaded - then it won't yield to allow the INT to happen. Whereas a SIGKILL would be more brutal and just try to kill it... but then under normal circumstances things may not get cleaned up nicely... So up to you if you want to go there... vs just using pkill when you have to.

(and as @hotNipi points out - (from 0.20) you can now start node-red --safe to start without running the flow again so you can edit it to fix the problem before deploying again.)

1 Like

#5

When systemd is told to stop a process it first sends the signal as specified in the service file (SIGINT in the case of the standard node-red service). If the process does not stop within a defined time then systemd will kill the process. If the service file includes a value for TimeoutStopSec= then that time is used, otherwise it uses the value of DefaultTimeoutStopSec= from the systemd defaults.
Since nodered.service does not have TimeoutStopSec= then it uses the default, which for my Ubuntu system appears to be 90 seconds.
So for a normal systemd setup, even if node-red is hung, node-red-stop should stop it within a couple of minutes.
https://www.freedesktop.org/software/systemd/man/systemd.service.html

1 Like

#6

I see the 90 seconds on my Pi too, so it is something standard. I am going to try to uncomment the DefaultTimeoutStopSec and change that to 10 seconds. But I see something weird: If I start Node-RED from one console window and then send the kill from another console window, I get this:

Stopping Node-RED graphical event wiring tool...
nodered.service: State 'stop-sigterm' timed out. Killing.
nodered.service: Killing process 1566 (node-red) with signal SIGKILL.
nodered.service: Main process exited, code=killed, status=9/KILL
Stopped Node-RED graphical event wiring tool.
nodered.service: Unit entered failed state.
nodered.service: Failed with result 'timeout'.

What I don't get is a new, useable command prompt. For that I have to crtl+c in the window where Node-RED runs. Why is that?

0 Likes

#7

Yep, the 20 seconds work nicely. And when I start Node-RED again from the Python script made bry Krambriw, it will reuse the same command prompt window.

1 Like

#8

To set a value for TimeoutStopSec specific to nodered.service you can use
systemctl edit nodered
That will create an override file /etc/systemd/system/nodered.d/override.conf and put you into the editor. Put in there
TimeoutStopSec=20
and that should do the job. That should survive if you upgrade node red.
I think it will use vi as the default editor, if you want nano (for instance) instead if you edit /root/.profile and add (before the includes that may be at the end)
export SYSTEMD_EDITOR="/bin/nano"
Alternatively you can just create the override file manually with your favourite editor.

2 Likes

#9

Great info & suggestions, thank you all!

0 Likes

#10

Thanks! I will probably go that route instead, I guess having standard 10 seconds can mess up something... Oh, btw, I aways use sudo Leafpad when editing something. Nano is just to old school primitive for me. :wink:

0 Likes

#11

Can you use leafpad in an ssh shell?

0 Likes

#12

No idea. Never use those, I only use VNC and the GUI. I only experimented with using SSH to send an automated single command to NR and HA, but Krambriw helped me with a Python script that does that job much better.

1 Like

#13

That's why you can use leafpad then. If the Pi (or whatever) does not have a GUI then you have to use a simple text based editor like nano.

0 Likes

#14

Probably. :slight_smile: Which is one of the reasons I never use anything that doesn't have a GUI. And of course nothing that doesn't support copy/paste from the computer I'm using as the host.

0 Likes

#15

A Pi zero with a GUI is not much fun to use and the GUI doesn't leave much processing or memory for anything else. All of my Pis are headless servers.
You can copy/paste in an ssh shell of course, and if you need a gui for copying files across you can mount an sftp share in your host. Everyone has there own tastes though, there is rarely a 'right' way of doing things.

0 Likes

#16

Of course. I only say what's right for me. :slight_smile: And I buy whatever's the newest when I need a new Pi, so I'm running 3b and 3b+ these days, with 64 gig USB SSD's. My final 2 went into the trash this winter when it became flakey. I don't do flakey. :rofl:

0 Likes

#17

Oh, and don't misunderstand. I'm as cheap as they come, regular Scrooge. But for me an hour spent on faulty equipment is much more expensive then just getting a new Pi that won't crash. Not to mention the WAF, that can take very expensive damage! A crash every other months is "this stuff never works"! :crazy_face::rofl:

0 Likes

#18

A Pi Zero with no gui is much cheaper than a Pi 3 with a GUI. It won't do every job of course but it is perfectly capable of running a modest node red flow and an MQTT server.

0 Likes

#19

Yeah, I know. But I mostly run that plus Home Assistant on them all. And as I said I prefer GUI, not to mention that the difference in price in Norway is small, only around 10-15 pounds.

0 Likes