Exec node timeout not working in 'exec' mode

General
1

Following on from a recent post it seems that the Exec node Timeout feature does not work when configured for exec mode, though it does work for spawn mode. See the attached example flow. The exec node runs a 20 second sleep and the timeout is set to 10 seconds, so the node should terminate after 10 seconds. In fact it does not terminate until the 20 seconds has elapsed, though at that point it does indicate a SIGTERM error. Looking at ps it seems that the exec node starts a shell which runs the sleep command and the timeout does kill the shell, but not the spawned sleep process.
If the Exec node is switched to spawn mode then it terminates correctly after 10 seconds.
This is not limited to the sleep command, I also see it running other software.
I am running node-red 1.0.6 with nodejs 12.18.0 on Ubuntu 19.10.

image
image694Ă—220 15.1 KB

[{"id":"30e1839a.cade6c","type":"exec","z":"8c99ec05.74e658","command":"sleep 20","addpay":false,"append":"","useSpawn":"false","timer":"10","oldrc":false,"name":"","x":420,"y":423,"wires":[["dccd5011.4926e"],["64252707.f9e7"],["a9768a27.692d6"]]},{"id":"127167a5.30506","type":"inject","z":"8c99ec05.74e658","name":"Go","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":124,"y":389,"wires":[["30e1839a.cade6c","7d39cdd0.42cc64"]]},{"id":"dccd5011.4926e","type":"debug","z":"8c99ec05.74e658","name":"OP/1","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","x":590,"y":383,"wires":[]},{"id":"64252707.f9e7","type":"debug","z":"8c99ec05.74e658","name":"OP/2","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","x":588,"y":425,"wires":[]},{"id":"a9768a27.692d6","type":"debug","z":"8c99ec05.74e658","name":"OP/3","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","x":588,"y":465,"wires":[]},{"id":"7d39cdd0.42cc64","type":"debug","z":"8c99ec05.74e658","name":"START","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","x":601,"y":334,"wires":[]}]
2

The exec node uses the node.js child.process module either in exec or spawn mode. Have a read at
https://nodejs.org/api/child_process.html
for a lot of info on how they work on different systems.
Unfortunately their is quite a few little caveats in how this module spawns processes that can lead to things like orphaned processes and so on.
Nodered has very little influence on how this happens as the exec and daemon nodes are pretty much simple straight forward wrappers around some of the functionality of the child.process module.
So probably your problem lies on the node.js side and not the exec node implementation side so you d have to do research there about this behavior.

Johannes

3

Did you try saving the pid from the node status and killing it with a kill command and a negative pid from a second exec node to also kill the spawned processes as suggested in the other thread?

4

It is not causing me problems, as I am happy to use spawn mode, but if Timeout does not work in exec mode then perhaps the feature should be disabled in that mode.

5

But timeout does work in exec mode. Just not for some processes that spawn their own childs.
All timeout does in the exec node when you look at the code is to use setTimeout() and when this timeout is reached call the child.process.kill() command.
So its works for most processes, just not for those that spawn their own childs in some instances. It will work just fine for a python script or an aplay command or something like that.
Unfortunately the node cant know if the command you enter will do that or not.

6

I don't think the sleep command spawns another process, if it did then it wouldn't work in spawn mode either, would it?
I can't get aplay to run at all from an exec node. I get

Command failed: aplay <fully qualified filename>
ALSA lib pcm_dmix.c:1108:(snd_pcm_dmix_open) unable to open slave
aplay: main:828: audio open error: No such file or directory
7

Dont know about sleep to be honest as I never used that as a child.process but it does work for almost all other commands as all it really does is send the TERM signal to STDIN. So this seems to be a sleep specific issue really if its not the orphaned process problem.

This looks more like an alsa configuration problem and is not nodered related.
Alsa configuration on the other hand is a completely diffrent nightmare :see_no_evil:

8

The actual command I was trying to use is influxd which shows the same issue. Works in spawn mode but not in exec.
Can you provide a simple example which does work that I can try to see if it is something specific to my system?

1 Like
9

First it looks like sleep just doesnt accept standard signals to stdin as of its manpage which would explain the behaviour.
https://pubs.opengroup.org/onlinepubs/9699919799.2016edition/utilities/sleep.html

10

That is probably a different issue to the sleep one. So the influxd command can be killed by the timeout in spawn mode but not in exec? The thing is that they both use the exact same clean up function when a timeout is reached.

11

Ok im seeing the same result as you after further testing. There is some async processes which can be killed in spawn mode but not in exec. This isnt just the timeout that doesnt work but any kill command from node.js wont work at all.
Edit:
Ok what you are seeing is this exact issue https://github.com/nodejs/node/issues/7281
So its not a timeout problem per se but a problem of the way that node.js child.process exec works.
So with some processes they will only be killable at all from node.js/nodered if used with spawn and or detached:true.

12

Does that mean exec should be using execFile instead of exec?

You said there are some that can be killed in spawn but not in exec, but are there any cases where the timeout works in the exec node, in exec mode?
I have tried it with a ruby script and that won't terminate either.

13

Yes works for me with aplay for example and from my experience using the exec child.process outside of the exec node in node.js the kill command works for most processes i would say.
No switching to execFile wouldn’t be good as it doesn’t spawn a shell and so has different limitations like not working on windows at all.

14

Sadly, as I said, aplay won't work for me. I don't do python, can you post a simple python script that works?

16

Ok now im scratching my head. A command running in exec doesnt seem to be able to be killed at all.

17

The issue seems to me to be the fact that the node invokes a shell to run the command, and the kill terminates the shell but not the command itself

18

yes looks that way im just doing some further testing :expressionless:
Edit i must have had the exec node in spawn mode by accident when i was testing before :see_no_evil:
Edit two:
I didnt i just overlooked that the aplay process was still running in htop.

19

Ok I found the issue.
Node.js spawns the shell for the process using /bin/sh which on debian and ubuntu symlinks to the dash shell and not bash. The dash shell seems to be not able to handle the Signals send by node.js properly and doesn’t kill the process when it receives the signal. I just tested this to be true.
You can add an argument to the exec command in node.js specifying which shell to use. So I added shell: "/bin/bash" to the code of the exec node js file that calls the exec command and after a restart of nodered everything works as expected now. You can kill anything started in exec mode and the timeout works too.
I’m not sure how this could be implemented as a standard feature as it’s only really a thing specific to certain linux distributions.
Edit:
@knolleary should i open an issue for this on the exec node on github? as it does render some of the functionality of the exec node when in exec mode useless on debian based systems like the raspberry pi

1 Like
20

Yes please raise an issue, but I'm not sure what the fix will be. We can't assume /bin/bash exists.

1 Like
21

Maybe check which platform it runs on and if it is linux check where the /bin/sh symlink is pointing with fs.realpath. If it’s dash add the bash line to the exec command.
But that does seem to be a rather complex and specific solution to that problem.