Exec node - piping payload to stdin for commands

[gregorius]

Hi There,

I was trying to create a flow that simulates who | wc -l (from this question) but I realised that it's not that simple...

What I wanted to do was connect to exec nodes together, one doing the who and the other doing the wc -l:

what happens is that the wc -l is killed because it gets no input via stdin.

I could do this by getting who to write a file and then applying wc -l to that file. Or use the pipe command line in the exec node: who | wc -l. But that makes no sense to me since Node-RED creates "pipes" in the form of flows, so why should I being using a Unix "pipe" in an exec node.

What I would like to have is a "map payload to stdin" option in the exec node, something like this:

Thoughts?

[gregorius]

I've created a prototype node that does this. Code is based on the original node with a minor extension of piping to the child process' stdin:

if (this.paytostdin) {
  var value = RED.util.getMessageProperty(msg, node.paytostdin);
  if (value !== undefined) {
    var stdinStream = new streamLib.Readable();
    stdinStream.push(value);  // Add data to the internal queue for users of the stream to consume
    stdinStream.push(null);   // Signals the end of the stream (EOF)
    stdinStream.pipe(child.stdin);
  }
}      

of course there is no error checking nor handling.

[dceejay]

While skeuomorphically nice - I'm not sure the shelling in and out of the OS multiple times is really worth it for the additional complexity of the config UI. (plus a lot of users tend to want fewer nodes around rather than more - no idea why

I think the config UI would need to be somehow made more obvious so it doesn't trip up naive users, as having two "variable" parts ("stdin" and "append") is just confusing to me at least, especially when the stdin one is only of use for a minority of use cases.

[gregorius]

@dceejay

as having two "variable" parts ("stdin" and "append")

Definitely, especially since I can now append the payload to the command line and pass it in as stdin - obviously a very rare use case!

So yes, the UI is confusing and the use case potentially limited (although I can think of one scenario - for me - where this would be useful (I ended up writing files to disk)).

Having said that, conceptually I think this is relevant and makes Node-RED more consistent - who | wc -l just feels wrong in an exec node.

@dceejay

I'm not sure the shelling in and out of the OS multiple times

This is a philosophical question: is it worth using any high level language instead of assembler? Would be the logical conclusion of that statement.

Sure the cost might be silly and it would be better to combine all shell activity into a single exec node but we do these things so that we have clarity. We only use high level programming language to better understand (usually six months later) what we are doing/did.

Else we should be doing everything using low-level assembler-like languages. Evening using NodeJS can be an overhead for certain situations aswell.

So I'm not looking to improve efficiency but clarity here ...

[jbudd]

Unix filters (read from stdin, write to stdout) and pipelines are elegant and beautiful, surely you should embrace rather than spurn them?

But if the appearance of a pipe symbol in your exec command causes concern - i recognise it may not be universally loved - why not hide it in a shell script ~/bin/whocount and exec the script?

[TotallyInformation]

Agree with Dave and jbudd here. And in fact, you are already using TWO high-level languages. But you are using the 2nd language (BASH presumably) in a not very effective way.

Having said that, the idea of making a node's input such that it accepts inputs more flexibly is certainly the modern "node-red way".

Perhaps though, it would also be an interesting contrib node for someone that looked like a function node but for BASH rather than node.js - rather like the python node. (other shells may be available! )

[dceejay]

I'm certainly not against the idea. (I like nodes for clarity after all !) - so yes happy to discuss/consider the idea - but need to get the config UI right so it is an enhancement and not a cause for confusion.

@gregorius

is it worth using any high level language instead of assembler? Would be the logical conclusion of that statement.

Well no - my conclusion is that the high level language should reduce (or hide) the complexity of the lower level language - so why just recreate all the low level commands 1 for 1 in a new form - yes maybe great for those that already know the existing command - but for newcomers a single "function" that just creates the desired output is more understandable - imho.

But as I started - yes I'm all for pipes "just working" - but need to make it easier not more confusing to set up.

[dceejay]

@TotallyInformation

also be an interesting contrib node

Well there is the daemon node for longer running processes that does accept stdin... BUT of course in this case it is the command (wc -l) that isn't long running - so that would need tweaking also...

(so for me this "feels like" the right place to do it - IF it can be done cleanly)

[gregorius]

@dceejay

but for newcomers a single "function" that just creates the desired output is more understandable - imho.

All I see is an LED and it's red.
-- The managerial view of software development!

@dceejay

But as I started - yes I'm all for pipes "just working" - but need to make it easier not more confusing to set up.

Definitely, I'm all for making things simpler - which I did but for me! For me it's truly simpler (and provides more clarity) by not having a | in an exec node (in the context of Node-RED, using Unix I'd be the first using pipes!). Of course the UI is a dumpster fire waiting to happen but that wasn't the point of the exercise

@dceejay

(so for me this "feels like" the right place to do it - IF it can be done cleanly)

We're on the same page, just different paragraphs then!

@TotallyInformation

Perhaps though, it would also be an interesting contrib node for someone that looked like a function node but for BASH rather than node.js - rather like the python node. (other shells may be available! )

You do know that I did also create a "function" node that runs in the browser Now if I chain a function node to a "Client Code" (as I called the function node for the browser) and then a shell "function" node together even I would be getting confused!

But python is possible, for example this:

[{"id":"a47e73d8010620c9","type":"inject","z":"46399d4214c0243f","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":447,"y":733,"wires":[["35e03f470f51d890"]]},{"id":"35e03f470f51d890","type":"template","z":"46399d4214c0243f","name":"some python code","field":"payload","fieldType":"msg","format":"handlebars","syntax":"mustache","template":"print(\"This is python code\")\nif True:\n    print(\"And that is the truth!\")","output":"str","x":689,"y":665,"wires":[["e01cf12d89d9d18c"]]},{"id":"e01cf12d89d9d18c","type":"execWithPipe","z":"46399d4214c0243f","command":"/usr/bin/python3.11","addpay":"","paytostdin":"payload","append":"","useSpawn":"false","timer":"","winHide":false,"oldrc":false,"name":"","x":906,"y":556.75,"wires":[["6093a571a538f58a"],[],[]]},{"id":"6093a571a538f58a","type":"debug","z":"46399d4214c0243f","name":"debug 294","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":1125,"y":434,"wires":[]}]

I.e. the template node contains python code that is passed to the msg.payload that is then piped into the execWithPipe node ... and it actually works!

Either way, the idea is there, the node is there, I won't create a contrib package but I will write a blog post about it!

EDIT: The blog post is online with screencasts of how I created the execWithPipe node within Node-RED.

[gregorius]

@jbudd

why not hide it in a shell script ~/bin/whocount and exec the script?

You might laugh but I actually did that for a project of mine. The only difference was that I created the script in Node-RED, saved it to disk and the used an exec node to run the script! Just don't tempt me to do it again

The advantage of doing it in Node-RED was that I could us the mustache templating engine to replace variables in the script before writing to disk. I found it to be an elegant solution since I didn't have to touch my text editor to create the script.

[jbudd]

Not laughing.

It's messy for Node-red to rely on an external script file.

You can define and execute a script entirely within Node-red, While I am happy with simple pipelines in the exec node, for slightly more complex scripts I do it like this: (Sorry my example uses RPi specific commands)

[{"id":"88c26d2f3ed7f321","type":"group","z":"db04fd79f3a93d19","name":"A non-interactive Bash script","style":{"label":true},"nodes":["106bb647629b1f06","b0369609aa7d7ab6","513da4b56a8bb46e","686db71073ded405"],"x":54,"y":1111.5,"w":592,"h":97},{"id":"106bb647629b1f06","type":"inject","z":"db04fd79f3a93d19","g":"88c26d2f3ed7f321","name":"Go","props":[{"p":"payload"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"go","payloadType":"str","x":150,"y":1160,"wires":[["686db71073ded405"]]},{"id":"b0369609aa7d7ab6","type":"exec","z":"db04fd79f3a93d19","g":"88c26d2f3ed7f321","command":"","addpay":"payload","append":"","useSpawn":"false","timer":"","winHide":false,"oldrc":false,"name":"","x":430,"y":1160,"wires":[["513da4b56a8bb46e"],["513da4b56a8bb46e"],[]]},{"id":"513da4b56a8bb46e","type":"debug","z":"db04fd79f3a93d19","g":"88c26d2f3ed7f321","name":"output","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":550,"y":1160,"wires":[]},{"id":"686db71073ded405","type":"template","z":"db04fd79f3a93d19","g":"88c26d2f3ed7f321","name":"Bash script","field":"payload","fieldType":"msg","format":"handlebars","syntax":"mustache","template":"# Discover and format CPU frequency & temperature\n\n# vcgencmd measure_clock_arm returns eg \"frequency(48)=600169920\"\nFREQ=$(vcgencmd measure_clock arm | sed -E 's/(.*=)(.*)/\\2/')\nFREQ=$[FREQ / 1000000]\n\n# vcgencmd measure_temp returns eg \"temp=30.6'C\"\nTEMP=$(vcgencmd measure_temp | sed -E 's/(.*=)([0-9]*)(.*)/\\2/')\nprintf \"CPU %d MHz, %d °C\" $FREQ $TEMP","output":"str","x":290,"y":1160,"wires":[["b0369609aa7d7ab6"]]}]

And an example passing parameters

[{"id":"2954d59d172778e7","type":"group","z":"db04fd79f3a93d19","name":"Passing parameters to a Bash script as environment variables","style":{"label":true},"nodes":["4d82f796d89d6044","f66b1b60040f4067","d9290c0c62e0f7c0","cc39d099b49cf9fe","af8dbd68b05f6313","bb3f370af4971b28"],"x":34,"y":951.5,"w":772,"h":129.5},{"id":"4d82f796d89d6044","type":"inject","z":"db04fd79f3a93d19","g":"2954d59d172778e7","name":"","props":[{"p":"payload"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"/home/pi","payloadType":"str","x":140,"y":1000,"wires":[["f66b1b60040f4067"]]},{"id":"f66b1b60040f4067","type":"template","z":"db04fd79f3a93d19","g":"2954d59d172778e7","name":"Bash script ","field":"payload","fieldType":"msg","format":"handlebars","syntax":"mustache","template":"DIR={{{payload}}}\nsudo du -sm $DIR | awk '\n{printf \"{\\\"directory\\\": { \\\"name\\\": \\\"%s\\\", \\\"diskuse\\\": %d, \\\"units\\\": \\\"MB\\\"}}\", $2, $1}'","output":"str","x":290,"y":1000,"wires":[["cc39d099b49cf9fe","d9290c0c62e0f7c0"]]},{"id":"d9290c0c62e0f7c0","type":"exec","z":"db04fd79f3a93d19","g":"2954d59d172778e7","command":"","addpay":"payload","append":"","useSpawn":"false","timer":"","winHide":false,"oldrc":false,"name":"","x":450,"y":1000,"wires":[["bb3f370af4971b28"],[],[]]},{"id":"cc39d099b49cf9fe","type":"debug","z":"db04fd79f3a93d19","g":"2954d59d172778e7","name":"Full Script","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":460,"y":1040,"wires":[]},{"id":"af8dbd68b05f6313","type":"debug","z":"db04fd79f3a93d19","g":"2954d59d172778e7","name":"output","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":710,"y":1000,"wires":[]},{"id":"bb3f370af4971b28","type":"json","z":"db04fd79f3a93d19","g":"2954d59d172778e7","name":"","property":"payload","action":"","pretty":false,"x":590,"y":1000,"wires":[["af8dbd68b05f6313"]]}]

[TotallyInformation]

@gregorius

The advantage of doing it in Node-RED was that I could us the mustache templating engine to replace variables in the script before writing to disk. I found it to be an elegant solution since I didn't have to touch my text editor to create the script.

With the advent of the uib-save node, I do this kind of thing quite a bit now so that setup is kept together with the runtime flows:

[gregorius]

@jbudd

execute a script entirely within Node-red,

You're using the append option that takes the payload and adds it to the command line for the command. That does not work in my python example above. I can't (or at least in the 30secs I tried) take a template with python code and pass it directly to the python command (I'm not talking about creating extra scripts or files) using append.

BTW what does an exec node execute if there is no command provided? As in your case. Do you provide a command via the msg object?

[jbudd]

@gregorius

Do you provide a command via the msg object?

I pass the entire script as msg.payload.
I assume that the exec node concatenates the command, msg.payload (if ticked) and extra parameters before it invokes bash to run the requested command, thus it does not care if the "command" box is empty.

@gregorius

That does not work in my python example above.

Nope. I have attempted to make it work for Python but I concluded it's not practical because the entire script needs to be wrapped in single quotes to pass to python, and thus may not contain single quotes.
(As far as I recall, a hashbang line at the start of any script formed in this way is not respected so no #! /usr/bin/python )
--- Perhaps @dceejay can suggest a work-around to trick the shell into processing a hashbang line at the start of a script passed in this way?
FWIW

[{"id":"3cce5247d5657414","type":"group","z":"db04fd79f3a93d19","name":"Executing a python script - NOT PRACTICAL","style":{"label":true},"nodes":["23a656330a015255","0231db705087ba50","d943c3197090c7a9","b3e6f7810bb068bb","1f69acd98bdddd94","33b0bca1f9569ef6"],"x":34,"y":699,"w":872,"h":122},{"id":"23a656330a015255","type":"inject","z":"db04fd79f3a93d19","g":"3cce5247d5657414","name":"You can pass parameters in msg.payload","props":[{"p":"payload"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"4.2","payloadType":"num","x":240,"y":740,"wires":[["1f69acd98bdddd94"]]},{"id":"0231db705087ba50","type":"template","z":"db04fd79f3a93d19","g":"3cce5247d5657414","name":"Python script","field":"payload","fieldType":"msg","format":"handlebars","syntax":"mustache","template":"python -c '            # -c argument - accept command inline. Must enclose whole script in single quotes.\nimport sys             # Process argument list\nnum1 = float(sys.argv[1])\nstr = sys.argv[2]           # Note python/bash style comments\n\nprint (str)\nif(num1 <= 10):\n  print (num1, \"is less than 10\")\nelse :\n  print (num1, \"is greater than 10\")\n\n' {{payload}} {{flow.printme}}     # Closing single quote followed by list of arguments\n","output":"str","x":370,"y":780,"wires":[["d943c3197090c7a9","b3e6f7810bb068bb"]]},{"id":"d943c3197090c7a9","type":"exec","z":"db04fd79f3a93d19","g":"3cce5247d5657414","command":"","addpay":"payload","append":"","useSpawn":"false","timer":"","winHide":false,"oldrc":false,"name":"","x":670,"y":760,"wires":[["33b0bca1f9569ef6"],["33b0bca1f9569ef6"],[]]},{"id":"b3e6f7810bb068bb","type":"debug","z":"db04fd79f3a93d19","g":"3cce5247d5657414","name":"Full Script","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":520,"y":740,"wires":[]},{"id":"1f69acd98bdddd94","type":"change","z":"db04fd79f3a93d19","g":"3cce5247d5657414","name":"or in context variables","rules":[{"t":"set","p":"printme","pt":"flow","to":"This\\ script\\ defined\\ in\\ Node-red.","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":180,"y":780,"wires":[["0231db705087ba50"]]},{"id":"33b0bca1f9569ef6","type":"debug","z":"db04fd79f3a93d19","g":"3cce5247d5657414","name":"output","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":810,"y":760,"wires":[]}]

[dceejay]

Meanwhile - back on the main thread... (unless it has now been abandoned...)
The UI to declutter is this

maybe

• append msg. xxx as [parameter]/[stdin] <-- select

other ideas ???

[jbudd]

@dceejay

(unless it has now been abandoned...)

Hmm. I apologise.

[gregorius]

@dceejay

[parameter]/[stdin] <-- select

you mean a dropdown since these two options are mutually exclusive? Sounds good to me - I'll give a whirl on the old FJ in a sec.

[gregorius]

@jbudd

Hmm. I apologise.

All good with me, no apology needed

[dceejay]

Yes - exactly that - with default being parameter (as today) of course.

[dceejay]

indeed - no need to apologise - but feel free to fork another discussion referencing back.