Bash code and data within Node-red

General
1

A great man recently said

Part of our thinking was that [Node-red] would never be perfect for everyone ... and ... by having flexibility like the function node, or jsonata ... we might encourage people to take some steps towards programming

In this admirable spirit of permitting other programming languages within a Node-red flow, I've been trying to use another scripting language - awk. It does work, but it's ugly and fragile - it breaks if msg.payload includes the single quote character.

Untitled 2
Untitled 21000×249 24.1 KB

[{"id":"592ffdee5f873981","type":"group","z":"909b559e8da63ab3","name":"Pass both data and code to Bash","style":{"label":true},"nodes":["4b1f05c457207ae7","651a384bfd552f20","1a2253e8bccbf044","24b6092031129039","40de53dbd53a8a68","6b30c9d1264f0ab6","0e47b3a6a310d146","dce992b473605617","1e226f7cee4efb99"],"x":314,"y":31.5,"w":692,"h":209.5},{"id":"4b1f05c457207ae7","type":"exec","z":"909b559e8da63ab3","g":"592ffdee5f873981","command":"echo '","addpay":"payload","append":"' | awk -v FS=\"[:,]\" '{ printf \"%f%%\\n\", (100 * ($2 - $6) / $2) }'","useSpawn":"false","timer":"","winHide":false,"oldrc":false,"name":"","x":770,"y":80,"wires":[["651a384bfd552f20"],["651a384bfd552f20"],[]]},{"id":"651a384bfd552f20","type":"debug","z":"909b559e8da63ab3","g":"592ffdee5f873981","name":"Direct","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":910,"y":80,"wires":[]},{"id":"1a2253e8bccbf044","type":"inject","z":"909b559e8da63ab3","g":"592ffdee5f873981","name":"Data as javascript string","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"{\"total\":467,\"used\":124,\"free\":88,\"available\":282,\"units\":\"MB\"}","payloadType":"str","x":460,"y":80,"wires":[["4b1f05c457207ae7"]]},{"id":"24b6092031129039","type":"inject","z":"909b559e8da63ab3","g":"592ffdee5f873981","name":"Same data as json","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"{\"total\":467,\"used\":124,\"free\":88,\"available\":282,\"units\":\"MB\"}","payloadType":"json","x":450,"y":120,"wires":[["40de53dbd53a8a68"]]},{"id":"40de53dbd53a8a68","type":"json","z":"909b559e8da63ab3","g":"592ffdee5f873981","name":"","property":"payload","action":"","pretty":false,"x":630,"y":120,"wires":[["4b1f05c457207ae7"]]},{"id":"6b30c9d1264f0ab6","type":"comment","z":"909b559e8da63ab3","g":"592ffdee5f873981","name":"{\"total\":467,\"used\":124,\"free\":88,\"available\":282,\"units\":\"MB\"}","info":"","x":660,"y":160,"wires":[]},{"id":"0e47b3a6a310d146","type":"comment","z":"909b559e8da63ab3","g":"592ffdee5f873981","name":"awk -v FS=\"[:,]\" '{ printf \"%f%%\\n\", (100 * ($2 - $6) / $2) }'","info":"","x":650,"y":200,"wires":[]},{"id":"dce992b473605617","type":"comment","z":"909b559e8da63ab3","g":"592ffdee5f873981","name":"Data:","info":"","x":390,"y":160,"wires":[]},{"id":"1e226f7cee4efb99","type":"comment","z":"909b559e8da63ab3","g":"592ffdee5f873981","name":"Code:","info":"","x":390,"y":200,"wires":[]}]

Does anyone have a more elegant way to spin off processing to Bash, without using a script file or temporary data file?

Is there any way the exec node could be made more welcoming of external code interpreters such as Bash, python, perl, Csh etc?

2

I think the example provided is a strange one, because you the data you are looking for here can already be provided in node-red. Do you have another example use-case ?

Then again, you could use a template node to create the full command and pass it to the exec node.

3

It's not that I think that calculating a percentage can only be done in awk. I just picked snippets of code and data which are neither complex nor trivial.

I see a Node-red flow as analagous to a Linux command. Data flows from process to process in a pipeline. Of course it is not specific to Linux.

Untitled 4
Untitled 4939×67 6.4 KB

Untitled 5
Untitled 5939×75 6.07 KB

Node-red is and should be capable of piecing together internal and external processes.
So "Process 2" above could be a switch node, javascript in a function, jsonata in a change node or an external process via exec.

I just wonder if exec could make it easier to call external processes without dodgy techniques like an opening single quote in the "Command" field and it's partner in the "Extra parameters" field.
So for example, would it be a useful enhancement if there was an option in exec to wrap the payload in the user's choice of delimiters?
What about a button to open a multi-line editor for the Command and Extra Parameters field like the Inject node's json editor?
Are there other ways it could be enhanced?

I don't know the answers but I suggest that the existence of at least two contrib nodes to include python code demonstrates that other people find exec insufficient.

4

in the example you've given you don't need the "dodgy quotes" ... as the info bar says you need to add quotes if your payload includes spaces. (As we treat them a separate parameters - as indeed does bash, awk, grep etc)

5

That's not what I see Dave. I expanded the awk code to print it's breakdown of the input data into fields.

With dodgy quotes:

[{"id":"032a996ff859ee48","type":"inject","z":"2bf1da3242bae24e","name":"Data as javascript string","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"{\"total\":467,\"used\":124,\"free\":88,\"available\":282,\"units\":\"MB\"}","payloadType":"str","x":220,"y":560,"wires":[["475eb32f395eb16d"]]},{"id":"475eb32f395eb16d","type":"exec","z":"2bf1da3242bae24e","command":"echo '","addpay":"payload","append":"'| awk -v FS=\"[:,]\" '{ printf \"$1_%s_ $2_%s_ $3_%s_ $4_%s_ $5_%s_ $6_%s_ Calculation: %f%%\\n\", $1, $2, $3, $4, $5, $6, (100 * ($2 - $6) / $2) }'","useSpawn":"false","timer":"","winHide":false,"oldrc":false,"name":"","x":530,"y":560,"wires":[["83479a7d28776d38"],["83479a7d28776d38"],[]]},{"id":"83479a7d28776d38","type":"debug","z":"2bf1da3242bae24e","name":"Direct","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":670,"y":560,"wires":[]}]

$1_ {"total"_ $2_467_ $3_"used"_ $4_124_ $5_"free"_ $6_88_ Calculation: 81.156317%

Without them:

[{"id":"032a996ff859ee48","type":"inject","z":"2bf1da3242bae24e","name":"Data as javascript string","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"{\"total\":467,\"used\":124,\"free\":88,\"available\":282,\"units\":\"MB\"}","payloadType":"str","x":220,"y":560,"wires":[["475eb32f395eb16d"]]},{"id":"475eb32f395eb16d","type":"exec","z":"2bf1da3242bae24e","command":"echo ","addpay":"payload","append":"| awk -v FS=\"[:,]\" '{ printf \"$1_%s_ $2_%s_ $3_%s_ $4_%s_ $5_%s_ $6_%s_ Calculation: %f%%\\n\", $1, $2, $3, $4, $5, $6, (100 * ($2 - $6) / $2) }'","useSpawn":"false","timer":"","winHide":false,"oldrc":false,"name":"","x":530,"y":560,"wires":[["83479a7d28776d38"],["83479a7d28776d38"],[]]},{"id":"83479a7d28776d38","type":"debug","z":"2bf1da3242bae24e","name":"Direct","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":670,"y":560,"wires":[]}]

$1_total_ $2_467 used_ $3_124 free_ $4_88 available_ $5_282 units_ $6_MB_ Calculation: 100.000000%

You can see that in the absense of quotes awk splits the input string differently.
It is not easy to tell if the problem is that the data contains { and }, that awk expects it's "program" to be within single quotes or that the FS parameter contains [ and ]
Edit - though it is the payload that the dodgy quotes are protecting, even though this particular data has no spaces.

6

Note that the input data contains 467,"used".
WIth quotes $2 is 467.
The version without quotes says $2 is 467 used.

Where does that space come from?
Where did the double quotes go?

Either the exec node or Bash interferes with the data before it gets to the script interpreter.

7

ah right - yes. Indeed escaping quotes / parameters is always fun, even with a shell script .

eg if I create a bash file a.sh

#!/bin/bash
echo $1 | awk -v FS="[:,]" '{ printf "%f%%\n", (100 * ($2 - $6) / $2) }'
exit

it will behave the same - ie if I add '' around the $1 it returns 81.156317% and 100% if not...

8

It's not a bug exactly, just that the exec node could offer more help in passing data to the external program.
It would be alleviated by an option to wrap msg.payload in [my choice of] delimiters.
In this case that would be single quotes but maybe other cases, other OSes would need different, perhaps more than one character.

Didn't set out to try and make that point, it's just sort of emerged!

9

indeed - I get your point - but I'm not sure just adding more options for users to then have to work out what they mean and if they need them and then trip themselves up with helps any more than them having to understand they need to use them in the first place - depending on their operating system environment and the comands they wish to use etc.

10

Is that the same thing? You have explicitly asked bash to split the input into fields and pass just the first one to awk.

I understand you don't want to add more options.
Maybe the help should warn that some payloads should be wrapped in single quotes?
Which is most easily accomplished with javascript or jsonata ...

11

yes - well it behaves identically - so I think it is

and I would think if you are using single quotes in your payload then you would need to wrap in double quotes... so I don't think we can say one thing. (but yes suggestions for better help appreciated)

12

How about this for help text?

On Linux, to protect " and special characters in data, wrap the string between $' and ' characters.
Embedded single quotes should also be escaped - $'{"name": "Bill\'s Bakery"}'

Explanation: Single quotes protect all special characters except for the single quote itself, hence the preceding $ and \'

I'm afraid I have no idea what the advice should be on WIndows.

I guess a change node with jsonata might be able to insert $' , escape any single quotes and add the final ' ?

[
    {
        "id": "01b34ebc9206d48d",
        "type": "change",
        "z": "db04fd79f3a93d19",
        "name": "",
        "rules": [
            {
                "t": "change",
                "p": "payload",
                "pt": "msg",
                "from": "'",
                "fromt": "str",
                "to": "\\'",
                "tot": "str"
            },
            {
                "t": "set",
                "p": "payload",
                "pt": "msg",
                "to": "\"$'\" & payload & \"'\"",
                "tot": "jsonata"
            }
        ],
        "action": "",
        "property": "",
        "from": "",
        "to": "",
        "reg": false,
        "x": 280,
        "y": 240,
        "wires": [
            [
                "fbc49f09748db0b5",
                "e18546aa9d0f06b3"
            ]
        ]
    }
]
13

I am a bit confused by this. Are you saying the commands run via an exec node need to be written differently than would the same command in a bash script file?

14

I think it is the same.

In Node-red and on the command line you can write the data to a temp file and pass the filename as a parameter. Shell expansion is not a problem since the shell doesn't see the data.

myscript mydatafile

If you pass data via the command either in NR or Bash, it needs protection from shell expansion.

Node-red could, in the spirit of language ambivalence, make the process of protecting the data easier.

NB Json data, which does need this protection because it's full of quotes and braces, is common in Node-red and rare in shell coding.

15

Then again, you could use a template node to create the full command and pass it to the exec node.

image
image1516×580 34.4 KB

Works fine ?

16

I suspect that it will just become even more confusing if node-red tries to make intelligent decisions about how stuff in the command should be quoted or escaped.

17

Thanks @bakman2, that's much more elegant. Unfortunately it doesn't cope with embedded single quotes.

@Colin I never said Node-red should make decisions.
That's enough attempts to explain. I have enough work-arounds to suit my needs.

18

Unfortunately it doesn't cope with embedded single quotes.

Example ? (note the payload here has triple { which indicates not to encode the data, make it 2 and it will encode it and still works)

19

@bakman2 Here's an example where the template you gave can't cope, the embedded single quote borks it. I think the shell confiscates "$3" too.

[{"id":"4f89b41b540d0db0","type":"tab","label":"Flow 4","disabled":false,"info":"","env":[]},{"id":"1b8352d8eb5096dd","type":"group","z":"4f89b41b540d0db0","name":"Template wraps payload in single quotes","style":{"label":true,"stroke":"#ff0000","fill":"#ffbfbf","fill-opacity":"0.36","color":"#0070c0"},"nodes":["613b06a6f86229ad","cc3379e3cb10a5cf","01af529f4db34d92","edee8c48a3483107"],"x":514,"y":191.5,"w":532,"h":129.5},{"id":"8c51f588e1254e3c","type":"group","z":"4f89b41b540d0db0","name":"Change node wraps the payload, escapes single quotes","style":{"label":true,"stroke":"#92d04f","fill":"#e3f3d3","fill-opacity":"0.34","color":"#0070c0"},"nodes":["25504e5de382ca5e","4f3e6798ddf63e6a","d1b91566aefe8ddf","72c5ee53b45939fc","5c4d405608167a6f"],"x":294,"y":31.5,"w":752,"h":129.5},{"id":"613b06a6f86229ad","type":"exec","z":"4f89b41b540d0db0","g":"1b8352d8eb5096dd","command":"","addpay":"payload","append":"","useSpawn":"false","timer":"","winHide":false,"oldrc":false,"name":"","x":790,"y":240,"wires":[["cc3379e3cb10a5cf"],["cc3379e3cb10a5cf"],[]]},{"id":"cc3379e3cb10a5cf","type":"debug","z":"4f89b41b540d0db0","g":"1b8352d8eb5096dd","name":"Errors","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":950,"y":240,"wires":[]},{"id":"01af529f4db34d92","type":"debug","z":"4f89b41b540d0db0","g":"1b8352d8eb5096dd","name":"Full command","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":620,"y":280,"wires":[]},{"id":"edee8c48a3483107","type":"template","z":"4f89b41b540d0db0","g":"1b8352d8eb5096dd","name":"","field":"payload","fieldType":"msg","format":"handlebars","syntax":"mustache","template":"echo '{{{ payload }}}'| awk -v FS='[\"]' '\n      {printf \"%s cost %s at %s\\n\", $6, $8, $4}'","output":"str","x":600,"y":240,"wires":[["613b06a6f86229ad","01af529f4db34d92"]]},{"id":"d5a9b3a83ff8b776","type":"inject","z":"4f89b41b540d0db0","name":"Inject as json string","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"{\"name\": \"Bill's Bakery\", \"Bagels\": \"$3.50\"}","payloadType":"str","x":150,"y":220,"wires":[["25504e5de382ca5e","edee8c48a3483107"]]},{"id":"25504e5de382ca5e","type":"change","z":"4f89b41b540d0db0","g":"8c51f588e1254e3c","name":"Protect Payload","rules":[{"t":"change","p":"payload","pt":"msg","from":"'","fromt":"str","to":"\\'","tot":"str"},{"t":"set","p":"payload","pt":"msg","to":"\"$'\" & payload & \"'\"","tot":"jsonata"}],"action":"","property":"","from":"","to":"","reg":false,"x":400,"y":80,"wires":[["5c4d405608167a6f"]]},{"id":"4f3e6798ddf63e6a","type":"exec","z":"4f89b41b540d0db0","g":"8c51f588e1254e3c","command":"","addpay":"payload","append":"","useSpawn":"false","timer":"","winHide":false,"oldrc":false,"name":"","x":790,"y":80,"wires":[["d1b91566aefe8ddf"],["d1b91566aefe8ddf"],[]]},{"id":"d1b91566aefe8ddf","type":"debug","z":"4f89b41b540d0db0","g":"8c51f588e1254e3c","name":"Works","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":950,"y":80,"wires":[]},{"id":"72c5ee53b45939fc","type":"debug","z":"4f89b41b540d0db0","g":"8c51f588e1254e3c","name":"Full command","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":620,"y":120,"wires":[]},{"id":"5c4d405608167a6f","type":"template","z":"4f89b41b540d0db0","g":"8c51f588e1254e3c","name":"","field":"payload","fieldType":"msg","format":"handlebars","syntax":"mustache","template":"echo {{{ payload }}}| awk -v FS='[\"]' '\n      {printf \"%s cost %s at %s\\n\", $6, $8, $4}'","output":"str","x":600,"y":80,"wires":[["4f3e6798ddf63e6a","72c5ee53b45939fc"]]},{"id":"99cc63e6af58f8f0","type":"comment","z":"4f89b41b540d0db0","name":"Required output: Bagels cost $3.50 at Bill's Bakery","info":"","x":230,"y":280,"wires":[]}]

Untitled 2
Untitled 21200×315 33.7 KB

20

What is wrong with just using a template ?

image
image768×162 10.2 KB