Jsonata, modify some elements in an array of objects

[jbudd]

Speaking of the names of debug nodes, @Colin recently said

Filter by change or x min

I imported a flow from here which bumped it up to a ridiculous value. I think node-red uses the current highest value as its base for new ones. What we need is a simple flow to renumber all the debug nodes in a flow to get it back down again. Well thats what I need anyway

And I too have "debug 2514" etc
So I thought "yes a simple flow to rename them ..."

I can get a subset of flows.json, the offending nodes, using Jsonata in a change node

payload[type = "debug"][$match(name, /debug [0-9]{2}[0-9]*/)].{ "id": id, "name": name}

I think it means

• Operate on msg.payload.
• Records where type == "debug"
• And name is "debug " + 3 or more digits. Edit - That regex is actually 2 or more digits?
• Show me the id and name (If I omit the dot and after, it returns the complete debug nodes)

Beware: if you upload this flow, you too will have debug names > 2500!

[{"id":"403d5b19c396432b","type":"inject","z":"cdc9555eb7388073","name":"go","props":[{"p":"payload"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":90,"y":460,"wires":[["1e92a64c341706ef"]]},{"id":"1e92a64c341706ef","type":"file in","z":"cdc9555eb7388073","name":"flows.json","filename":".node-red/flows.json","filenameType":"str","format":"utf8","chunk":false,"sendError":false,"encoding":"none","allProps":false,"x":220,"y":460,"wires":[["b85e1bec4d611ad4"]]},{"id":"b85e1bec4d611ad4","type":"json","z":"cdc9555eb7388073","name":"","property":"payload","action":"","pretty":false,"x":350,"y":460,"wires":[["156e8643a18d7812"]]},{"id":"156e8643a18d7812","type":"change","z":"cdc9555eb7388073","name":"","rules":[{"t":"set","p":"payload","pt":"msg","to":"payload[type = \"debug\"][$match(name, /debug [0-9]{2}[0-9]*/)].{ \"id\": id, \"name\": name}","tot":"jsonata"}],"action":"","property":"","from":"","to":"","reg":false,"x":500,"y":460,"wires":[["5a0b2209abc9f3b7"]]},{"id":"5a0b2209abc9f3b7","type":"debug","z":"cdc9555eb7388073","name":"debug 2519","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":670,"y":460,"wires":[]}]

gives me

[{"id":"2472d98c49fc78e8","name":"debug 156"},
{"id":"668298de4e1a94be","name":"debug 2514"},
{"id":"5a0b2209abc9f3b7","name":"debug 2519"}]

But how to modify the node names to "debug 0" + count?
I suspect I need to use $map() and / or $merge() but I just can't get it to work for flows.json

[Colin]

If you delete those debug nodes, deploy, restart node red and refresh the browser page then that may sort it.

[jbudd]

Yes it does but it doesn't advance my Jsonata knowledge!

[E1cid]

You definitely picked a tricky one this time

(
    $debug := $$.payload[$contains($.name, /debug \d*/)].id;
    $reduce($$.payload, function($acc, $k) { 
        $k.id in $debug ? (
            {"data": $append($acc.data,$k ~> |$|{"name": "debug " & $acc.count}|),
            "count": $acc.count + 1 }
        ) : (
            {"data": $append($acc.data,$k),
            "count": $acc.count }
        ) 
      }, {"data":[], "count":1}
    ).data
)

[edit] remove unneeded variable binding and function

[jbudd]

That certainly seems to work, I end up with "debug 1" to "debug 11".
It's going to take some study to try and understand what's going on.

Unfortunately I can't do a straightforward diff flows.json newflows.json because my flows.json has new lines and the new file doesn't.

I do have a working flow too, Jsonata to extract the iffy debug nodes and a sed command to edit the file. It feels like cheating to shell out to the OS, but of course the ability to do that is part of why Node-red is so useful.
I once tried to get the powers to consider building sed/awk/bash etc ability into the change node like Jsonata is, for situations like this. It didn't go down well!

[{"id":"c5f8b30b53cb1fda","type":"group","z":"cdc9555eb7388073","name":"Use jsonata to assemple a list of \"debug [0-9]{3}[0-9]*\" nodes and sed to edit them","style":{"label":true},"nodes":["203c41e8b6075f7c","8597d919d8d47788","951d72920008289f","1ef6f40a85ec11be","eb5bbc2b54cda6d6","152f43d6a8e4ba33","fc9a1b0790b3d6fa","3799d1023fc7c885","9887ed66d6f09245","772f0dbfdd07c24a","21b40b8cb56d2520","cc1e8a1c53da4cbe"],"x":14,"y":299,"w":812,"h":202},{"id":"203c41e8b6075f7c","type":"inject","z":"cdc9555eb7388073","g":"c5f8b30b53cb1fda","name":"go","props":[{"p":"payload"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":110,"y":340,"wires":[["8597d919d8d47788"]]},{"id":"8597d919d8d47788","type":"file in","z":"cdc9555eb7388073","g":"c5f8b30b53cb1fda","name":"flows.json","filename":".node-red/flows.json","filenameType":"str","format":"utf8","chunk":false,"sendError":false,"encoding":"none","allProps":false,"x":240,"y":340,"wires":[["951d72920008289f"]]},{"id":"951d72920008289f","type":"json","z":"cdc9555eb7388073","g":"c5f8b30b53cb1fda","name":"","property":"payload","action":"","pretty":false,"x":370,"y":340,"wires":[["1ef6f40a85ec11be"]]},{"id":"1ef6f40a85ec11be","type":"change","z":"cdc9555eb7388073","g":"c5f8b30b53cb1fda","name":"","rules":[{"t":"set","p":"payload","pt":"msg","to":"payload[type = \"debug\"][$match(name, /debug [0-9]{3}[0-9]*/)].{ \"id\": id, \"name\": name}","tot":"jsonata"}],"action":"","property":"","from":"","to":"","reg":false,"x":520,"y":340,"wires":[["152f43d6a8e4ba33","fc9a1b0790b3d6fa"]]},{"id":"eb5bbc2b54cda6d6","type":"debug","z":"cdc9555eb7388073","g":"c5f8b30b53cb1fda","name":"Result","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":730,"y":420,"wires":[]},{"id":"152f43d6a8e4ba33","type":"function","z":"cdc9555eb7388073","g":"c5f8b30b53cb1fda","name":"sed command","func":"const debugs = msg.payload\nlet cmd = \"sed -i '\"\n\nfor (var index in debugs) {\n    const nn = Number(index) + 1\n    debugs[index].newname = \"debug 0\" + nn\n    const sub = \"s/\\\"\" + debugs[index].name + \"\\\"/\\\"\" + debugs[index].newname + \"\\\"/; \"\n    cmd = cmd + sub\n}\n\ncmd += \"' .node-red/flows.json \"\nmsg.payload = cmd\nreturn msg;","outputs":1,"timeout":0,"noerr":0,"initialize":"","finalize":"","libs":[],"x":140,"y":420,"wires":[["cc1e8a1c53da4cbe"]]},{"id":"fc9a1b0790b3d6fa","type":"debug","z":"cdc9555eb7388073","g":"c5f8b30b53cb1fda","name":"debug 2523","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":710,"y":340,"wires":[]},{"id":"3799d1023fc7c885","type":"exec","z":"cdc9555eb7388073","g":"c5f8b30b53cb1fda","command":"","addpay":"payload","append":"","useSpawn":"false","timer":"","winHide":false,"oldrc":false,"name":"Run it","x":330,"y":420,"wires":[[],[],["9887ed66d6f09245"]]},{"id":"9887ed66d6f09245","type":"switch","z":"cdc9555eb7388073","g":"c5f8b30b53cb1fda","name":"rc == 0?","property":"payload.code","propertyType":"msg","rules":[{"t":"eq","v":"0","vt":"num"},{"t":"else"}],"checkall":"true","repair":false,"outputs":2,"x":460,"y":420,"wires":[["772f0dbfdd07c24a"],["21b40b8cb56d2520"]]},{"id":"772f0dbfdd07c24a","type":"change","z":"cdc9555eb7388073","g":"c5f8b30b53cb1fda","name":"Restart","rules":[{"t":"set","p":"payload","pt":"msg","to":"Now restart Node-Red and reload the page","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":600,"y":400,"wires":[["eb5bbc2b54cda6d6"]]},{"id":"21b40b8cb56d2520","type":"change","z":"cdc9555eb7388073","g":"c5f8b30b53cb1fda","name":"Error","rules":[{"t":"set","p":"payload","pt":"msg","to":"Something went wrong!","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":590,"y":440,"wires":[["eb5bbc2b54cda6d6"]]},{"id":"cc1e8a1c53da4cbe","type":"debug","z":"cdc9555eb7388073","g":"c5f8b30b53cb1fda","name":"debug 2525","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":270,"y":460,"wires":[]}]

[E1cid]

@jbudd

Unfortunately I can't do a straightforward diff flows.json newflows.json because my flows.json has new lines and the new file doesn't.

Surely you can pass the JS object through a JSON node and check format JSON string to add newlines/format.

Or use the $string(object, true) to format the object as a formatted JSON string.
e.g.

(
    $debug := $$.payload[$contains($.name, /debug \d*/)].id;
    $string(
        $reduce($$.payload, function($acc, $k) { 
            $k.id in $debug ? (
                {"data": $append($acc.data,$k ~> |$|{"name": "debug " & $acc.count}|),
                "count": $acc.count + 1 }
            ) : (
                {"data": $append($acc.data,$k),
                "count": $acc.count }
            ) 
            }, {"data":[], "count":1}
        ).data,
        true
    )
)

[jbudd]

@E1cid

Surely you can pass the JS object through a JSON node and check format JSON string to add newlines/format.

Umm. I certainly didn't think of doing this, but actually I still seem to get everything on a single line.

@E1cid

Or use the $string(object, true) ...

A diff command after this gives me

pi@zerotwogreen:~ $ diff -w  .node-red/flows.json newflows
95c95
<         "name": "Jsonata get \"debug nnn+\" nodes",
---
>     "name": "debug 1",
115c115
<         "name": "Use jsonata to assemple a list of \"debug [0-9]{3}[0-9]*\" nodes and sed to edit them",
---
>     "name": "debug 2",
142c142
<         "name": "Get list of debug nodes without jsonata",
---
>     "name": "debug 3",
1275c1275
<         "name": "debug 1",
---
>     "name": "debug 4",
1923c1923
<         "name": "debug 156",
---
>     "name": "debug 5",
2034c2034
<         "name": "debug 2514",
---
>     "name": "debug 6",
etc

Which shows it's a bit overeager. The first three amendments are actually groups which happen to contain "debug" in the name.
On the other hand, it drew my attention to a typo in "assemple"

I'm tremendously impressed so far

A simple tweek to the regex in your first line, \d+ rather than \d* eliminates those three substitutions.

$debug := $.payload[$contains($.name, /debug \d+/)].id;

[E1cid]

Looked at this again this morning, with fresh eyes.
This also works and seems simple and less intensive

(
    $debug := $$.payload[$contains($.name, /debug \d+/)].id;
    $$.payload ~> |$| 
        $.id in $debug ? (
            {
                "name": "debug " & $map($debug, function($v, $i){
                    $.id = $v ? $i + 1
                })
            }
        ) 
    | ~> $string(true)
)
        

[shrickus]

If you don't mind having "holes" in the debug numbering sequence, you could just use the index of the node inside the flow array -- makes for a much simpler JSONata expression, using the transform operator:

payload#$idx.(
    $ ~> | $[type='debug'] | { "name": name.$replace(/^debug (\d+)$/, 'debug ' & $idx) } |
)

Think of the transform as an inline stream editor (i.e. sed) that modifies data inside the stream but leaves the outer structure intact -- essentially the syntax says this:

array of objs to transform ~> | obj to be changed | { fld1: 'val1', fld2: 'val2', etc.} |

(Notice that I have also used a non-greedy regex that only replaces names matching the default naming scheme)

Of course, this does not really help you if you have 1000's of nodes in your flows...

[shrickus]

Actually, I thought of a better way to do it without using the index... first build a list of just the debug nodes to be changed, and create a lookup table of the { id: 'debug #' }

Then just transform the entire payload in-place, substituting the new names from the lookup table, if found:

(
    $dbgNodeIds := payload[type='debug' and name.$match(/^debug (\d+)$/)]#$idx.{id: 'debug ' & ($idx+1)};
    payload ~> | $ | { "name": $lookup($dbgNodeIds, id) ? $lookup($dbgNodeIds, id) : name } |
)

[E1cid]

I don't think the ternary after the lookup() is required

payload ~> | $ | { "name": $lookup($dbgNodeIds, id)  } |

Should be all that is required.

[jbudd]

@shrickus

If you don't mind having "holes" in the debug numbering sequence, you could just use the index of the node inside the flow array

Oh no, that would never do!

I am overwhelmed. No, honestly!

Going to have to insist on a blow by blow explanation, like this:

@shrickus

Think of the transform as an inline stream editor (i.e. sed) that modifies data inside the stream but leaves the outer structure intact -- essentially the syntax says this:

array of objs to transform ~> | obj to be changed | { fld1: 'val1', fld2: 'val2', etc.} |

I started by 00Going Pascal assignment operator in Jsonata (without success) but then | $ | grabbed my eye. That can't mean anything, surely?

I'm playing with try.jsonata.org but every time I think I've grasped a concept and use it to build an expression it gives me a bloody nose.

Thanks for all the contributions!

[shrickus]

Oh right -- If the $lookup returns nothing, then the object name is not going to be transformed, is it? Good catch!

[shrickus]

@jbudd

Going to have to insist on a blow by blow explanation

How about this => https://try.jsonata.org/PW05fSKqW

(                                               /* start of compound expression */
    $dbgNames :=                                /* var to hold lookup table */
        payload[
            type='debug'                        /* filter for just debug nodes ... */
            and                                 /* ... and ... */
            name.$match(/^debug (\d+)$/)        /* names like "debug ####" */
        ]
        #$idx                                   /* var to hold the array index number */
            .{                                  /* map operator - create a new obj for each node */
                id: 'debug ' & ($idx+1)         /* key = node "id", value = indexed debug name */
            };

    payload ~>                                  /* Now transform the entire flow array of nodes */
        |                                       /* start of transform operator */
            $                                   /* set the "context" of the following expr to each node */
        | 
            {                                   /* object with fields to replace the context object fields */
                "name": $lookup($dbgNames, id)  /* lookup the new name -- if not found, nothing is changed */
            }
            ,[ ]                                /* optional list of fields names to be removed (fyi) */
        |                                       /* end of transform operator */
)                                               /* end of compound expression */