Apply transformation on nested object fields (based on dot notation)

Developing Nodes
1

Hi folks,

I am searching for a way to transform easily nested object properties, for one of my new nodes.

For example the input message of my node could be like this:

{
   "payload": [
    {
      "someProperty": "A",
      "someOtherProperty": "B"
    },
    {
      "someProperty": "C",
      "someOtherProperty": "D"
    }
  ]
}

On the config screen, the user should be able to specify which fields need to be transformed. For example (but other syntax might also do the job...):

  • "payload[1].someProperty means that the transformation only needs to be applied to the second element of the array. Which means that value "A" needs to be transformed to "E":
    {
   "payload": [
    {
      "someProperty": "E",
      "someOtherProperty": "B"
    },
    {
      "someProperty": "C",
      "someOtherProperty": "D"
    }
  ]
}
  • "payload[].someOtherProperty means the transformation needs to be applied to all the elements in the array. Which means value "B" and "D" both need to be transformed to "G" and "H":
    {
   "payload": [
    {
      "someProperty": "A",
      "someOtherProperty": "G"
    },
    {
      "someProperty": "C",
      "someOtherProperty": "H"
    }
  ]
}

I need to support multiple levels of nesting, because the property (that needs to be transformed) can be anywhere inside the complex input message object. The input can by any kind of object ...

Have been looking at many libraries on NPM, but can't find what I need. I just want to specify on which nested fields of an object my transformation function needs to be executed.

Has anybody seen a library that can accomplish like that?

Thanks !!!!
Bart

2

I'd start with jsonata given that is built-in to Node-RED and is a library for transforming JSON structured data.

3

Hey Nick,
Yes that indeed has popped up in my head already. So I had a quick look at jsonata yesterday. But I have to be honest that it is ages ago that I used jsonata, and now it has become one big cryptogram for me ...

I have in my config screen:

  1. A dropdown list containing 4 transformations, from which the user has to choose.
  2. A input field where the user can specify on which fields (in the input message) the selected transformation has to be executed.

But it is not clear to me how to accomplish something like that with jsonata...

4

Ah I now see that Jsonata has a ~> (transform) operator. So I assume I need to use that one ...

5

I'm very interested in how you solve this. In one of my nodes I needed to find a particular key-value pair buried in an object of arbitrary structure. I was able to do it (barely) using the JS Object.getOwnPropertyNames() function and the NR RED.util.compareObjects() function, but I am sure that there has to be a better way using JSONata.

[EDIT] It just dawned on me that @BartButenaers and I are talking about custom nodes. @knolleary, is it possible to use JASONata in the JS file of such a node?

6

Hey @drmibell,

Well you can use jsonata fairly easily in a custom node (see here).

However I am not sure whether I can use it to transform an object via a custom transformation call back function. I would like to have Jsonata to call my transformation callback function, for all the object properties that match the dot pattern. Because I want to have the transformation logic in my Javascript cod, and not inside the Jsonata expression.

P.S. Long long time ago I have asked Andrew Coleman whether he could add json manipulation to jsonata. In that discussion you will see that a.o. @shrickus were discussing such a callback function (for transform/replace). But it is not clear to me at the moment whether it has been implemented or not...

7

Thanks for that. My code is working now, but I'll give JSONata a try just to see if I can expand my toolbox. If I can keep track of what is a JS object and what is a JSON string, it might be fun.

8

Then you might also have a quick look at their Javascript API. I see that they also offer a registerFunction, so that might be what I am looking for. Will try it...

9

That seems to do the job. My callback function is nicely called with the required input, and my output is used by Jsonata. Case closed :+1:

2 Likes
10

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

12

Hi @drmibell,

I got it working using the Jsonata transforms. And it works fine!

However few days afterwards my Node-RED became suddenly completely unresponsive, and I had no clue what was causing that. So I started experimenting with profiling in Node-RED (via ClinicJs, my message tracer and my own profiler), and via all ways it immediately became visible that my Jsonata transforms were causing this:

image

Jsonata does not modify (see here) the original input objects, instead it deep clones the input msg and it transforms the clone and returns that modified clone.

However my new custom node - for which I need tranformations - is used to handle images from camera streams, so those messages contain LOTs of data. The image cloning was causing all my troubles, because that is something that need to be avoided as much as possible when dealing with images.

So I ended up writing my own custom transformation (with my own expression syntax) at the end, and that works MUCH faster because I now transform the original input message ...

Summarized: the Jsonata transforms work very well, but be aware about the performance impact! It all depends on the amount of data in your input messages...

1 Like