Revisiting custom functions in Jsonata

Core Development Feature Requests
1

New to Node Red and I like several people are interested in the ability to create custom functions in Jsonata. I know this has been brought up before and died every time because people could not see how to really implement that. I think i may have a fresh take on this and perhaps a path that could get something implemented.

I think the issue is perhaps that what we need is not custom "functions", as in full blown JS functions, but what we really need is the ability to invoke named Jsonata expressions.

So my idea is that in addition to the other types of environment variables you can define, you can also define environment variables that are Jsonata expressions. You then provide a way to reference and invoke those environment variable expressions passing the parameters into the Jsonata expression as an array.

For example, NR in addition to the other functions it defines (like global, env) defines another function which we might name call. The first parameter to call is the name of an environment variable, which would need to be of type Jsonata expression. It invokes that Jsonata expression passing to it an array of the additional parameters.

So if I had an environment variable named foo that was a Jsonata expression, within some other Jsonata expression I could have an expression like this:

$call("foo", "a", 1, true)

The Jsonata expession in foo would be evaluated, passing to it this array of data:

[
    "a", 1, true
]

This seems to me that it would not be that hard to implement. The key part of it are:

  • Allow environment variables to be Jsonata expressions
  • Add register the call function in Jsonata to get the environment variable and prepare and execute that function with the parameters given.
2

but what we really need is the ability to invoke named Jsonata expressions.

Can you elaborate on why "we really need" it ?
Note that jsonata is separate library, developed by other developers.

3

Why not just install it as an external module for use in the function node?

4

Because you will have common expressions that may be complex that you need to reuse in multiple places and you only want to define that once and not copy paste it over and over again.

There have been multiple requests for it in the past, so there is some demand. See e.g.:

The difference in my proposal is that instead of creating custom full blown JS functions you are just creating the ability to invoke Jsonata expressions stored in environment variables which is much easier to accomplish.

If you want a use case from what I am trying to do, I have the need to extract a date string like 06/28/2022 from a payload string based on a regex and convert that into either a string like 2022-06 or 2022-06-28 depending on the use case. In what I am doing I will need that conversion in dozens of places.

Here is the Jsonata to do that in one case:

$fromMillis($toMillis($match(payload, /Period Ending:\s+(\d{2}\/\d{2}\/\d{4})/, 1).groups[0],"[M01]/[D01]/[Y0001]"),"[Y0001]-[M01]-[D01]")

I could copy and paste that code over and over again, editing the regex in each case, but imagine instead if I could instead define an environment variable to hide the dirty parts of that and instead I can just invoke this:

$call("getFullDate", payload, /Period Ending:\s+(\d{2}/\d{2}/\d{4})/)

And what I am suggesting does not involve any changes whatsoever to Jsonata, so that doesn't matter. NR currently registers 5 functions to make them callable from Jsonata, e.g. flowContext, which you can see in the: node-red/packages/node_modules/@node-red/util/lib/util.js at master · node-red/node-red · GitHub. My proposal simply adds a 6th one named call.

5

Could this not be done by storing JSONata expressions in variables/env, then just using JSONata $eval($env("name"), context) to run them

2 Likes
6

Why allow Jsonata at all, since everything can be done in a function node? This sounds like the proverbial if the only tool you have is a hammer, every problem is a nail.

Note not every person is a node JS developer (I certainly am not and not even a a fan of JS), but JSonata is much simpler and acessible to a wider audience.

7

Thank you, that is exactly what I was looking for! I didn't know about the eval function. My proposed call function was literally shorthand for that.

So most of my proposal is moot then as I can just use $eval. But it would certainly be nice if the environment variables editor allowed you to use Jsonata as the type of the environment variable.

1 Like
8

:thinking:

I have been programming for more years than I care to mention - and I just don't understand the fascination. It don't find JSONata simple, nor intuitive and it has been demonstrated many times how much slower than JS it can be.

Not to mention, JS has more developers than any almost any other language - so a lot of talent around to lean on when you get stuck. JSONata? I doubt there are more than a few thousand active users. According to webrate, Jsonata.org traffic volume is 131 unique daily visitors, Mozilla.org (MDN) traffic volume is 2,533,487 unique daily visitors. Even Nodered.org traffic volume is 3,083 unique daily visitors

Dont get me wrong, I use JSONata for the simple tasks that it is good at. Anything more (like calling functions or sorting etc) - i'm outa there in a flash - far to finicky

Mind - I do like your use of proverb "if the only tool you have is a hammer, every problem is a nail" however, I put to you that JSONata is a kiddie safe rubber mallot and JS is a scalpel :wink:

3 Likes
9

Jsonata reminds me of python, for basic things it all makes perfect sense, until it doesn't :wink:

you are just creating the ability to invoke Jsonata expressions stored in environment variables which is much easier to accomplish.

Note that you can store complete js functions in context as well and re-use them.

1 Like
10

Haha, I think that applies to ALL programming languages - I've certainly used dozens over the years and they all fall short in different areas.

Like other "meta" languages, JSONata is really useful for a relatively limited number of tasks but when the logic is reasonably obvious, it can be very succinct. Reshaping JavaScript Objects is the obvious thing. something that can be very long-winded in JavaScript itself. But like its XML forebears (XLST), JSONata is very hard to get your head around in many more complex cases.

I've always said that I might give JSONata 5-10 minutes and if I can't work out the logic by then, it is probably going to take me hours or days and I'd be better off with JavaScript.

If course, the suggestion here certainly has merit in that it would be possible to build a library of useful transformations and that might make it worth-while to invest the time in working them out. But on the other had, would it? Because if the logic isn't obvious, it may still be better to build a library of JavaScript functions instead.

Hmmm :thinking: - no, I'm no nearer working out what is best - sorry.

1 Like
11

I see what you mean. However, I just find that the function node is the powerhouse of Node-RED.

https://flows.nodered.org/flow/29fd01f8a62fec86d875ecbd68001cb0
This wont do what you want, but it might be a good starting point if you want to learn how to use JSONata in a function node, or if you just want a bunch of pre-coded examples of the library of JSONata functions. (these are not nodes that install to the palette, I just keep a flow that I can copy and paste them from)

12

I'm certainly not looking at Jsonata for its power and expressiveness. I think where Jsonata is preferable in NR is that it can be used much more simply and cleanly as it can be applied anywhere without creating yet another node in the flow.

What would really be ideal is if in the places that you can use Jsonata (e.g. when specifying a value for a rule in a change node) that you could say you wanted to apply a JS function and then you could edit the function code right there without having to add a function node to the flow, but maybe others don't mind the extra nodes.

So regarding the plan of using Jsonata snippets in environment variables I ran into a snag with that. I did have one flow and broke code down into subflows, and putting the Jsonata snippet in the flow made it available to the subflows. I then felt that putting code in a subflow was abusing subflows and instead decided to use multiple flows and links between them instead. This then meant that I had no good place to define the snippet and make it visible to all flows.

There is no concept of setting global environment variables within the NR UI that I can find. I know you can set them in settings.js, but that is not acceptable for this case, because it requires going to the file system and not using the UI and because settings.js is not something that is versioned in a project.

I think I am going to go with a subflow to define the reusable functionality with perhaps a function node inside.

13

There is global context where you can set & retrieve anything you like (including Js functions for reusibility)

My recomendation would be unless you need to store contextual data, use link-call instead. These are more akin to PURE functions

14

I know and considered it but did not feel like the right abstraction to say I want to store some Jsonata expressions for later use. For one I would have to put a node somewhere to initialize the context with these expressions.

Can you elaborate on this as I don't see the connection to "contextual data"? What I like here about the subflow is the ability to define parameters of the subflow in its environment variables that can be set in the properties of the subflow instance rather than having to pass everything in the message itself. And in this case though the subflow will be simple it will be used dozens of times across different branches each with a different parameter in each case (a regex pattern) which is essentially constant for that instance and not dependent on the message.

1 Like
15

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.