Defining utility functions for re-use in a function node

Hi all, I am well aware there are a lot of threads that ask a variation of "How can I write a utility function and call it from other functions. e.g:

I have had some ideas floating around for some time (I've even written code for some of them) but I've always felt they lack a certain something I dont know what!

Today felt like a good day to explore some solutions (old and new)

I would appreciate your feedback / thoughts.

Please note: There are many ways this could be approached and a ton of ways it could be better, but in the spirit of achieving the "re-usable" aspect and limited time (for NR 5), please try to stay on topic :smiley:


Function Node - ideas and thoughts around shared function definitions

Story

As a Node-RED user, I want to define reusable logic that can be reused across multiple function nodes, so that I can avoid code duplication and maintain consistency.
Implementing a way to define global functions would streamline this process and enhance code maintainability. Typically we suggest the user does this visually via link and link-call nodes (creating a subroutine), or by adding their functions to global context in a function node's on-start script or by adding code to the users settings.js file. However these approaches have limitations in terms of practicality/usability (if I am in a function, I typically want to stay in the function), maintainability (logic defined in a file or a random function somewhere not obvious is not ideal), reusability, and FBP familiarity / visibility.

Implementation Ideas

  1. Create a new configuration node type for global functions, allowing users to define and manage them in a centralized location.
  2. Modify the Function node to permit registration and usage of these global functions.
  3. Add a RED.functions (or similar) API to facilitate the registration and calling of global functions.
  4. A RED.util.linkcall (or similar) API to facilitate calling link-in→logic→link-return subroutines programmatically from within function nodes.

1 - new configuration node type

  • Users can create a "Global Function" config node where they can define functions.
  • Function nodes can reference these global functions by name.
Pro
  • Centralized management of these utility functions.
  • Possibility of presenting the callable with typedoc defs (intellisense)
Con
  • Somewhat additional complexity in managing config nodes
  • New node type to create
  • Not backwards compatible
  • Chance of name collisions of function definitions?

2 - Modify Function node

Extend the Function node's setup to allow users to make the function node a place where a global function can be defined (e.g.: Function Scope "Normal" / "Global")

Pro
  • Easy to use (familiar node-edit panel)
  • Visual
  • Could be backwards compatible (if we save the config of the node in the flow as if it were a standard function node and utilise the on-start function to register the functions in global context - anyone loading an old flow would see a disconnected function with some onstart code adding a utility function to global context)
Con
  • Mode based UI to show/hide parts of the function node
  • Disconnected / isolated function nodes feel odd.
  • Only function nodes can be used

3 - RED.functions API

Introduce a new API within Node-RED (e.g., RED.functions) that allows users to register global functions programmatically. Something like RED.functions.register('myFunction', typedoc) then other functions can call RED.functions.myfunction

Pro
  • No need to modify existing nodes or introduce new config type.
  • Slightly simpler implementation.
  • Could potentially provide function node type hints (intellisense)
Con
  • Less user-friendly for non-developers - but then users requesting this kind of functionality are typically not beginners!
  • not visual
  • not backwards compatible (though a patch plugin might be possible to add this for older Node-RED versions)

4 - RED.util.linkcall API

  • Introduce a new API within Node-RED (e.g., RED.util.linkcall) that allows users to call visually defined subroutines from a function node.
Pro
  • No need to modify existing nodes or introduce new config type.
  • Simpler implementation.
  • leverages existing link-call nodes for reusability
  • benefits from visual aspect of link-call nodes
  • benefits from supporting more than just functions (core and contrib nodes can become re-usable logic too) - this was a surprising (but ultimately obvious) benefit!
Con
  • Less user-friendly for non-developers - but then users requesting this kind of functionality are typically not beginners!
  • not backwards compatible however a patch release on 4.x stream would be relatively simple.
    • failure mode mode in NR <=3.x would be a regular node error complaining about RED.util.linkcall being undefined (or similar)

Decision Matrix

:grinning_face:=5=best/easiest :smiling_face:=4=good/not too difficult, :neutral_face:=3=neutral, :slightly_frowning_face:=2=worse/hard :sob:=1=complex/worst/poor

Approach Implementation User Friendliness Maintainability Backwards Compatibility Visual Aspect Benefits Score
1 - New Config Node :sob: :neutral_face: :neutral_face: :slightly_frowning_face: :neutral_face: :neutral_face: 15
2 - Modify Function Node :slightly_frowning_face: :smiling_face: :smiling_face: :smiling_face: :smiling_face: :neutral_face: 21
3 - RED.functions API :smiling_face: :neutral_face: :smiling_face: :neutral_face: :neutral_face: :neutral_face: 20
4 - RED.util.linkcall API :grinning_face: :neutral_face: :smiling_face: :neutral_face: :smiling_face: :grinning_face: 24

So, based on the result of that​:backhand_index_pointing_up:, I did a POC.

And, you know what - I dont hate it!

I thought I would, but no, it just works ™️and I havent had an ah but moment either - which leads me here.

The API

        /**
         * Utility function to call a reusable flow defined as a subroutine (link-in/link-out nodes).
         * 
         * @param {string} target - the target of the link-in subroutine to call (can be node ID or name of link-in node)
         * @param {Object} msg - the message object to pass to the subroutine
         * @param {Object} [options] - call options
         * @param {number} [options.timeout=5000] - the maximum time to wait for a response (default: 5000ms)
         * @return {Promise<Object>} - resolves with the returned message
         * @memberof @node-red/util_util
         */
        function linkcall(target: string, msg: object, options?: object): Promise<object>;

Example code in a function:

// get the outside temprature, send it to an AI, return YES or NO
const msgTemp = (await RED.util.linkcall('get-temperature', msg))
const m2 = {
    payload: `Outside temperature is ${msgTemp.payload}C. Should I turn heating on? (answer YES or NO only)`
}
msg.payload = (await RED.util.linkcall('ask-my-ai', m2, { timeout: 30000 })).payload
return msg

Testing: behaviour of function invocation vs existing link-call

Sequential

chrome_s95z51aEWX

Nested calls

Missing/bad target

Missing link-return


Potential future improvements:

  • dynamic typing - present the list of targets in the intelligence as an enum choice for the target parameter
  • Support overloads to simplify calling
    • e.g. RED.util.linkcall('target-name', "a string for payload for simplicity")
    • e.g. RED.util.linkcall('target-name') // simple fire and forget (no params, dont timeout, dont care about reply msg)
8 Likes

Interesting.

I think it would be useful to consider one additional use-case though. That is: custom nodes that want to register utility functions.

Your example I think is fine on the visual side of things and suited to where flow developers want to add utility functions.

However, it does not feel well suited where node developers want to add common functions.

So I'm not saying that your choice doesn't have merit, it clearly does. But I would like to see a solution for the other use-case that is more in keeping for people who are developing nodes.

It feels to me as though 4 is worth pursuing but that 3 (which feels as though it should be fairly easy to implement?) would also be worth while.

2 Likes

I can see the use of option 4, but I have around 50 utility functions that I currently have in a global via IIFE that runs at flow startup so this would get messy unless there was a way of picking a function in the link.

That is why I like option 3 as it gives the same kind of access. As for cons

  • Not visual - direct observation in the function node as to what is happening (similar to require)
  • Not backwards compatible - would someone really be using this in an old version of NR?

Having thought about it a bit it is possible to select a function using option 4 but it is not pretty

1 Like

Great write up Steve.

I like the idea of being able to call link calls in a function node(4), would make for clearer flows and reusing functions and other nodes direct in the function node. But as other have stated I think being able to add functions with a RED.api (3) has great merit also. So I would like to see both for 3 and 4.

2 Likes

I have to agree with the others, while 4 has it's merits, it doesn't work for "developers" that have many functions. Option 3 would be a better fit.

For people like myself and @Buckskin, options 1 and 2 would be a non-starter I think.

To make it user friendly, perhaps in conjunction with #3:

in that 'functions' section, have an overview of defined functions, where the user can add/modify/delete. This could perhaps also be used for installing external modules, removing the need for the 'setup' tab in function nodes (thinking out loud)

3 Likes

#1 is a half baked idea that I didn't really word well. I did mean centralised management (I said it but called it a config for ease) and would probably have deferred to this design once if I put more than 5 mins thought into it :wink:

The issue with the config / global settings version of this story is the amount of UI work and other scaffolding (like import/export, search/discoverability, where to store and instantiate functions) (its not a quick win)


Well, yes, of course you could have a single link-flow-subroutine with many functions - for example could simply pass the function name and args in the msg

chrome_pvz6MU8sbi

A little clunkier but would be possible with #4.


It seems Option 3 is winning more hearts but I fear they might be for the same selfish reasons I like options 3 (its just easier to set-up a bunch of functions in a code block), but, Option 3 alone would provide a lot less benefits that Option 4.

I understand (as a programmer and lazy person (the best kinda lazy)) why #3 is more attractive but it is ultimately a bit harder to implement and a fair bit more restrictive than #4. So, perhaps a combination of the 2 solutions is the answer!. An API that not only lets you call linkTargets but declare reusable functions too. That way, when you need to call upon the functionality of a contrib node, you can. When you need a simple utility function, you get that too.

That said, I think both #4 and #3 are achievable (#4 obviously since I have a working PoC) but #3 was something I wrote a few months back. The hardest parts were the API naming and parameters (for picking up function signatures/jsdoc for monaco), presenting the generated functions as real things in the monaco editor dynamically and where they live/execute (I chose global context) but was never truly happy with it - too many what-if edge cases - something that didnt crop up when doing #4!

I will let it swish around my head for a few days (and see if there is any other feedback in the coming days).

This is not about being selfish or lazy - from my perspective anyway.

I've already abused RED.util for uibuilder to expose some useful functions to function nodes. Others have said similar things. There is clearly a need here. I would hope for something like a RED.fns object to be exposed where nodes can declare a unique property root, for example RED.fns.uibuilder for uibuilder. That root would allow functions or perhaps also constants to be added at least by custom nodes.

Ideally, there would also be a method for non-node authors to also similarly add a collection of functions.

For safety, it might be wise to add a switch/flag to settings.js to allow turning the feature off though.

If you want a REALLY easy way to enable this just for node developers, you could simply make it official that nodes can use RED.util for adding custom functions. With, of course, suitable warnings about name collisions.

Not sure about harder, since everything is node.js under the skin, I think this should be fairly easy. But yes, it should be more restrictive which is why most of us are also saying that option 4 should also be considered in parallel.

Which is great for visual developers but naff for node developers.

Well, this would be fantastic to have. But, not really necessary for a first release. So if time/resources are short, maybe this part should be considered for v5.1?

Off the top of my head, I would agree. I don't think that global context would be good for #3, it is specifically for function nodes isn't it? So it should be a function node API. It really doesn't make sense to expose #3 to anything else I don't think?

Not sure if this is relevant to the topic?

From the point of view of a non-native javacript speaker:

I know that if I want to include an extra library in a function node, I can include the library in the On Setup tab.

It seems to be a perfectly adequate mechanism to access extra functions.
I would not welcome a distinct way to include more just because I wrote them myself.

Briefly I used the functions defined in global context approach, but I abandoned it as too unwieldy.

I don't care how it works in the background but I think the code should be included in the function explicitly rather than globally, and in exactly the same way as I currently include eg the chroma-js library.

I would agree with that.

Steve's solution to using option 4 was better than mine (giant switch statement) but I have thought of another way: keep the library as an IIFE and return the whole thing as a msg property which can then be collected as part of a link call. (individual functions can also be returned)

Non functional example (no functions include)

const utility = (function () {
    'use strict';

    /*
    *   Static Properties
    *       libraryVersion                          v 2.2
    *
    *   Functions Available
    *
    *       fixedDecimals,
    *       roundToHalf,                            v 2.1
    *       toSentenceCase,
    *       toTitleCase,
    *       toCamelCase,
    *       camelCaseToText,
    *       dewPoint                                v 2.2
    *
    */

    const libraryVersion = 2.2
.
.
.
.

    return {

        libraryVersion,
        fixedDecimals,
        roundToHalf,
        toSentenceCase,
        toTitleCase,
        toCamelCase,
        camelCaseToText,
        dewPoint,

    }

}())

msg.utility = utility
msg.utilityFunction = utility[msg.payload]

return msg

Unfortunately, I think that approach is somewhat fragile because functions cannot be serialized. It may work but there might be unforeseen circumstances where it would not.

Possibly, what sort of circumstances were you thinking of, (I have to add that this is not my preferred approach which is option 3)

At the moment I recreate the library global whenever the first flow starts because, as you say, a function cannot be serialised and therefore cannot be saved in a file. However, with the approach posited each time the link call runs the IIFE function recreates the entire library so it never has to be saved. The only downside is that the library functions ARE recreated each time the link call runs.

Just throwing my hat in the ring to describe what I do. It works great!

  1. In my Start Run subflow, I have a Lib node that defines an object full of the shared functions I want available everywhere that gets saved to global context as lib, if not already present.
  2. Soon afterward, I hoist that context to msg.lib for direct use in the rest of the flow.
  3. Calling happens like this: const myDataBlob = msg.lib.data.getCurrenBlob(msg); I often send in msg so the given lib method will have the data context it needs to do stuff. For utilities, often it helps to send in node as well so you can do stuff with the caller node’s context.
1 Like

IMHO, it is possible to pass functions as msg properties/return values in flows, without any need for serialization.
A node is actually a JS function (registered dynamically into the Node-red runtime). "Sending a message" to a node is actually calling that node directly (with the parameter msg), which then invokes a callback handler, all in the same memory space.

This will not work for dashboard nodes (widgets) where msg does need to be serialized when sent/received over a socket. But even here it is easy to serialize functions, as I do in my @omrid01/node-red-dashboard-2-table-tabulator dashboard node, where custom callback functions can be defined & sent as metadata objects (with name, parameters & body), and instantiated at the client side e.g. myFunc = new Function(<metadata>).

All I was doing was pointing out that there may be edge-cases to take into account. For example, people often forget that, while you can put a function into a context variable - you can only do so if you are using the in-memory based context. These things are easy to forget but I don't have enough knowledge of the deep workings of the Node-RED core to be able to say whether or not this would need dealing with in this instance. Just that some measure of care is needed.