Formalising reusable functions with autocompletion, parameter hints, click-through-navigation

[Steve-Mcl]

Anticipating the pushback: some folk will argue this detracts from Visual Flow based Programming - and to a degree, that's a valid point. But there's already a common pattern, often suggested, of storing functions in context to write once, use many times (DRY). This prototype is an attempt to formalise that existing practice while bringing it as close to expected IDE behaviour as possible (within the existing confines of the Node-RED editor).

What...

What really bothers me about Node-RED is the cognitive load of working on larger projects where common utility functions are needed to reduce repeated code / keep things DRY. The usual pattern (stash a function in global/flow context) works, but has severe limitations and is just a pain in the you know what:

• No function signature or parameter hints when you call it elsewhere
• No "go to definition" - you have to close the editor you're in, hunt down the node that defined the function, open that, and then find your way back
• Nothing tells you what a context-stored value even looks like until you go read the source

I've been prototyping something and want some feedback.

1. A "Module" mode for the Function node. Flip a function node to Module mode and it stops processing messages (no inputs/outputs) - instead it just defines and exports plain JS via module.exports = {...}. Other Function nodes pull it in with a new node.import(idOrName) - synchronous, a live reference (not a message round-trip).

2. Real editor tooling for it - this is the part that addresses the cognitive load:

• Real autocomplete (with JSDoc) for whatever a library node exports, driven by actual TypeScript inference against its live source — not a hand-rolled scrape, so it stays accurate as the code changes
• Ctrl+click (Cmd+click on macOS) on node.import("mathlib") opens that node's own editor, stacked on top of the one you're in (closing returns you back to where you were)
• Alt+F12 / Peek Definition shows a read-only preview of the library's source without leaving your current editor

Why this pattern

I went through a few iterations before this one clicked:

RED.function.register(name, fn) inside existing code - the obvious "just add an API" move. The problem: the function code still lives buried inside whatever node's code happens to call it - there's no independent, addressable "home" for it. The editor would have to somehow parse arbitrary user code looking for register() calls and extract just the referenced function's text to do anything useful with it. And since it's not a node, there's no id, no name, no natural place for "go to definition" to land you - i'd have to invent all of that from scratch just to get back to feature parity with what nodes already give you for free.

A brand new "Function Library" node type - cleaner than the above, but it means learning a whole new node, a new palette entry, a new mental model, for something that's conceptually 95% identical to a Function node (JS sandbox, Setup/On Start tabs, Monaco editor, deploy lifecycle). Every one of those pieces already exists and already works; a new node type would mean re-solving problems the Function node solved years ago.

What made "just add a Mode to the Function node" click was realising almost everything I needed was already sitting there:

• Identity is free. A Module-mode node is still a real node - it already has an id, a name, an editor, a place on the canvas. Ctrl+click-to-navigate isn't a new feature so much as it is RED.editor.edit(RED.nodes.node(id)), a call Node-RED already makes internally whenever you click through to a referenced config node.
• The addressing model already existed. Link Call node already solves "reference another node by id or name, scoped to the current flow, refuse if ambiguous" for calling a link-in subroutine. node.import() just mirrors that exact resolution order for a synchronous, non-message-passing case. I didn't have to invent a new mental model - users who already understand Link Call already understand this.
• The code stays in a normal node property, not in a runtime value. This is the one that made the tooling possible at all. If a reusable function lives in global.set(...), its shape only exists once the flow is running - the editor has zero visibility into it at design time. Keep the function's source as a node's own editable text (exactly like a Function node's body always has been) and the editor can reason about it statically, before any deploy has ever happened - which is the whole reason real TypeScript inference and ctrl+click became possible in the first place.
• module.exports is a standard pattern. Exporting via module.exports = {...} is the same CommonJS convention JS/Node developers already know (no bespoke Node-RED API to learn). It's also why the real TypeScript inference is possible at all: TypeScript has understood CommonJS module.exports shape-inference for years, so I get accurate, JSDoc-aware types for free rather than having to invent a custom type-extraction scheme.

Demo

Click me for bigger view

[TotallyInformation]

@Steve-Mcl

Anticipating the pushback

None from me. As you know, I am a proponent for using function nodes, particularly for reducing visual overloads for complex flows.

@Steve-Mcl

detracts from Visual Flow based Programming

Actually, I would argue the reverse. As I've said previously, visual programming is great - until it isn't! That switch point is rather individual but the point is that there comes a point where the visual complexity outweighs the visual view benefits.

That also leaves aside the potential AI-based use-cases which are much easier to achieve with text-based programming than visual flows (due to the vast corpus of examples).

@Steve-Mcl

A "Module" mode for the Function node.

Sounds good to me. Not sure if Dave & Nick would prefer a separate node though on the basis of a node doing one thing? Can't say I would be unhappy with either solution.

One question though. Function nodes operate inside a VM, would module nodes operate the same way or would they be more open? I can see arguments for both and I'm not sure I have an opinion one way or the other just now but the point is important I think.

@Steve-Mcl

Real editor tooling for it

Ah! You had me excited for a minute there. I thought you were going to propose an IDE integration capability which I would LOVE to have :hint:.

---

Love the idea and the concepts.

---

Another question though. This (excellent) proposal then begs the question - can plugins provide modules that also contribute new functions with JSDoc hints to the code editor?

It would be fantastic if there were an official method for node packages to expose modules to function nodes rather than the somewhat "naughty" method of extending the RED.utils object.

It feels like node.import(...) would be the right mechanism for this as well.

[Steve-Mcl]

@TotallyInformation

Another question though. This (excellent) proposal then begs the question - can plugins provide modules that also contribute new functions with JSDoc hints to the code editor

I did explore this (in my head only) and only arrived at questions. For example: doesn't the setup->module list not already permit this? If I did support importing contrib libs the new import method, would it be targeting a node instance defined set of exports or module level stuff (I think the former, but not sure)

Any how, I have not forgotten this - it's on my phase 3 Todos

[Buckskin]

Works for me since I already use the IIFE method of creating libraries which are stored in context objects. Having the abilities in Part 2 would be magic

I love Node-RED because it allows me to write code without having to worry about inputs / outputs (MQTT) etc. so this would enhance this.

@TotallyInformation

Sounds good to me. Not sure if Dave & Nick would prefer a separate node though on the basis of a node doing one thing? Can't say I would be unhappy with either solution.

I would also go along with this.

[TotallyInformation]

@Steve-Mcl

doesn't the setup->module list not already permit this

Not sure what that is? I'm not aware of any official method for doing this but perfectly possible I've missed something.

@Steve-Mcl

would it be targeting a node instance defined set of exports or module level stuff (I think the former, but not sure)

I mentioned plugins specifically because of this. I think it only makes sense for a "global" extension to function nodes. AFAIK, there is no other officially supported way for a node package to add a fully supported and documented function/method/class/constant to a function node.

I "cheated" with UIBUILDER to make some really useful functions available and was very careful to create a unique sub-namespace on RED.util (RED.util.uib) to avoid possible namespace clashes. But an officially supported method would be super helpful.

Ah, just realised that my overrides DO seem to provide code editor smarts - I'd forgotten that:

Though I don't think that RED.util.uib provides anything, you have to know that it exists.

Even so, an officially supported method would be much better.

@Steve-Mcl

it's on my phase 3 Todos

[dceejay]

So far I am liking this. I like that you auto-change the icon and remove inputs/outputs. Some things off top of my head. What happens if you have two with same name ? Is it possible to scaffold up the basic internals (like the export statement ) for the initial user ?

Looks good so far

[gregorius]

Interestingly enough I already did exactly what your prosposing in Erlang-Red with the module node. Erlang is based much around modules and uses these as statemachines or other larger components, so it was necessary to create a module node there.

I'd would suggest using a different colour for module nodes, function and modules both being the the orange colour is confusing. Or make the symbols large, ATM the colour dominates and hence a different colour would make the different functionality clearer.

Speaking of colour, the join and split nodes should really be different colours to the change and switch nodes - a split really means "here things get confusing because many messages are being generated and being handled in - pseudo - parallel".

So in this image:

The left half is being done "singularly" while the right is "parallelised" - of course multiple messages can also occur on the left but the right is explicitly being parallelised. The other confusing thing is that the right flow might or might not complete because there might or might not be a join needed to continue - again, this isn't visually immediately apparent when using the same colours for switch, change, join, split, range, batch, filter, ....

[TotallyInformation]

@dceejay

What happens if you have two with same name ?

Since Node-RED will need to make the functions "live" when they are deployed, I would imagine that a name clash would cause an error at that point. I guess this will need to be trapped to prevent a total crash with warnings given and the deploy backed off for that node and it would be left in an error state.

Would be nice to detect the clash earlier, at coding time, but not sure how/if that would be possible.

---

This raises a secondary point, should there be some kind of listing available to make the functions more discoverable?

And a not quite connected tertiary point. It would be really nice to have a sidebar listing by node type. We have a number of explorer views:

and filters:

But I don't think we have any way of listing all of a nodes instances. Currently, you have to do a manual search and this is one of the weaker parts of the Editor in my view. This would become even more important if Steve's proposal is adopted, especially if no central list of functions is provided, since it would be the only easy way to see all module nodes together in one place.

[Steve-Mcl]

@dceejay

What happens if you have two with same name

I'm currently validating the name (a validation error occurs/name field goes red/triangle badge explains why)

This is new and I'm not 100% sure if I should introduce a different naming mechanism (but I was really hoping to keep it simple - i.e. the name is what you import)

Lets put it this way, if I don't get any complaints, I'll gladly stick with this design!

Example

@dceejay

Is it possible to scaffold up the basic internals

Absolutely.

[Steve-Mcl]

@TotallyInformation

This raises a secondary point, should there be some kind of listing available to make the functions more discoverable?

This is defiantly possible and has crossed my mind.

@TotallyInformation

and filters:

Good catch - yes.

---

But, for now, in the short term, since the imported functions behave more IDE like, it is pretty easy to search for imports and CTRL-click through. Not 100% ideal I agree, but also not totally un-findable (for want of better words)

[TotallyInformation]

@Steve-Mcl

Example

But is that the node instance name or the exported module name, sorry I can't quite read the gif. If the node only allows a single export and that is the name, then that should be fine. If it allows more complex exports then I am not sure that is sufficient.

[Steve-Mcl]

@TotallyInformation

But is that the node instance name

The node instance name.

Since module.exports is a one time thing (but can be an object) multiple exports are possible.

Consider this example:

Function Module/lib

/**
 * Adds 2 numbers then returns the result
 * @param {number} x - x parameter
 * @param {number} y - y parameter
 */
function add(x, y) {
    return x + y
}

/**
 * subtracts y from x and returns result
 * @param {number} x - x parameter
 * @param {number} y - y parameter
 */
function subtract(x, y) {
    return x - y
}

/**
 * A Maths Library with commonly used functions.
 */
module.exports = {
    add,
    subtract
}

Regular function

const { add, subtract } = node.import("math-lib")
let result = add(x, y)
result = subtract(result, z)
msg.payload = result
return msg

[TotallyInformation]

But that means that function name clashes will be very likely won’t it? Unless you create a namespace for each module node.

How will the name clashes be handled?

Ok, ignore that, I’m being slow today!

[Steve-Mcl]

If anyone is interested in taking this for a spin there is a PR here: feat: Formalise reusable functions with autocompletion, parameter hints, and click-through navigation by Steve-Mcl · Pull Request #5901 · node-red/node-red · GitHub

There is a demo flow in the PR.

To run it locally:

git clone https://github.com/node-red/node-red.git
cd node-red
git switch feat-enhanced-function-reuse
npm install
npm run dev

[Buckskin]

Yowser. I had a bunch of error messages with the install but... the Function as Module works a treat. Loaded one of my IIFE-function-library-to-context as a module and it worked.

PS. love all the different ways that a module function can be loaded, and the fact that the module does NOT have to be on the same tab.

PPS Being able to load & create new classes without all the context faff is icing on the cake

[TotallyInformation]

@Steve-Mcl

npm run dev

Been so long since I ran a default Node-RED install that I completely forgot that it would use my default ~/.node-red settings folder. I've no idea what is in that! Needless to say, it really didn't like it.

I have to say that I don't believe that the dev run should use default config?

npm run dev -- --userDir ./data

Automatically creates the data sub-folder and uses it to create a new config.

But that still fails if you are already running Node-RED on the default port of course!

Unfortunately, Windows cannot set an immediate environment variable like you can on Linux/Mac (e.g. PORT 1990 npm run dev -- --userDir ./data), instead you have to use something like: $env:PORT="3000"; npm run dev -- --userDir ./data; Remove-Item Env:\PORT The chance of remembering that is virtually NIL.

Possible request incoming for something that might fix that issue. I have some code that auto-selects a free port.

[TotallyInformation]

Quick tests - all seems to work as expected. Couple of small comments:

1. Code outside the module.exports runs at deploy time. This would need documenting I think.
2. Seems to inherit the same environment as the normal function node? So RED.util is available and even has the RED.util.uib "additions"