Introducing node-red-utils

Developing Nodes
1

I've been writing custom nodes for my own projects for a couple of years, and kept copying the same chunks of code. Every node that did real work needed a worker thread, most of them needed the editor side to talk to the runtime side, and half of them needed an endpoint to serve a script to the editor. I got tired of that and pulled it all into one re-usable package. I'm putting it out here in case my structure is useful to anyone else

It's on npm as @nrutils/node-red-utils and the source is at GitHub - UpDownMe733/node-red-utils: Mixin toolkit for authors of Node-RED custom nodes: editor messaging, worker threads, popups, singletons, resource serving, pub/sub and optional GPU compute. · GitHub
(Disclaimer: the readme's are a lot of AI slop)

So you know what you're looking at, it's not a node, and it doesn't do anything you can't do already. It's a different way to lay out the code, and a way to use some of the harder bits of the Node-RED API (worker threads, admin endpoints, editor-to-runtime messaging) the same way every time instead of reinventing it per node.

You could write node's with this shape without my library, today. You would just need the base class.

With AI you don't really need any library but this gives it ways to do things consistently like worker comms and front-end comms. I'm sure as AI gets better none of this will matter.

The basic shape

You use the base class and declare what functions to use in your node in an array param.

const { NodeRedNodeBase, nodeRedEditorComms } = require('@nrutils/node-red-utils');

module.exports = NodeRedNodeBase.createNode('counter', [nodeRedEditorComms], (RED, thisNode, config) => {
    let count = 0;
    const show = () => thisNode.status({ text: `${count} messages` });

    // the button in counter.html sends this
    thisNode.nodeRedEditorComms.subscribe('reset', () => {
        count = 0;
        show();
    });

    thisNode.on('input', (msg, send, done) => {
        count++;
        show();
        send(msg);
        done();
    });
});

Each feature is a mixin. If you put it in the array it's on thisNode, and if you leave it out it isn't. Writing your own mixin is a small function, so if you have your own "I always need this" chunk of code, it fits in the same way.

The mixins I use

  • nodeRedWorker runs your node's work in a worker thread. The part I like is that if two worker nodes are wired together in the flow, the messages go from thread to thread without bouncing through the main event loop. That's been handy for anything CPU heavy.
  • nodeRedEditorComms is pub/sub between the node in the editor and the same node in the runtime. The example above uses it for a reset button. You don't write an admin endpoint and you don't poll.
  • nodeRedPubsubBus is pub/sub between nodes in the runtime without wires. I use it for the stuff that isn't really a message flow, like config changes and shared state.
  • nodeRedResourceEndpoint serves your scripts, CSS and images to the editor.
  • nodeRedPopup, nodeRedSingleton and nodeRedGpu (Vulkan compute from a worker) work, but I've marked them experimental because I'm the only one who has used them.

The README has a full counter example and a worker example, and a few snippets here and there. I didn't include full worked nodes bc that's not what this is.

Expectations

I'm one person doing this on the side. I'll keep up with issues and PRs for as long as it holds my interest, which, going by my track record, is a while but not forever. If it's useful to you, fork freely. It's Apache-2.0.

Questions, bug reports and "why on earth did you do it this way" are all welcome, here or on GitHub.

Enjoy!

2

Some really interesting ideas in there, thanks for developing and sharing this.

I took a rather different approach with UIBUILDER simply because I wanted to minimise the number of nodes but maximise the shared code. So I went for a series of singleton class libraries. In more recent versions, moving more startup code into a runtime plugin.

My API endpoints are served from one of the libraries. I've just finished migrating older API's to the later v3 library which reduces the number of handler functions.

Similarly on the Editor side, I've a load of common functions and settings in an Editor plugin - this is really useful to reduce the complexity when using some of the node-red features, some of which are quite boilerplate heavy.

But I'm very interested in worker threads as one of the next optimisations I want to look at. I've a number of areas of code that are filing system heavy and some might be better done in a worker.

Not sure about the GPU part, something I might look at later to see what it might be used for.

Thanks again.

3

Thanks Julian, I've looked at your UIBUILDER.

I think our two approaches are solving different problems and they'd sit together fine. Yours is about sharing as much code as possible across a family of nodes. Mine is about what one node can do without the author writing the plumbing again. Which for me was the difficult-to-use APIs. Nothing in the library cares how the node got built.

On worker threads, you wouldn't need to rework anything to try it.

There's a section in the node-red-base README called "Adding the library to an existing node". You keep your constructor as it is and add two lines, one to compose a base class with the mixins you want and one to call it after RED.nodes.createNode. That gives you this.nodeRedWorker on the node and everything else stays put.

Talking to the worker works two ways. For the file-system heavy stuff where the node waits for an answer, you move that function into a worker file, register it with handle(), and call it from the node with request(), which gives you a promise and passes errors back as real errors so done(err) still reaches a Catch node. For anything that's more of a stream, like progress or a flow of results, both sides have publish() and subscribe(), and if two worker nodes are wired together in the flow they can publish to each other directly without the main thread in the middle. The rest of UIBUILDER wouldn't know anything changed.

The GPU mixin is there because I needed it for image work and it was a short step from the worker code, but it's experimental for a reason.

If you do try the worker mixin and anything gets in the way, open an issue, I'd like to know.

4

Absolutely.

I need to not sidetrack myself right now with the worker stuff as I'm trying to get uibuilder v8 out the door. It is generally pretty performant right now but I've noticed a couple of places that can be a bit slow in extreme cases. methods with FS watchers for example would probably benefit most in shifting workload off the main thread. Reindexing of a markweb is another example where a worker would probably be a good idea. Anyway, added to the backlog for a future look. :exploding_head: