New structure for building node-red nodes

I've been trying to improve my developer experience while creating nodes for node-red. Could you guys give me some feedback and help me to improve it?

Features:

  • clear separation of client and server pieces of a node. The server side has a Class to shape its form. It provides auto completion.
  • directory structure convention. The folder name of a node define its type. All nodes within the src/nodes folder will have its type automatically prefixed. This way, a dev does not need to prefix all his node folders manually. For example, the set of folder names for nodes created to Salesforce like these
    -./src/nodes/salesforce-connection,
    -./src/nodes/salesforce-crud,
    -./src/nodes/salesforce-soql
    would be created as
    -./src/nodes/connection
    -./src/nodes/crud
    ./src/nodes/soql
    But once built, their types would be registered as "salesforce-connection", "salesforce-crud", "salesforce-soql"

This is the directory structure, which would lead to the creation of a node type called "salesforce-connection"

salesforce/
โ””โ”€โ”€ src/
    โ””โ”€โ”€ nodes/
        โ”œโ”€โ”€ connection/
            โ”œโ”€โ”€ client/
            โ”‚   โ”œโ”€โ”€ i18n/
            โ”‚   โ”‚   โ”œโ”€โ”€ dictionaries/
            โ”‚   โ”‚   โ”‚   โ”œโ”€โ”€ de.json
            โ”‚   โ”‚   โ”‚   โ””โ”€โ”€ en-US.json
            โ”‚   โ”‚   โ””โ”€โ”€ docs/
            โ”‚   โ”‚       โ”œโ”€โ”€ de.html
            โ”‚   โ”‚       โ””โ”€โ”€ en-US.html
            โ”‚   โ”œโ”€โ”€ icons/
            โ”‚   โ”‚   โ””โ”€โ”€ icon-1.png
            โ”‚   โ”œโ”€โ”€ index.html
            โ”‚   โ””โ”€โ”€ index.js
            โ””โ”€โ”€ server/
               โ””โ”€โ”€ index.js
  • the JS and HTML form of the client side portion are now separate pieces, instead of a single .html. The code highlightning for each language will work without a problem.
  • A lib that enables devs to write the server side of the node using ES6 Classes. It has the Node class definition and a mixin that does part of the setup of a node. It is similar to what Salesforce did for the LWC Framework.
  • enables modern javascript in both front and backend. The js code you write today, is built for the browser, or node version, of tomorrow without much trouble thanks to ESBuild (I will try rollup too)
  • bundling, tree shaking, update js to use modern browser features automatically
  • quick development environment using docker-compose, if needed
  • sourcemaps for debugging
  • builder => possibly a Vite plugin in the future

Once this framework is round, I will try to incorporate it into NR directly.

This is the .html form of a node. No more <script> tags. The type is defined by the name of the folder. And the registration lines are written automatically for the dev in the final build. This is closer to what modern js frameworks do.
image

client side js is modular and uses ESM module system. You no longer need to write all your client side JS in a single file. And you can use any of the libraries published in npm registries. The registration is also done automatically by the framework, in the final build. It uses the name of the folder as the type.

As an example for the server side code, the following

export default function(RED){
     function Node1(config){
         RED.nodes.createNode(this, config); 
          
         const node = this;
         this.on("input", (msg, send, done) => {
             node.log("hello world");
             send("something");
         }
     } 

     RED.nodes.registerType("node-1", Node1);
}

Becomes this

import { Node } from "@allanoricil/node-red-node";

export default class Node1 extends Node {
  constructor(config) {
    super(config);
  }

 onInput(msg, send, done) {
      this.log("hello world");
      send("something");
  }
}

As you can see, a dev no longer has to remember to write RED.nodes.registerType("node-1", Node1); or that he/she MUST write RED.nodes.createNode(this, config); before calling this.on to register event listeners. For those who don't know, if you write this.on before RED.nodes.createNode, listeners won't be registered. But in my framework, you don't need to remember it.

Additionaly, while you are writting your node class, because there is a parent class "shaping" your node's class, auto completion features will recommend function names available for you automatically as soon as you type, for example, "on".

Hi Allan, welcome to the forum.

A very interesting package!

Couple of things I'm not quite yet understanding, perhaps you could clarify.

Firstly, the use of Vue in the Editor (.html) panel file - not sure why you would want to add the complexity of VueJS when Node-RED already has a LOT of stuff packed into the Editor such as jQuery, jQuery-UI, and a whole lot more? The Editor panels are generally pretty simple even for complex nodes and I just wonder how much extra load this adds?

Secondly, why is Docker part of the build? Again, this seems overkill since it is dead-simple to test a dev version of a node against a Node-RED instance?

Actually, you've also got handlebars in the mix and again, I'm not quite understanding what benefit that adds?

There is also this in your server files: import { Node } from "@allanoricil/node-red-node"; and I'm struggling to find where @allanoricil/node-red-node is defined?

Personally, anything that requires any kind of installing servers that need to 'build' I quickly skip.

There are some node-building solutions available, one that completely does the building (package.json, readme etc) and publishing in npm via node-red flows and there is another that does most of the work via subflows.

I was reading it, until I seen Docker being apart of chain.
Many here don't use Docker, and in fact discourage it.

I agree with Julian, I'm not sure why this needs docker.

EDIT
With that said, if this is tooling for your self, and one self uses docker, then I get the ambition to make it easier.

@TotallyInformation Thanks for the reply.

Vue was added in that example with the sole purpose of demoing that is now possible to use any library you want when building the "client" portion of a node. It was just a demo to show that the builder can be configured to do it, if the Developer needs. Another example of external dependency that I added in the demo is "axios". It is used in both "client" and "server" ports of the node. I just added it to demo to show how easy is to import external dependencies to your custom nodes projects.

Docker isn't required, I know. It is just that it covers a Use case I had while developing nodes that integrate with other Dockerized Services. Using docker-compose as the default dev environment runner, ensures Developers clearly specify all the requirements for running and testing changes for that particular node. For example, I have a set of nodes that interact with a vault-service. This vault-service is a containerized application that loads Secrets and Parameters from AWS and make them available to my custom node-red nodes. My nodes can't be tested without having this other service running. When the next developer takes on this project, he will easily get all the information he needs to quickly setup the dev environment to test his changes. He will not need to run the vault-service manually, or write a mock service. It will be just there. He adds his environment variables, then run "npm run dev" to quickly have his dev environment ready for tests.

Handlebars is part of the "builder", not the node. The builder is going to become a dependency in the future. At the moment it is coupled just to show its usage and test things while I'm creating it. The builder does some transformations in the code while removing some burdens/responsibilites from developers. In summary, you write less, and what you write is garanteed to return a build that works with NR. For example, you no longer have to remember to write RED.nodes.registerType("type", Node) for every node. You just write the Node logic, the framework ensures it will be added to built code automatically. The "type" is defined by the name of the folder, you no longer have to write it in multiple places. The same for the client side portion. No more , or "RED.nodes.registerType("type", {})". Another thing you no longer have to write is RED.nodes.createNode(this, config); Did you know that if you write it after adding event handlers, the event handlers are just not going to work? You have to write it before calling any this.on('input', ()=>{}), for example. If you do it after, the handler just doesn't work.

Handlerbars templates are part of the builder, and it is what enables you to just write .html file for the form and have Sintax Hightlighting working for the html tags. Without my framework, devs write all their node's client side code (html and js) in a single, non modular, html file, using multiple tags. Imagine a package with thousands of nodes, all maintaining their html forms in a single file. It is just a mess. Open one of the node's .html and verify that it contains only the form of the node, and that sintax highlighting just works. All nodes html files are late built, and composed into a single .html file, like NR expects. But while developing, they are all modular and separate, as any other major JS Framework, like Vue does for their components.

This is a lib I'm writting to enforce a strict structure for the server portion of a Node. Think of it like a Vue component using the "options api". All methods a Node can have are defined in the "Node" class. With it, "auto completion" just works, and the Node object that RED.nodes.registerType function expects is garanteed to work. If you open some older node-red custom nodes you will see that they are functions, and there is no "auto completion" feature because there is nothing that can "shape" a function return unless you use a Class/Type/Interface. It is how modern javascript developers develop create APIs for frameworks that other developers will follow. Take a look at what Salesforce did for the LWC framework. For every LWC component someone writes, they do the following

import { LightningElement } from "lwc";

export default class MyComponent extends LightningElement {
  name = "LWC";
}

For every Node-RED Node built with my framework, devs will do the following

import { Node } from "@allanoricil/node-red-node";

export default class MyNode extends Node {
  ...
}

As another example, for comparison, the following:

export default function(RED){
     function Node1(config){
         RED.nodes.createNode(this, config); 
          
         const node = this;
         this.on("input", (msg, send, done) => {
             node.log("hello world");
             send("something");
         }
     } 

     RED.nodes.registerType("node-1", Node1);
}

Becomes this

import { Node } from "@allanoricil/node-red-node";

export default class Node1 extends Node {
  constructor(config) {
    super(config);
  }

 onInput(msg, send, done) {
      this.log("hello world");
      send("something");
  }
}
1 Like

The builder will have a non docker runner too. You just have to run "npm run dev", and a script will launch node-red and add your nodes in it. I'm still developing it. The plan is to add node-red as a dev dependency, and then run it from the "node_modules" that were installed. But I have not tested it yet.

However, In this example I used docker because it covers a Use case I had while developing nodes that integrate with other Dockerized Services. Using docker-compose as the default dev environment runner, ensures Developers clearly specify all the requirements for running and testing changes for that particular node. For example, I have a set of nodes that interact with a vault-service. This vault-service is a containerized application that loads Secrets and Parameters from AWS and make them available to my custom node-red nodes. My nodes can't be tested without having this other service running. When the next developer takes on this project, he will easily get all the information he needs to quickly setup the dev environment to test his changes. He will not need to run the vault-service manually, or write a mock service. It will be just there. He adds his environment variables, then run "npm run dev" to quickly have his dev environment ready for tests.

To conclude, the way you launch your dev environment for testing your nodes isn't going to force using docker. It is just another option to show devs how they can test their nodes when they depend on other dockerized services.

Could you explain it further?

Their solutions are for different "niche" of people. Developers won't create nodes using "flows", they will write code. They need a strict pattern to enable maintenance. Right now, a node-red node is not modular, and everyone writes everything in a single .html or .js file. It is impossible for anybody else to jump into the project and help to maintain it. Let me give you example:

with all due respect, Im not saying it with bad intentions

But this is an example of a node that isn't maintainable.

node-red-contrib-cron-plus

Search its github repo and take a look at the code. It has 3K plus line in a single .html file

As a developer I would never install a dependency in my NR instance that I can't read and comprehend it as a whole. To ease comprehension, a node's source code must be modularized. Like any other JS framework does.

Yeah! @Steve-Mcl :smile:

To be fair, he does holiday a lot!

@marcus-j-davies could you please update your quote to include the "polite" message I wrote before? Without it @Steve-Mcl will probably be triggered and say I'm a jerk. I did not say that with bad intentions.

2 Likes

Don't worry about it. I'm not triggered.

There are reasons for it's size - a combination of the amount of features, when it was started (before node-red natively supported auto resources directory) and my aversion to build tools/map files for basic stuff like html/js that has no dependencies.

So while many might find it unmaintainable, the fact it is "all in one place" and has zero build, I find it's actually more maintainable (and I wrote it so I know it :wink: )

If I were to start again from scratch there are a few things I'd change (e.g the inner components would be separated out and put into the resources directory) but a build step is not one of them tbh.

This is the argument that proves it is not maintainable. When you retire, who is going to continue? how easy is to someone to join the project?

Modularization is "one of the things" that makes it easier for devs to maintain projects. Imagine if the NR code was all written in a single file. Would it be maintainable just because its creator "know it"?

A build step is done in all major code written in JS for several reasons:

transpile code to modern javascript. The JS runtime where your written JS runs varies every release. New features are added, deprecated and eventually fully removed. Without a bundler for JS, like esbuild/rollup, your code will have to manually be updated to use modern javascript features. Imagine having thousands of customers updating their Browser and discovering that your code no longer runs because the new major version of the browser deprecated one of the features you used. With a bundler, you can just keep writting code like you are used to, and the bundler will update it for you automatically, based on a list of "supported browsers".

All major JS frameworks out there are switching over to Vite, a builder, for a reason. It bundles their JS using Rollup/Esbuild to keep their code up to date, do transpilations and other file transformations. It makes things maintainable in the long run.

You aren't telling me anything I don't already know. I do use build tools (webpack, vite, rollup, esbuild, ...) in various projects and my day job but as I've already said...

I appreciate what you are saying and love that you are contributing to this vibrant community but pushing overly complex setups for simple html/js where it is just not necessary, I'm not gonna do it and I know plenty of folk who feel the same.

Each to their own

Without a building step, how a dev will know which features no longer work with the current browsers or Node version?

How a dev will add polyfils to his work? manually?

And again, a project without dependencies can still be extremely complex. For example, the node I mentioned above that you wrote has 3K lines of code in 1 single file. It has no dependencies, right? A builder would have helped you to write it in separate modules. If you had written it in a modular way, I can assure you someone "without any prior context of it" would be able to understand it way easier, and with less time. Dont you agree?

Let's try an empathetic exercise. If someone else had written that node of yours, and you were tasked with analyzing it and understanding how it works, without any prior experience, would you find it easy and enjoyable to read?

And this is what I'm particularly interested in. But I can't see the definition for that class. Not sure where it comes from.

Personally, I'd want to have that in a separate node-red dev setup and not in each node. It makes each node far too complex.

I fully agree with this. Which is why my own nodes are far more structured than the "traditional" example nodes. I just think that your example is simply too complex to easily comprehend and includes things that would be better as a separate module.

This is something I think about a lot with UIBUILDER. Which is why I've continuously refactored the code into more logical modules.

This is something I've struggled with. I certainly use esbuild for all of my front-end code but I've not really managed to do this reliably with my back-end code as I don't seem to be able to get .map files to generate properly and so debugging becomes impossible. One of the reasons I'm interested in how you are doing things. Unfortunately, your code is too impenetrable for my limited understanding right now.

I would like to invite you all to try the project in the following repo and branch, and then give me some feedback about your DX.

I added the first iteration of watch, start, build scripts. No docker is required.

  • clone the repo
  • checkout builder branch
  • run npm install
  • run npm run watch
  • open NR editor
  • edit any file in ./src
  • verify that NR is restarted and the browser refreshes automatically
  • open chrome inspector, and check these 2 props to clearly see the "original" code and the built one (client side code)
  • Add some breakpoints and try to debug the clientside portion of the node

Debugging in the server side is already possible, but I will clean things up, modularize the builder, write some tests, and finally turn the builder into a CLI before doing a complete tutorial.

The cli name will be called nrg, which reads as "energy", if you pronounce the letters N R G in english. The acronym means "Node-RED Generator. It will take away some of the time you spend developing and maitaning nodes, which can result in energy savings, efficiency.

The builder will be released in this other repository, eventually.

2 Likes

obs: this isn't ready yet, but let me show how it will be done.

Debugging server side demo from within vscode.

I just have to run npm run start or npm run watch, with debug mode enabled

and then attach a debugger

You can also attach a debugger directly from the browser itself, using chrome://inspect

You can now try nrg in this project template!

I would like you to try it out and give feedback!
Is there a node you can't built with it?

You can configure any node-red settings in the nrg.config.js. For example

module.exports = {
  version: "0.0.0",
  nodeRed: {
    logging: {
      console: {
        level: "debug",
      },
    },
  },
};

These are all the defaults for nrg config: nrg/defaults/nrg.json at main ยท AllanOricil/nrg ยท GitHub

The only dependencies you need in an nrg project are:

@allanoricil/nrg-cli and node-red

I'm going to make node-red a peer dependency of @allanoricil/nrg-cli so that the cli is the only thing you need.

The cli repo can be found here: GitHub - AllanOricil/nrg-cli: A CLI tool to quickly generate, build, and manage Node-RED nodes and plugins, empowering your development workflow.

Hi everyone. Came here to share I released the first version.

You can all start refactoring your nodes to comply with the nrg framework :smiley:

1 Like