[uibuilder] The next step - 3rd-party comms with a uibuilder front-end

Hi all, with the ongoing discussion about the future of Node-RED dashboards, I've been doing yet more work on making uibuilder a better foundation to build on.

This proposal lays out a likely approach to enabling anyone's nodes to talk to a uibuilder-defined front-end with a minimal amount of code and effort. This is aimed at node authors primarily.

Please do comment freely, as I know you will :grinning:

This isn't a done deal, it is an initial proposal and it is pretty likely that any end result won't look quite like things are laid out here.


uibuilder v5 will use the Node.js package ti-common-event-handler to create a shareable event handler
that facilitates communications between 3rd-party Node-RED nodes and uibuilder node instances (and back again).

Each uibuilder instance will act as a hub for sending data to its connected web clients and receiving messages back and routing them back to the originating node.

Two pieces of data will be required to enable this:

  1. The uibuilder URL
  2. A component identifier

How it works - the component node

A component node will need to:

  • Use the same event handler package. The module in the package is defined as a singleton class instance so it doesn't matter which node require's the module. All require's get exactly the same instance.

  • Use a standard event naming pattern (defined below).

  • Define a component identifier that front-end code will use.

  • Send events to the defined event name with a standard Node-RED msg object.

  • Add pre-defined metadata to the msg so that uibuilder and front-end code know how to process it. See below for details.

  • Register a return event handler with a specific naming convention (defined below).
    The uibuilder instance will maintain an internal map of component-id's to return event names so that messages back from the front-end can be automatically routed back to the originating node.

  • Send an initial control msg on Node-RED startup that will allow uibuilder to create the return map.

  • Send a final control msg on node removal, before re-deployment and on Node-RED closedown so that all event handlers are unregistered and the map is updated.

Front-end web components

Note that the component node does not need to define any front-end components. You could just write suitable code manually for your front-end.
However, there will be a set of standards that will allow a component node to make resources available to uibuilder-based front-ends.

The important thing is that messages will be automatically routed both to and from the front-end.

It is likely that there will be a mechanism in the uibuilderfe.js front-end library that will enable it to auto-load defined resources. Details to be defined later.

How it works - the uibuilder node

When a uibuilder node receives the first event from a component node as defined below, it creates an entry in a mapping table.

The table maps the links between the component nodes (based on the node's Node-RED id) and the component-id.
This allows return messages from the front-end to be routed back to the correct originating node for any processing that node wishes to do.
For example, it may choose to pass the returned message to its output port. This is likely to be the most common scenario.

Event naming standards

Event names will use the following standard to allow component nodes to send data:

node-red-contrib-uibuilder/<uib_url>/<component_id>

To facilitate return messages, the following standard will be used:

node-red-contrib-uibuilder/<node_instance_id>/return

Where:

  • <uib_url> - is the uibuilder URL which is set in the uibuilder node and is the unique identifier for uibuilder instances.
  • <component_id> - is a string identifier that is unique to your front-end code. It can be arbitrary. Component nodes can define defaults if preferred.
  • <node_instance_id> - is the Node-RED node id of the instance of the component node.

Standard msg metadata

The message metadata will take two forms:

  1. Firstly, some uibuilder standard data. This will define the root property to use so that we keep the pre-defined properties to a minimum which avoids name-clashes and avoids limiting how you can use a msg object.

    There is already a standard that uibuilder uses and it is likely that this would continue to be used and extended as needed.

    A component identifier string would certainly be required. As described above, this only needs to be unique within your front-end code.

    It is likely that a "component type" property would also be used to differentiate between data schema's.

  2. Secondly, some component-specific data. This would be defined by the component node's author.

    This would sit under the uibuilder property so as to avoid polluting the namespace of the msg object.

    It is possible that a schema standard would be defined to help but the data would mostly be defined by the component author.

Note that uibuilder already pre-defines some metadata for some tasks including security.

The future

Obviously, this design note only covers communication.

The approach has the strong advantage of being open. Which is to say that it doesn't tie you down to a specific framework since it does not define how your front-end code will process the data.

However, it does lay a foundation that will let nodes be created that will allow authors to create nodes with front-end components and will enable such nodes to fully automate both the component code delivery and the communications between the node and the front-end (and back of course).

This means that this approach is the first step towards a low-/no-code, simple to use, node-based web ui builder.

And yet, despite that, it does not break the core design principal of uibuilder which is to be framework agnostic and unopinionated.

2 Likes

And as expected, the implementation didn't quite follow the script :slight_smile:

However, I've now pushed a new branch to GitHub called vNext if you would like to try things out. Still some rough edges but those are mainly due to the part-finished security changes (don't try those, they don't work properly yet in this branch).

Check out the CHANGELOG file which details the changes so far and also contains a very messy To Do list.

If you want to try the external send/receive, create a simple uibuilder flow then deploy. Then change to the full Vue/bootstrap template. Then add a simple flow with a uib-sender node. Change the settings to select the correct uibuilder URL and check the "Allow return" checkbox and deploy.

In your front-end index.js file, change the uibuilder.on function to add the lines:

if ( msg._uib && msg._uib.originator ) {
    uibuilder.send( {payload:"This is a returned message!"}, msg._uib.originator )
}

Now, every time to send a msg in Node-RED to the sender node, you will automatically get a reply message to the same node.

You can also do uibuilder.setOriginator('<originator_node_id>') (obviously replacing the id with an actual id of a uib-sender node instance or picking one up from a received msg as above. This sets ALL sent messages from your client to go back to the sender node.

I will add a better example and suitable template as soon as I get a chance.

Have a play if you have time and let me know what you think. I'm pretty pleased with it so far. It seems to work OK but there is still a fair bit to do in order to get it ready to handle automated component messaging.

Some of the things still to do:

  • Work out the best way to map from a uibuilder-compatible node (like uib-sender) to a specific on-page component instance.
  • Decide whether uibuilder control messages should also be routable back to sender nodes (currently all control messages go back to the uibuilder node).
  • Work out a standard method for a node to tell the front-end what component needs to be loaded.
  • Work out a standard method for the front-end library to lazy-load components which may (or may not) be defined in a node
  • Work out a standard way for a node to provide a folder of web resources (including maybe web or front-end framework components) to uibuilder front-ends. Not only adding the folder(s) to uibuilder's web server but also automating the load of specific resources.

Then there is still a bunch of work to do finishing off the editor rework, updating package management to allow scoped packages, GitHub packages and specific package versions to be installed along with indicators and controls for updates needed. Then there is the security stuff that I've left alone for a bit in order to try and get some more inspiration.

Anyway, install the vNext branch direct from GitHub if you want to have a go.

Without a doubt, when vNext is released, it will be uibuilder v5.

1 Like

Tihis sounds promising :+1:. Can somebody buy me some time to play with it ...

2 Likes

Another round of changes pushed to the GitHub vNext branch today.

  • A couple of annoying bugs got squashed.

  • Package Management is now almost completely rewritten - still some work to do to allow you to install local packages though. But installing from npmjs and from GitHub both work and you can add tag/version/branch specifiers as well. Useful if you want a specific version or branch.

    As mentioned, uibuilder packages are no longer merged into the Node-RED userDir folder but instead are now in the uibRoot folder (defaults to ~/.node-red/uibuilder for most people). This means that you will need to remove the packages from your userDir and re-add them in the correct folder. Using uibuilder itself is the easiest way to add them back in but the changes also mean that you can install them yourself, when you restart node-red, uibuilder will work everything out as long as the package.json file in uibRoot was updated with the installed packages (the default for npm).

  • The Editor panel has been extensively rewritten as well.

    Some areas of the panel are inaccessible until you have deployed a new instance - which creates the server folders required. Should save confusion, especially if you haven't used uibuilder before.

    New error information is available when you are setting/changing the URL.

    The extra information about installed packages is now included on the Libraries tab. Including the installed version and an estimate of the correct URL to use in your front-end code. There is also a link to the packages homepage (if specified in its package.json file), you can click on the package name. If not known, an npmjs.com search is performed.

    And of course, everything in the panel is laid out much better than before.

  • Behind the scenes, the ExpressJS route handling has been extensively reworked and now uses Express's Router capabilities. This greatly simplifies how routes are created and removed. Especially when instances of uibuilder have their URL changed or when an instance is removed from the flows.

Still left to do is simplifying and tidying up the security feature - I fear it won't have all the features I'd hoped for in this release. Also I want to include @shrickus's idea to allow a custom middleware to be added to an instance so that API's can be added.

A couple of other bugs to squash too.

The changelog file in the branch has details of the changes and a long todo list (not all of which will make it into this release) :slight_smile:

Please do try it out if you have a chance and let me know of any bugs, issues or wishes.

1 Like