[Update on progress] node-red-contrib-uibuilder vNext/v5

Share Your Nodes
1

Hi all, trying to keep the pressure on myself to get v5 out the door. So posting a quick update on all of the changes so far. This is turning into a really major release and if you are already using uibuilder, I do recommend that you find some time to do at least some quick tests on the new version.

Assuming I can stop myself from doing any further innovations, there are only a few bits left now. A couple of fixes mostly. However, the vNext branch should be perfectly usable at the moment.

Install using:

cd ~/.node-red
npm install totallyinformation/node-red-contrib-uibuilder#vNext

CHANGELOG

WARNING: Though I've done some work on the security features, they are still not ready. Do please try them but NOT IN PRODUCTION.

BREAKING

  • Clear client cookies - You may get some odd results if you don't because cookie handling as been significantly reworked.

  • Installation of packages for use in your front-end code has been moved from the userDir to uibRoot.

    Note that only packages installed into the uibRoot folder will be recognised.

    Unfortunately, this means that you will need to re-install your packages in the correct location. You should uninstall them from the userDir.

    By default: userDir = ~/.node-red/, uibRoot = ~/.node-red/uibuilder/. However, both can, of course, be moved elsewhere.

    uibuilder will automatically create a suitable package.json file in uibRoot. That file not only lists the installed packages but also has a custom property uibuilder that contains metadata for the uibuilder modules. Specifically, it lists all of the necessary detailed data for the installed packages.

    The files <uibRoot>/.config/packageList.json and <uibRoot>/.config/masterPackageList.json are no longer used and may be deleted.

    You can now install not only packages from npmjs.com but also from GitHub and even local development packages. @scopes are fully supported and versions, tags, and branches are supported for both npmjs and GitHub installs.

  • Peer installation of VueJS and bootstrap-vue yet again removed. Since these now need to be in the uibRoot folder which
    we don't necessarily know at preinstall time.

    We will be looking at another alternative method. Now that template switching is even more powerful and so is
    package management, it is likely that we will build something into the template installation process.

    Until then, please install the vue and bootstrap-vue packages via the uibuilder library manager if you need them.

  • Minimum Node.js version supported is now v12.20. Minimum browser version remains the same and must be one that supports ES6.

New

  • New node uib-sender - this node allows you to send a msg to any uibuilder instance's connected front-end clients.

    That means that it is pretty much the same as sending a message directly into a uibuilder node.

    You select the instance of uibuilder you want to use by selecting an existing uibuilder URL from the dropdown.

    You can also select whether you want input messages to go straight to the output port as well.

    Or, more usefully, you can allow "return messages". This allows a front-end client to send a message to node-red with some pre-defined metadata added that will route the message back to the uib-sender node. In this way, the sender node can be used as a semi-independent component.

    Note that this same method can be used by ANY custom node, check out the code to see how it works. It requires the use of an external, shared event module @TotallyInformation/ti-common-event-handler. The msg metadata looks like: { _uib: {originator: <sender_node_id>}, payload: ... }. The sender node id is just that, the Node-RED node id for the sender node instance.

    The uibuilderfe.js library has been updated to allow easy use of the originator property for uibuilder.send(). See below for details.

    There is a new page in the Tech Docs on using the sender node.

  • New node uib-cache - this node allows you to cache input messages in various ways and recognises uibuilder's
    cache control messages so that a client browser (re)connecting to a web page will automatically get a copy of the cached pages.
    See the Tech Docs for details. An example flow is included in the uibuilder examples library.

    Note that you can use this node without uibuilder if you want to.

  • New Feature Instance API's. You can now define your own API's to support your front-end UI. These run as part of the Node-RED server and can be called
    from your UI, or indeed from anywhere with access to the Node-RED server's user endpoints.

    You can add any number of *.js files to a folder <uibInstanceRoot>/api/. Each file will be loaded into the uibuilder instance and tested to make sure that it
    contains either a single function or an object containing functions who's property names match either an HTTP method name (get, put, etc) or the generic use.

    Such functions are added to the instances router. See the Tech Docs for more information on how to use the instance API's.

  • Extended Feature Package Management - You can now install not only packages from npmjs.com but also from GitHub and even local development packages. @scopes are fully supported and versions, tags, and branches are supported for both npmjs and GitHub installs.

    Note that only packages installed into the uibRoot folder will be recognised.

    Also note that if you manually install a package rather than using the library manager, you will need to restart Node-RED.

  • New layout for the Editor panel

    This is a much cleaner and clearer layout. It also blocks access to parts of the config that don't work until a newly added node has been Deployed for the first time so that its server folder has been created.

    There are also some additional error and warning messages to make things clearer.

  • Extended Feature Updated node status display - Any instance of uibuilder will now show additional information in the status. In addition to the existing text information, the status icon will be YELLOW if security is turned on (default is blue). In addition, if Allow unauthorised msg traffic is on, the icon will show as a ring instead of a dot.

  • New Feature - Added a version checker that allows uibuilder to notify users if a node instance must be updated due to a change of version.

  • Extended Feature - Added uib version to the connect msg to clients and a warning in the client console if the client version not the same as the server.

  • Extended Feature - Now allows socket.io options to be specified via a new property in settings.js - uibuilder.sioOptions. See the discussion here. The Tech Docs have also been updated.

  • If using a custom ExpressJS server for uibuilder, allow different https settings (key and cert files) from Node-RED itself. Uses a new property in settings.js - uibuilder.https.

  • uibuilderfe library

    • New Feature - Received cookies are now available as an object variable key'd on cookie name. uibuilder.get('cookies').
    • New Feature - A new unique client id set by uibuilder is available as a string variable. uibuilder.get('clientId'). This changes if the page is reloaded but not if the client loses then regains a Socket.IO connection (where the socket id will change). It is passed to the client as a cookie. The client sends it to the server as a custom header but only on Socket.IO polling requests since custom headers are not available on websocket connections). It also adds it to the socket.handshake.auth.clientId property which should always be available to the server event handlers. Caution should be used if making use of this feature since it is likely to change in the future. See the updated sioMiddleware.js for an example of use. The client id is also included in the uibuilder control msgs output to port #2 on a client connect and disconnect. The ID is created using the nanoid package.
    • Extended Feature - Toast notifications (notifications that overlay the UI) are now available even without VueJS and bootstrap-vue. They can be styled using the uib-toaster, uib-toast, and uib-toast-head classes when not using bootstrap-vue. Toast notifications can be set either by a standard msg from Node-RED or by calling uibuilder.setToast(msg) (where the msg matches the same format used from Node-RED). Internal uibuilder visual notifications will also use this mechanism. Notifications auto-clear after 10s (used to 5) unless otherwise controlled via the options.
    • New Feature - A hidden style sheet has been introduced. This is created by uibuilderfe and is attached to the start of the head section so that it can easily be overridden by including your own style sheet in the head. In addition to the toast classes, :root { --main-bg-color: white; } is defined. This is used by the document body background and by the toast message by default. This makes it easy to change the background colour. Use the Elements tab and Styles panel in your browser developer tools (F12) to review the styles and CSS variables.

Changed

  • Cookie handling has changed for the better. There are 3 cookies set by uibuilder: uibuilder-namespace (what SockeT.IO needs to communicate), uibuilder-client-id (see new features above), and uibuilder-webRoot (if you are using httpNodeRoot in settings.js). Each is set as a session cookie which means that if you close the window/tab showing your UI, the cookies are deleted. However, if you are being super-strict about EU and California law, you should inform your users that they exist. The cookies are limited to the exact path for the uibuilder instance they come from so there shouldn't be any cross-contamination. However, you should clear any old cookies when upgrading to uibuilder v5 from v4 or before.

    Custom headers are also added by uibuilder. These are only accessible via XHR API calls, not by uibuilderfe itself. x-powered-by (set to uibuilder), uibuilder-namespace, and uibuilder-node (the node id of the uibuilder node). In addition, uibuilder sets X-XSS-Protection to 1;mode=block and X-Content-Type-Options to nosniff for added security. You can, of course, override all of these using custom middleware.

  • uibuilder nodes now show the url in angle-brackets. If the url is not defined, <no url> shows. If the node has a name, this is shown before the url. e.g. My UI <myui>. If you want to have the url show on a different line to the name, add \n to the end of the name.

  • When adding a new uibuilder node, the url is now blank. This helps prevent accidentally creating two nodes with the same url which is confusing to recover from. As a blank url is not a valid configuration, the red triangle will show.

  • When copying and pasting a uibuilder node, the pasted node(s) will have their URL changed to blank to prevent nodes with duplicate url's being deployed.

  • Improvements to the "uibuilder details" page should make it easier to read. The data for ExpressJS Routes is much improved.

  • Editor panel

    • Improvements to the Editor help panel. Should hopefully be clearer and includes all of the settings and custom msg properties. Now uses a tabbed interface.
    • File editor now excludes .git/**, .vscode/**, node_modules/** and _*.
    • When editing the configuration for a uibuilder node, if the URL is invalid or the server folder hasn't yet been created, you cannot access various parts of the panel.

  • uibuilderfe.js client library updated to allow for the use of an originator metadata property. This facilitates routing of messages back to an alternative node instead of the main uibuilder node.

    There are three ways to make use of this:

    • Use the new uibuilder.setOrigin('<sender_node_id>') function. This will then route ALL messages from the client back to the specified node. This is of marginal use because the main use-case for the property is to automate routing of data to/from web components of which there are likely to be several on a web page.
    • Use the new override parameter for the send function. uibuilder.send(msg, '<sender_node_id>'). This will send this one message back to the specified node. It will override the setOrigin. The utility uibuilder.eventSend() method has also been updated to allow the originator parameter.
    • Manually add the metadata to the node { _uib: {originator: <sender_node_id>}, payload: ... }. This is not generally recommended as it is error prone. However, if writing custom front-end components, you may want to include the origin property as an option to allow end-to-end automatic routing of messages to/from your component instances.

    See the new uib-sender node details above for an example of using the originator property. That node adds the property to its received msgs before sending to your connected clients.

    Note that at present, control messages from the front-end cannot be routed to a different originator node, they all go to the main uibuilder node. This will be reviewed in a future release. Let me know if you think that it is needed.

  • Added documentation for Socket.IO middleware and error handling.

  • Minor improvements to the .config middleware templates.

  • Improved logging for npm commands in library manager.

  • Added client IP address to client connect & client disconnect control msgs

  • Security improvements:

    • When security is active, a client that re-connects to Node-RED will attempt to reuse its existing authorisation (see the localStorage bullet below).

      This means that opening a new tab or window in the same browser profile will automatically connect without having to log in again.

      This process assumes that your local security tracks connected clients and is able to reconnect to them without needing the password to re-authenticate.

      To achieve this, the server sends the 'client connect' control msg to the client which responds with an 'auth' control msg containing the existing msg._auth recovered from localStorage. The server validates the JWT and then runs the customisable userValidate process.

    • JWT processing now includes more checks. In addition to expiry, subject, issuer and audience are validated. Optionally the client IP address can also be validated (see new flag in the security section of the config panel in the Editor).

      Also, returned errors and messages should be clearer to indicate what went wrong. Additional error information is shown in the Node-RED log.

    • When security is active, any uibuilder node in a flow will have a status with a yellow (instead of the normal blue) icon. If the icon is filled, no messages
      can flow unless a client is authenticated. If it is a ring, messages will flow regardless and it is up to the flow author to control things.

    • When security is active, pass flag to front-end. Use uibuilder.get('security') to get the current status. The flag is passed on the initial connection message from the server.

    • uibuilderfe.js

      • Added auth details to localStorage so that they are available on page reload and available from any browser window or tab on the same machine/browser profile.
      • Made sure that all updates to auth details use self.set to trigger update events.

    • Move core security functions from /nodes/libs/uiblib.js to /nodes/libs/security.js which is a singleton class instance to match the style of socket.js and web.js

    • /front-end/src/uibuilderfe.js

      • New security flag
      • Only run security related functions when security flag is active
      • Add some bootstrap_vue toast warnings to match the console output warnings (only does anything if you are using bootstrap-vue, otherwise does nothing)

    • /nodes/libs/security.js

      • Add security flag to initial control message to client
      • Prevent client from sending msgs if security is on but client not authorised (is dependent on keeping track of clients on the server)

    • /nodes/libs/uiblib.js

      • Start to work on blocking msgs from node-red to client when security is on but client not authorised. WARNING: NOT WORKING YET messages will always get through.
      • sendControl() - make Socket.ID optional.
      • authCheck() - change from socket parameter to socketId to make it easier to call from more places. Also add more extensive _auth and _auth.id checks.
      • logon() - change warnings to remove note about not permitted in production as this is no longer the case (see change notes for v4.1.1)

    • /templates/.config/security.js

      • Add new functions jwtCreateCustom and jwtValidateCustom. In readiness for more flexible and secure JWT handling.
      • Add new functions captureUserAuth, removeUserAuth, and checkUserAuth. These will support being able to restrict sending of msgs from Node-RED to clients.
        Without them, any msg input to a uibuilder node will be passed to clients regardless of whether they are logged in or not.
      • Add new function userSignup. This will (optionally) allow you to offer a self-service sign-up process.

    • Tech docs - some minor improvements to the security process docs and bring into line with current process.

Internal and development improvements

  • Shared event handler implemented. This enables external nodes to send and receive data to/from uibuilder front-end clients.

  • Gulp implemented

    • initially for composing the uibuilder.html from the contents of src/editor
    • and to replace the previously manual minify step for uibuilderfe.js
    • other tasks likely to be added in the future to make more efficient code and ease the release/publish process.

  • New eslint rulesets implemented & config restructured. Along with the .html file decomposition, this makes for a much more accurate linting process.

  • Massive number of minor code improvements to uibuilder.html and uibuilder.js & to the supporting libs and uibuilderfe.js thanks to the impoved linting.

  • Removed deprecated functions.

  • Remove old console.log statements used for testing and no longer required.

  • Even more massive restructuring of uibuilder.js.

    • A lot of the core logic now moved into dedicated modules, each containing a singleton class.
    • Removing the need for the node object. This meant the use of some arrow functions to be able to retain the correct context in event handlers and callbacks.
    • Destructuring the big exported function into a series of smaller functions. Makes the code a lot clearer and easier to follow. Also helped identify a few bits of logic that were not quite sane or not needed at all (the result of evolutionary growth of the code).
    • Using named functions throughout should make future debugging a little easier.
    • npm package handling moved to a separate singleton class in package-mgt.js

  • Removed inputHandler function from uiblib.js. Code folded into the inputMsgHandler function in uibuilder.js which has been destructured so is small enough to have it as a single function.

  • Package management rewritten. Should be faster and uses async/Promise functions.

  • Editor changes

    • Added a version checker that allows uibuilder to notify users if a node instance must be updated due to a change of version.
    • Minified the Editor panel html file (using Gulp) - should load faster now.

  • v3 admin API changes

    • Moved v3 admin API to its own module (libs/admin-api-v3.js) and changed to be an ExpressJS router instance.
    • Moved the setup from uibuilder.js to web.js
    • New v3 admin API command added to list all of the deployed instances of uibuilder. Issue a GET with cmd=listinstances. This allows other nodes to get a list of all of the uibuilder instance URL's and the ID's of the nodes that create them. See the uib-sender node's html file for details.

  • v2 admin API changes

    • Moved v2 admin API to its own module (libs/admin-api-v2.js) and changed to be an ExpressJS router instance.
    • Moved the setup from uibuilder.js to web.js

  • web.js changes

    • Both admin and user routes restructured, making use of Express.Router's to improve layout and control.
    • Setup of v3 admin API moved to web.js class module (out of uibuilder.js).
    • Setup of v2 admin API moved to web.js class module (out of uibuilder.js).
    • New web.dumpRoutes(print=true) method added to web.js - dumpRoutes outputs a summary of all the relevant ExpressJS routes both for uib user facing web and Node-RED admin web servers. Also added individual methods: web.dumpUserRoutes(print=true), web.dumpAdminRoutes(print=true), and web.dumpInstanceRoutes(print=true, url=null) (where passing a uib url will just dump that one set of routes).
    • Removed the separate serve-static npm package. This is now built into ExpressJS and not required separately.
    • Removed the separate body-parser npm package. This is now built into ExpressJS and not required separately.
    • Moved the user-facing API's to web.js from uibuilder.js and moved to their own Express.Router on ../uibuilder/....
    • Added a new this.routers object - this helps with uibuilder live configuration documentation as it records all of the ExpressJS Routes that uibuilder adds.

Fixed

  • URL validation should now work as expected for all edge-cases.
  • Fixed the problem that required a restart of Node-RED to switch between src and dist folder serving.
  • Fixed Issue #159 where sioMiddlware.js wasn't working due to the move to Socket.Io v4.
  • Fix issue reported in Discource of an error in masterMiddleware when setting headings. Corrected heading syntax for ExpressJS v4.
  • Client connect and disconnect msgs not being sent to uibuilder control port (#2). NOTE: As of Socket.io v4, it appears as though the disconnect event is received after the connect when a client is reconnecting. You cannot rely on the order.
  • Fixed CORS problems after move to Socket.IO v4. (NB: CORS is defaulted to allow requests from ANY source, override with the uibuilder.sioOptions overrides available in settings.js).
  • uiblib.js logon() - Fixed error that prevented logon from actually working due to misnamed JWT property.
  • A number of hard to spot bugs in uibuilder.html thanks to better linting & disaggregation into component parts
  • In uibuilderfe.js, security was being turned on even if the server set it to false.
  • Fixed an issue when removing uibuilder nodes caused by the move to socket.io v4. Should fix the failure to remove unused uib instance root folders and fix renaming problems as well.
  • URL rename failed if user updates template before committing url change. This is now blocked.
  • File editor failed if the node hadn't been deployed yet. Blocked if instance folder hasn't yet been created.
  • Change degit call to turn off cache which was producing a could not find commit hash for HEAD error. See degit Issue #37. Partial fix for Issue #155.
  • If deleting a node that hasn't been deployed, a delete folder warning is given - add check to see if the folder actually exists before giving the error.
  • If using Node-RED Docker with recommended install, uib couldn't find the Socket.IO client folder to serve. Issue. Extra check and cleared warnings added.
  • Spurious instance folder rename when it wasn't needed.
  • Bad cookie handling!
9 Likes
2

Absolutely fantastic work Julian. :tada::clap:

I must make time to test out the updates.

3

Totally agree with Steve on those sentiments! Cheers!

From looking at the vnext code, it appears there are some new configurations available in settings.js, too? Where can I find an example of what goes into the new uibuilder section, especially around the customServer settings?

4

Ah, well maybe you haven't perused the updated Tech Docs? If you have vNext installed, open the link to the Tech Docs and you will find loads of new information including the new uib-configuration page. :grinning:

Oops, though I've just noticed it is missing some stuff - ah well, I'll add that now - in the meantime, here is the annotated version:

   /** Custom settings for all uibuilder node instances */
   uibuilder: {
        /** Optional HTTP PORT. 
         * If set and different to Node-RED's uiPort, uibuilder will create
         * a separate webserver for its own use.
         */
        //port: process.env.UIBPORT || 3001,

        /** Optional: Change location of uibRoot
         * If set, instead of something like `~/.node-red/uibuilder`, the uibRoot folder can be anywhere you like.
         */
        uibRoot: process.env.UIBROOT || '/src/uibRoot', //path.join(os.homedir(), 'myuibroot')',
        
        /** Only used if a custom ExpressJS server in use (see port above)
         * Optional: Default will be the same as Node-RED. @type {('http'|'https')} 
         */
        customType: 'http',
        
        /** Only required if type is https, http2. Defines the cert & key. See Node-RED https settings for more details.
         * @type {Object<Buffer,Buffer>}
         */
        // https: {
        //     key: 'keyname.key',
        //     cert: 'fullchain.cer'
        // },
        
        /** Optional: Socket.IO Server Options. See https://socket.io/docs/v4/server-options/
         * Note that the `path` property will be ignored, it is set by uibuilder itself.
         * You can set anything else though you might break uibuilder unless you know what you are doing.
         * @type {Object}
         */
        // sioOptions: {
        //     // Make the default buffer larger (default=1MB)
        //     maxHttpBufferSize: 1e8 // 100 MB
        // },

        /** Controls whether the uibuilder instance API feature is enabled
         *  Off by default since uncontrolled instance api's are a security and 
         *  operational risk. Use with caution.
         */
        instanceApiAllowed: true,
   },
2 Likes
5

Aha, although I did see the TechDocs (very nice, btw!) I did NOT notice that my
uibuilder: { ... } settings were incorrectly placed at the end of settings.js,
inside the editorTheme: { ... } section -- D'oh! :man_facepalming:

So I guess you can buy me a beer for discovering another negative test? Hahahaha :wink:
Seems the initialization steps get confused if there is NO uibuilder section found...

6

Oh drat, that shouldn't happen! Thanks. I'll check that out. I did make a last minute change to move most of the processing for that all to one place from the multiple places it had ended up previously.

Honestly, I'd be more than happy to buy you a beer if we ever meet IRL - you can get the 2nd round in :grinning: :beers:

1 Like