UI_Builder and FlowForge-Device-Agent

I've been using UI_Builder and a certain problem is causing me a certain amount of trouble.

The plugin works well although I still don't know all the features it is very good.

But my application: It is distributed in several places. And for that I use the flow-forge system here from our creator.

These locations will use flowforge-device-agent.

This flow-forge-agent works well for updating system versions. But the UI_Builder has HTML files that are static on the machine and are not together with the node-red json. This means that every time I update the system in terms of HTML, CSS, JS files, I need to manually take these files.

Do you have any suggestions or is this just a future plugin update?

I have spoken with Julian on this a couple of Months back.
Currently, uibuilder node uses localfilesystem - something many (most) cloud based systems do not have.

I did propose a kinda VFS (virtual file system) config node that essentially stores the files in the flow but Julian is a busy man I'm afraid.

At this point in time, uibuilder is less usable for you. (i.e. you have to manage local copies yourself)

He is, but I remember I put out a possible alternative approach that I think you were thinking about?

I don't have a Flowforge account so hard for me to fully understand the issues. Though uibuilder does rely on various files, so does Node-RED and by default, uibuilder puts its files into the userDir folder that is also used by other Node-RED files. So if Node-RED works, so should uibuilder. Of course, you can also move the uibuilder files elsewhere if needed but the point is that uibuilder doesn't do anything that Node-RED itself doesn't also do in terms of files. It even uses the same libraries.

There is free months trial with your name on it :wink:
but dont join right now if you dont have time to spend

In cloud env, node-red flows & associated files are (via a plugin/storage driver) stored in database.

There IS file storage but it is volatile - so every time the container is restarted, all file work is lost / discarded. That includes NPM modules but since package.json is saved/loaded via the storage driver, it survives and any nodes installed by the user get reinstalled. That includes UIBuilder.

However, for uibuilder, that means while the nodes and settings are saved/restored (due to storage driver backed by database), the written files (those written to fs by uibuilder) are lost - since there is no actual fs bound to the container & these are not (cannot) be mapped to the storage driver.

This is why dashboard works (all templates/nodes settings etc) are stored in the flows.

Hope that makes sense (and sheds some light) on why I suggested a kinda vfs (virtual file system) that reads from/writes users js/html/css files to a custom config node - since that would get stringified and stored in the flows.

So settings.js isn't saved? I assume that retained context storage also not available (also uses the fs - or is that also intercepted)?

I could envisage ways to deal with this issue but it would be a lot of work.

I removed the library management from the main Node-RED package.json because the libraries are not actually a node-red feature. They exist purely for the front-end, not the node-red back-end.

Similarly, one of the early design assumptions for uibuilder was that web development already has excellent tools and development workflows so that uibuilder should support those. Node-RED provides the data management more than the front-end code management. As you know, the zero-code features of uibuilder are relatively new, having been introduced only in the last 6-10 months or so.

The model for uibuilder has existed now for over a decade and this is the first time that this issue has been raised. So I'm not sure that this is really a uibuilder issue or a commercial FlowForge issue? I'm happy to try to make things work but, at the end of the day, I don't get paid for uibuilder work so time is necessarily limited.


Happy to consider this but it is a new area for me. If there is someone who can give a steer in regard to a VFS library that could be used, I would certainly consider it.

But this would still need to work in a way that continues to support the majority requirement which is that front-end work continues to support existing development workflows. Including build steps, 3rd-party tooling, etc. In this way, uibuilder acts as a bridge that helps people deliver data-driven web apps without having to ditch their existing front-end development knowledge.

So yes, I remain open to further discussion but this isn't an easy problem and I don't think there are easy solutions unless a VFS solution can be implemented that intercepts FS calls.

Of course settings.js is also handled by the cloud provider (not just flowforge)

This may prove to be simpler than you suspect?

For example, using memfs (which is API compatible with fs) you would simply use the vfs vol in place of fs then serialise the file system, add it to your uibuilder instance & all files will be shared with export / import / stored in the flow.

Here is a demo:

const {fs: vfs, vol} = require('memfs')
const snapshot = require('memfs/lib/snapshot')
const fs = require('fs')
const { assert } = require('console')

// fake HTML files for uibuilder
const image1Png = 'iVBORw0KGgoAAAANSUhEUgAAABgAAAAYCAYAAADgdz34AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAApgAAAKYB3X3/OAAAABl0RVh0U29mdHdhcmUAd3d3Lmlua3NjYXBlLm9yZ5vuPBoAAANCSURBVEiJtZZPbBtFFMZ/M7ubXdtdb1xSFyeilBapySVU8h8OoFaooFSqiihIVIpQBKci6KEg9Q6H9kovIHoCIVQJJCKE1ENFjnAgcaSGC6rEnxBwA04Tx43t2FnvDAfjkNibxgHxnWb2e/u992bee7tCa00YFsffekFY+nUzFtjW0LrvjRXrCDIAaPLlW0nHL0SsZtVoaF98mLrx3pdhOqLtYPHChahZcYYO7KvPFxvRl5XPp1sN3adWiD1ZAqD6XYK1b/dvE5IWryTt2udLFedwc1+9kLp+vbbpoDh+6TklxBeAi9TL0taeWpdmZzQDry0AcO+jQ12RyohqqoYoo8RDwJrU+qXkjWtfi8Xxt58BdQuwQs9qC/afLwCw8tnQbqYAPsgxE1S6F3EAIXux2oQFKm0ihMsOF71dHYx+f3NND68ghCu1YIoePPQN1pGRABkJ6Bus96CutRZMydTl+TvuiRW1m3n0eDl0vRPcEysqdXn+jsQPsrHMquGeXEaY4Yk4wxWcY5V/9scqOMOVUFthatyTy8QyqwZ+kDURKoMWxNKr2EeqVKcTNOajqKoBgOE28U4tdQl5p5bwCw7BWquaZSzAPlwjlithJtp3pTImSqQRrb2Z8PHGigD4RZuNX6JYj6wj7O4TFLbCO/Mn/m8R+h6rYSUb3ekokRY6f/YukArN979jcW+V/S8g0eT/N3VN3kTqWbQ428m9/8k0P/1aIhF36PccEl6EhOcAUCrXKZXXWS3XKd2vc/TRBG9O5ELC17MmWubD2nKhUKZa26Ba2+D3P+4/MNCFwg59oWVeYhkzgN/JDR8deKBoD7Y+ljEjGZ0sosXVTvbc6RHirr2reNy1OXd6pJsQ+gqjk8VWFYmHrwBzW/n+uMPFiRwHB2I7ih8ciHFxIkd/3Omk5tCDV1t+2nNu5sxxpDFNx+huNhVT3/zMDz8usXC3ddaHBj1GHj/As08fwTS7Kt1HBTmyN29vdwAw+/wbwLVOJ3uAD1wi/dUH7Qei66PfyuRj4Ik9is+hglfbkbfR3cnZm7chlUWLdwmprtCohX4HUtlOcQjLYCu+fzGJH2QRKvP3UNz8bWk1qMxjGTOMThZ3kvgLI5AzFfo379UAAAAASUVORK5CYII='
const styleCss = 'body { background-color: red; }'
const mainJs = 'console.log("Hello World");'
const mainHtml = '<html><head><link rel="stylesheet" href="style.css"></head><body><script src="main.js"></script><img src="image1.png"></body></html>'

// fake NodeRED flow
const NodeREDFlow = [
    {"id":"1111aaaa","type":"tab","label":"Flow 1","disabled":false},
    {"id":"2222bbbb","type":"uibuilder","z":"1111aaaa","name":"","x":110,"y":80,"fs":"", "wires":[[]]},

// init vfs
vol.fromJSON({}, '/')

// add files to memfs
vol.writeFileSync('/image1.png', image1Png)
vol.writeFileSync('/style.css', styleCss)
vol.writeFileSync('/main.js', mainJs)
vol.writeFileSync('/index.html', mainHtml)
console.log('files in vfs vol', vol.readdirSync('/')) // []

// store mem fs in flow JSON
const snap1 = snapshot.toBinarySnapshotSync({fs: vol, dir:'/'});
const buf1Snapb64 = Buffer.from(snap1).toString('base64')
NodeREDFlow[1].fs = buf1Snapb64

// write flow JSON to file
fs.writeFileSync('flow.json', JSON.stringify(NodeREDFlow))

// clear vol
console.log('files in vfs vol after reset', vol.readdirSync('/')) // []

// read flow JSON from file
const NodeREDFlow2 = JSON.parse(fs.readFileSync('flow.json'))
console.log('uibuilder fs', NodeREDFlow2[1].fs) // the filesystem is stored as base64 string

// restore mem fs from flow JSON
const fsData = NodeREDFlow2[1].fs
const buf = Buffer.from(fsData, 'base64')
snapshot.fromBinarySnapshotSync(buf, {fs: vol, dir: '/'});

// console log files in memfs
console.log('files in vfs vol after restore', vol.readdirSync('/')) // [ 'image1.png', 'index.html', 'main.js', 'style.css' ]

// assert image file is OK
const vfsImg = vol.readFileSync('/image1.png')
assert(vfsImg.length === image1Png.length, `vfsImg.length (${vfsImg.length}) === (${image1Png.length}) image1Png.length`) // OK


This is the output

files in vfs vol [ 'image1.png', 'style.css', 'main.js', 'index.html' ]
files in vfs vol after reset []
files in vfs vol after restore [ 'image1.png', 'style.css', 'main.js', 'index.html' ]

OK, I will add to the backlog and look at this in earnest when I've got the next release out.

Folk may already have noticed that, after a prolific 6-12 months of development, I've slowed down significantly recently. I needed a bit of a break to recharge but I'm also in the middle of some massive changes at work which has left me with less energy for side-projects for a bit. I'll get back to it because I'm nowhere near done with improving uibuilder. uibuilder is here for the long-haul, it has completed a decade and has plenty more to offer.



I guess this all poses a bigger question - what is the general direction that flowforge are going in terms of a Dashboard ??

We all know the issues with the current Dashboard - so there must be something being planned for in the back end (From the FLowForge side of Nick's brain rather than the Node Red Side !) if you are building commercial solutions that will hopefully stand the test of the time ?

This could well influence how much time and effort Julian would put into uib and getting it to work withe the flowforge deployment paradigm (once he has recharged his batteries !)


There has recently been an announcement Craig: The Next Step in Data Visualization - Announcing the Successor to the Node-RED Dashboard • FlowFuse

OH Wow - OK thats great news then

Hopefully the guys will look at what has been done with Flexdash and UIB


I hope they do not miss out multi-user and multi-tenant this time

From the article:

We believe in the power of the community, and we want your feedback. If there are features you'd like to see or improvements you think can be made, we invite you to open a Github issue.

Don't be shy now Walter :wink:

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.