I've been using UI_Builder and a certain problem is causing me a certain amount of trouble.
The plugin works well although I still don't know all the features it is very good.
But my application: It is distributed in several places. And for that I use the flow-forge system here from our creator.
These locations will use flowforge-device-agent.
This flow-forge-agent works well for updating system versions. But the UI_Builder has HTML files that are static on the machine and are not together with the node-red json. This means that every time I update the system in terms of HTML, CSS, JS files, I need to manually take these files.
Do you have any suggestions or is this just a future plugin update?
I have spoken with Julian on this a couple of Months back.
Currently, uibuilder node uses localfilesystem - something many (most) cloud based systems do not have.
I did propose a kinda VFS (virtual file system) config node that essentially stores the files in the flow but Julian is a busy man I'm afraid.
At this point in time, uibuilder is less usable for you. (i.e. you have to manage local copies yourself)
He is, but I remember I put out a possible alternative approach that I think you were thinking about?
I don't have a Flowforge account so hard for me to fully understand the issues. Though uibuilder does rely on various files, so does Node-RED and by default, uibuilder puts its files into the userDir folder that is also used by other Node-RED files. So if Node-RED works, so should uibuilder. Of course, you can also move the uibuilder files elsewhere if needed but the point is that uibuilder doesn't do anything that Node-RED itself doesn't also do in terms of files. It even uses the same libraries.
There is free months trial with your name on it but dont join right now if you dont have time to spend
In cloud env, node-red flows & associated files are (via a plugin/storage driver) stored in database.
There IS file storage but it is volatile - so every time the container is restarted, all file work is lost / discarded. That includes NPM modules but since package.json is saved/loaded via the storage driver, it survives and any nodes installed by the user get reinstalled. That includes UIBuilder.
However, for uibuilder, that means while the nodes and settings are saved/restored (due to storage driver backed by database), the written files (those written to fs by uibuilder) are lost - since there is no actual fs bound to the container & these are not (cannot) be mapped to the storage driver.
This is why dashboard works (all templates/nodes settings etc) are stored in the flows.
Hope that makes sense (and sheds some light) on why I suggested a kinda vfs (virtual file system) that reads from/writes users js/html/css files to a custom config node - since that would get stringified and stored in the flows.
So settings.js isn't saved? I assume that retained context storage also not available (also uses the fs - or is that also intercepted)?
I could envisage ways to deal with this issue but it would be a lot of work.
I removed the library management from the main Node-RED package.json because the libraries are not actually a node-red feature. They exist purely for the front-end, not the node-red back-end.
Similarly, one of the early design assumptions for uibuilder was that web development already has excellent tools and development workflows so that uibuilder should support those. Node-RED provides the data management more than the front-end code management. As you know, the zero-code features of uibuilder are relatively new, having been introduced only in the last 6-10 months or so.
The model for uibuilder has existed now for over a decade and this is the first time that this issue has been raised. So I'm not sure that this is really a uibuilder issue or a commercial FlowForge issue? I'm happy to try to make things work but, at the end of the day, I don't get paid for uibuilder work so time is necessarily limited.
settings.js?
Happy to consider this but it is a new area for me. If there is someone who can give a steer in regard to a VFS library that could be used, I would certainly consider it.
But this would still need to work in a way that continues to support the majority requirement which is that front-end work continues to support existing development workflows. Including build steps, 3rd-party tooling, etc. In this way, uibuilder acts as a bridge that helps people deliver data-driven web apps without having to ditch their existing front-end development knowledge.
So yes, I remain open to further discussion but this isn't an easy problem and I don't think there are easy solutions unless a VFS solution can be implemented that intercepts FS calls.
Of course settings.js is also handled by the cloud provider (not just flowforge)
This may prove to be simpler than you suspect?
For example, using memfs (which is API compatible with fs) you would simply use the vfs vol in place of fs then serialise the file system, add it to your uibuilder instance & all files will be shared with export / import / stored in the flow.
Here is a demo:
const {fs: vfs, vol} = require('memfs')
const snapshot = require('memfs/lib/snapshot')
const fs = require('fs')
const { assert } = require('console')
// fake HTML files for uibuilder
const image1Png = 'iVBORw0KGgoAAAANSUhEUgAAABgAAAAYCAYAAADgdz34AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAApgAAAKYB3X3/OAAAABl0RVh0U29mdHdhcmUAd3d3Lmlua3NjYXBlLm9yZ5vuPBoAAANCSURBVEiJtZZPbBtFFMZ/M7ubXdtdb1xSFyeilBapySVU8h8OoFaooFSqiihIVIpQBKci6KEg9Q6H9kovIHoCIVQJJCKE1ENFjnAgcaSGC6rEnxBwA04Tx43t2FnvDAfjkNibxgHxnWb2e/u992bee7tCa00YFsffekFY+nUzFtjW0LrvjRXrCDIAaPLlW0nHL0SsZtVoaF98mLrx3pdhOqLtYPHChahZcYYO7KvPFxvRl5XPp1sN3adWiD1ZAqD6XYK1b/dvE5IWryTt2udLFedwc1+9kLp+vbbpoDh+6TklxBeAi9TL0taeWpdmZzQDry0AcO+jQ12RyohqqoYoo8RDwJrU+qXkjWtfi8Xxt58BdQuwQs9qC/afLwCw8tnQbqYAPsgxE1S6F3EAIXux2oQFKm0ihMsOF71dHYx+f3NND68ghCu1YIoePPQN1pGRABkJ6Bus96CutRZMydTl+TvuiRW1m3n0eDl0vRPcEysqdXn+jsQPsrHMquGeXEaY4Yk4wxWcY5V/9scqOMOVUFthatyTy8QyqwZ+kDURKoMWxNKr2EeqVKcTNOajqKoBgOE28U4tdQl5p5bwCw7BWquaZSzAPlwjlithJtp3pTImSqQRrb2Z8PHGigD4RZuNX6JYj6wj7O4TFLbCO/Mn/m8R+h6rYSUb3ekokRY6f/YukArN979jcW+V/S8g0eT/N3VN3kTqWbQ428m9/8k0P/1aIhF36PccEl6EhOcAUCrXKZXXWS3XKd2vc/TRBG9O5ELC17MmWubD2nKhUKZa26Ba2+D3P+4/MNCFwg59oWVeYhkzgN/JDR8deKBoD7Y+ljEjGZ0sosXVTvbc6RHirr2reNy1OXd6pJsQ+gqjk8VWFYmHrwBzW/n+uMPFiRwHB2I7ih8ciHFxIkd/3Omk5tCDV1t+2nNu5sxxpDFNx+huNhVT3/zMDz8usXC3ddaHBj1GHj/As08fwTS7Kt1HBTmyN29vdwAw+/wbwLVOJ3uAD1wi/dUH7Qei66PfyuRj4Ik9is+hglfbkbfR3cnZm7chlUWLdwmprtCohX4HUtlOcQjLYCu+fzGJH2QRKvP3UNz8bWk1qMxjGTOMThZ3kvgLI5AzFfo379UAAAAASUVORK5CYII='
const styleCss = 'body { background-color: red; }'
const mainJs = 'console.log("Hello World");'
const mainHtml = '<html><head><link rel="stylesheet" href="style.css"></head><body><script src="main.js"></script><img src="image1.png"></body></html>'
// fake NodeRED flow
const NodeREDFlow = [
{"id":"1111aaaa","type":"tab","label":"Flow 1","disabled":false},
{"id":"2222bbbb","type":"uibuilder","z":"1111aaaa","name":"","x":110,"y":80,"fs":"", "wires":[[]]},
]
// init vfs
vol.fromJSON({}, '/')
// add files to memfs
vol.writeFileSync('/image1.png', image1Png)
vol.writeFileSync('/style.css', styleCss)
vol.writeFileSync('/main.js', mainJs)
vol.writeFileSync('/index.html', mainHtml)
console.log('files in vfs vol', vol.readdirSync('/')) // []
// store mem fs in flow JSON
const snap1 = snapshot.toBinarySnapshotSync({fs: vol, dir:'/'});
const buf1Snapb64 = Buffer.from(snap1).toString('base64')
NodeREDFlow[1].fs = buf1Snapb64
// write flow JSON to file
fs.writeFileSync('flow.json', JSON.stringify(NodeREDFlow))
// clear vol
vol.reset();
console.log('files in vfs vol after reset', vol.readdirSync('/')) // []
// read flow JSON from file
const NodeREDFlow2 = JSON.parse(fs.readFileSync('flow.json'))
console.log('uibuilder fs', NodeREDFlow2[1].fs) // the filesystem is stored as base64 string
// restore mem fs from flow JSON
const fsData = NodeREDFlow2[1].fs
const buf = Buffer.from(fsData, 'base64')
snapshot.fromBinarySnapshotSync(buf, {fs: vol, dir: '/'});
// console log files in memfs
console.log('files in vfs vol after restore', vol.readdirSync('/')) // [ 'image1.png', 'index.html', 'main.js', 'style.css' ]
// assert image file is OK
const vfsImg = vol.readFileSync('/image1.png')
assert(vfsImg.length === image1Png.length, `vfsImg.length (${vfsImg.length}) === (${image1Png.length}) image1Png.length`) // OK
console.log('Done');
This is the output
files in vfs vol [ 'image1.png', 'style.css', 'main.js', 'index.html' ]
files in vfs vol after reset []
uibuilder fs gwCgpHgKaW1hZ2UxLnBuZ4MBoFkFCGlWQk9SdzBLR2dvQUFBQU5TVWhFVWdBQUFCZ0FBQUFZQ0FZQUFBRGdkejM0QUFBQUJITkNTVlFJQ0FnSWZBaGtpQUFBQUFsd1NGbHpBQUFBcGdBQUFLWUIzWDMvT0FBQUFCbDBSVmgwVTI5bWRIZGhjbVVBZDNkM0xtbHVhM05qWVhCbExtOXlaNXZ1U
EJvQUFBTkNTVVJCVkVpSnRaWlBiQnRGRk1aL003dWJYZHRkYjF4U0Z5ZWlsQmFweVNWVThoOE9vRmFvb0ZTcWlpaElWSXBRQktjaTZLRWc5UTZIOWtvdklIb0NJVlFKSkNLRTFFTkZqbkFnY2FTR0M2ckVueEJ3QTA0VHg0M3QyRm52REFmamtOaWJ4Z0h4bldiMmUvdTk5MmJlZTd0Q2EwMFlGc2ZmZWtGWSt
uVXpGdGpXMExydmpSWHJDRElBYVBMbFcwbkhMMFNzWnRWb2FGOThtTHJ4M3BkaE9xTHRZUEhDaGFoWmNZWU83S3ZQRnh2Umw1WFBwMXNOM2FkV2lEMVpBcUQ2WFlLMWIvZHZFNUlXcnlUdDJ1ZExGZWR3YzErOWtMcCt2YmJwb0RoKzZUa2x4QmVBaTlUTDB0YWVXcGRtWnpRRHJ5MEFjTytqUTEyUnlvaHFxb
1lvbzhSRHdKclUrcVhrald0Zmk4WHh0NThCZFF1d1FzOXFDL2FmTHdDdzh0blFicVlBUHNneEUxUzZGM0VBSVh1eDJvUUZLbTBpaE1zT0Y3MWRIWXgrZjNOTkQ2OGdoQ3UxWUlvZVBQUU4xcEdSQUJrSjZCdXM5NkN1dFJaTXlkVGwrVHZ1aVJXMW0zbjBlRGwwdlJQY0V5c3FkWG4ranNRUHNySE1xdUdlWEV
hWTRZazR3eFdjWTVWLzlzY3FPTU9WVUZ0aGF0eVR5OFF5cXdaK2tEVVJLb01XeE5LcjJFZXFWS2NUTk9hanFLb0JnT0UyOFU0dGRRbDVwNWJ3Q3c3QldxdWFaU3pBUGx3amxpdGhKdHAzcFRJbVNxUVJyYjJaOFBIR2lnRDRSWnVOWDZKWWo2d2o3TzRURkxiQ08vTW4vbThSK2g2cllTVWIzZWtva1JZNmYvW
XVrQXJOOTc5amNXK1YvUzhnMGVUL04zVk4za1RxV2JRNDI4bTkvOGswUC8xYUloRjM2UGNjRWw2RWhPY0FVQ3JYS1pYWFdTM1hLZDJ2Yy9UUkJHOU81RUxDMTdNbVd1YkQybktoVUtaYTI2QmEyK0QzUCs0L01OQ0Z3ZzU5b1dWZVloa3pnTi9KRFI4ZGVLQm9EN1krbGpFakdaMHNvc1hWVHZiYzZSSGlycjJ
yZU55MU9YZDZwSnNRK2dxams4VldGWW1IcndCelcvbit1TVBGaVJ3SEIySTdpaDhjaUhGeElrZC8zT21rNXRDRFYxdCsybk51NXN4eHBERk54K2h1TmhWVDMvek1Eejh1c1hDM2RkYUhCajFHSGovQXMwOGZ3VFM3S3QxSEJUbXlOMjl2ZHdBdysvd2J3TFZPSjN1QUQxd2kvZFVIN1FlaTY2UGZ5dVJqNElrO
WlzK2hnbGZia2JmUjNjblptN2NobFVXTGR3bXBydENvaFg0SFV0bE9jUWpMWUN1K2Z6R0pIMlFSS3ZQM1VOejhiV2sxcU14akdUT01UaFoza3ZnTEk1QXpGZm8zNzlVQUFBQUFTVVZPUks1Q1lJST14CXN0eWxlLmNzc4MBoFgfYm9keSB7IGJhY2tncm91bmQtY29sb3I6IHJlZDsgfXgHbWFpbi5qc4MBoFg
bY29uc29sZS5sb2coIkhlbGxvIFdvcmxkIik7eAppbmRleC5odG1sgwGgWIQ8aHRtbD48aGVhZD48bGluayByZWw9InN0eWxlc2hlZXQiIGhyZWY9InN0eWxlLmNzcyI+PC9oZWFkPjxib2R5PjxzY3JpcHQgc3JjPSJtYWluLmpzIj48L3NjcmlwdD48aW1nIHNyYz0iaW1hZ2UxLnBuZyI+PC9ib2R5Pjwva
HRtbD4=
files in vfs vol after restore [ 'image1.png', 'style.css', 'main.js', 'index.html' ]
Done
OK, I will add to the backlog and look at this in earnest when I've got the next release out.
Folk may already have noticed that, after a prolific 6-12 months of development, I've slowed down significantly recently. I needed a bit of a break to recharge but I'm also in the middle of some massive changes at work which has left me with less energy for side-projects for a bit. I'll get back to it because I'm nowhere near done with improving uibuilder. uibuilder is here for the long-haul, it has completed a decade and has plenty more to offer.
I guess this all poses a bigger question - what is the general direction that flowforge are going in terms of a Dashboard ??
We all know the issues with the current Dashboard - so there must be something being planned for in the back end (From the FLowForge side of Nick's brain rather than the Node Red Side !) if you are building commercial solutions that will hopefully stand the test of the time ?
This could well influence how much time and effort Julian would put into uib and getting it to work withe the flowforge deployment paradigm (once he has recharged his batteries !)
We believe in the power of the community, and we want your feedback. If there are features you'd like to see or improvements you think can be made, we invite you to open a Github issue.
