My brain is going in lockdown modus... Could you please explain this a bit more in detail, e.g. with an example.
I 'think' with this approach there are still two issues, due to the fact that the voice2json files can be changed outside of Node-RED:
When you change the config in the node's screen, and you open afterwards again the node's screen. Which config do we need to use: the one from the server (i.e. the voice2json config files) or the update config in the browser.
When you save you updated config in Node-RED, you will overwrite the voice2json config files without having a visual indication that the files have been updated meanwhile. There is no merge dialog box...
While editing a node the this object reflects the current configuration of the node and not the current form element value.
Lets say there is a form element with id "node-input-A" and a corresponding property "A".
The user opens the dialog, in oneditprepare you load the voice2json configuration file, extract the relevant information and send it to the client. Let us assume that there is the "A" property in the voice2json configuration. In the oneditprepare function you now have "A" from voice2json and you have "this.A", which is the value of "A" stored in the flows file and you have $('node-input-A').val() which is the value of the editor. Now you can compare "A" from voice2json to $('node-input-A').val(), if they differ, you set $('node-input-A').val() to "A" from voice2json.
If the user now clicks the done button of the editor and the values have differed, the node gets dirty.
If the user hits deploy, in the constructor of the node you can now save "node.config.A" to the voice2json file.
If the user does not hit deploy immediately, but opens the editor again, $('node-input-A').val() contains already the updated value, which is the same as "A" from voice2json and hence nothing gets updated.
This solution has the problem 2. you are mentioning:
When the voice2json config is changed in between, i.e. after the node editor has been closed (done pressed) but no deploy yet, then you would overwrite the file silently.
By the way, why is the voice2json file changed outside of node-red? And how often?
Argh! This conversation went a long way in a short time! Not had a chance to read the whole thing. However, if you are looking for examples on controlling your node from files or using files to send to the front-end, uibuilder has both of those.
Thanks for tuning in. Its more about how to deal with config files that live outside nodered and can be edited both from nodered or outside of nodered and how to deal with what is when something gets changed in the node config and at the same time it gets changed from the outside. And how to make the file integration the most compatible with s standard nodered workflow and factors like undo in nodered. And multiple people opening the same config from different browsers at the same time.
Did you implement a custom merge dialog or something for when a user edits the frontend files from within nodered and somebody else edits them manually from outside nodered at the same time? So that the changes made from nodered wouldnât interfere with the external changes made to the file in parallel when you deploy them?
Id be very interested in something like that.
I ll definitely take a look at the code of the ui builder node but im afraid ill be in way over my head as i have like no frontend experience
Been a while since I looked at that part of the code. It is certainly a custom merge and you should find the guts of it in the uiblib.js file. I won't be free of race conditions because I don't expect many people to manually edit the file, while that may have been of some use when uibuilder was newer, the features now would mean that there is little point to doing a manual edit. But it may give some ideas.
Not to worry, all of this is in the back-end code. It is all to do with the uibuilder nodes being able to serve up front-end packages. Because there are multiple ways to install packages, I needed something reasonably flexible that will discover them and consistently serve them up via Node-RED's built-in user-facing web server.
I do tend to over-comment my code so hopefully, most of it will be understandable but feel free to get in touch if there is anything you'd like to know.
Sorry, I should have made clear that the merge doesn't use a dialogue as I'm quite happy for users to do things manually and my code doesn't assume that users have done something correctly, it validates the file and the contents each time it is used.
@JGKK is the voice2json expert, so he will be able to explain it better. But some files (e.g. containing all the different statusses of a device) can be updated in background e.g. by a database.
Hi Julian,
Thanks for joining our discussion!
Do you mean this part of your code:
There is a function called checkInstalledPackages that does a comparison with the filesystem.
I'm not using uibuilder at the moment, but is that the behaviour: you cannot leave the node's config screen when it is not in sync with the files on the server's filesystem? But I assume the files can't change between the "Done" button click and the "Deploy" button click (which can be a long time)?
I did suggest that there could be race conditions with the current logic. Let me know if you find anything fixable.
The checkInstalledPackages does quite a bit more than just check things of course. It combines the current list with the master list and optionally with a newly installed package. It then checks whether each entry in the list exists as an installed package under your <userDir>. For each installed package, it works out the installed folder name tries to work out an entry point. It documents all of that in the uib.installedPackages global along with some other useful metadata. It also tries to add the appropriate folder as a static web endpoint. If an entry isn't found but is currently in the list of served endpoints, it "unserve"s the static folder.
This ensures that manual editing of the file as well as changes from the Editor are always reflected correctly.
You will see that checkInstalled Packages is called from about 5 different locations in the processing, not just from the highlighted endpoint. The one instance you highlighted is just handling changes from the Editor but no assumption is made that the list file has remained static in-between.
Hey Steve,
Let's assume I could this comparison working. But at which point could I trigger that editor... That will be the problem, because after a deploy it is too late. Indeed then the state is already stored in the server side, so it is too late to make a decision...
You update the file when you install or remove a package from the Editor, not when you deploy.
Since a deploy will restart the node, the file is re-read to re-set up the served folders. (line 208 if you are looking at the security branch of uibuilder.js). The list of served folders is also torn down when redeploying or when removing uibuilder from your flows so that you don't get duplications or unwanted side effects.
In general, if you want to use an external file and need to update it, do so as soon as possible to reduce the chance of a clash.
In my case, the external file is a convenience as it helps keep things consistent between startups. In most cases, you will want to treat external files as being read only on Node-RED startup, not at other times if it impacts the back-end. In my case, I'm not particularly expecting anyone to change that file externally, all I try to do is minimise any issues if they do. I don't have the resources to test many edge-cases which is why I warned that they likely exist.
Hi @ristomatti,
The problem is that then the background jobs (updating the config files) would also fail, without the user noticing it. However your proposal reveals another weakness: when somebody decides to schedule background jobs to update the files, his manual changes (done via the flow editor) would be overwritten anyway without any kind of notification.
So I think we need to separate our setup into two kind of files:
Files that are created and update manually.
Files that are created and update automatically (via background jobs).
As a result it makes NO sense at all to mix both!!
So I think it should go like this:
The automatic files should be visible in our node's config screen, but read-only. We can easily detect this: we this simply list all files on the filesystem (via an endpoint), and all files that are not available (in our node properties) will be loaded from filesystem and become readonly in our node's screen.
The manual files should be created and maintained from Node-RED. So Node-RED is the MASTER for this data. Which means the data is stored inside our node (i.e. in the flow.json file) and the files will be overwritten at every deploy.
It is the responsibility of our users how to do the setup of the files. Most uses will do everything via our node's config screen, and more advanced users will setup background jobs (and will have the technical knowledge to understand the implications).
This way I think we can use the Node-RED deploy/undo/redo/merge mechanisms without problems.
Unless somebody sees I problem that I have forgotten, this issue is solved for me...
Actually, I don't because generally I'd assume that a) external changes are unlikely and b) having multiple people working on installing packages seems like a very unlikely occurrence.
Undo/redo is feasible. Especially if you keep the external file in a structured format. JSON is ideal for that. The other thing you could do to stop race conditions is to lock the file while you process it which I see @ristomatti has also questioned. Open it early with an exclusive lock. Just make sure that you take care to release the lock if there is an error.
There are some voice2json configuration files on the server, which are automatically updated by background processes (e.g. based on a database). The background job is the master of that data.
And there are some voice2json configuration files on the server, which are manually updated by the user (via the Node-RED flow editor). The Node-RED node is the master of that data.
The node config contains some variables which represent the content of the manually updated files (see 2).
When the flow editor opens, the node config is loaded from the server to the client.
When the node is double clicked, the node config variables will be displayed in the node's config screen. Which means that the content of the manually updated config files can be edited here.
The content of the automatic updated config files is loaded (via an endpoint) and displayed read-only.
The user edits the node config and clicks the "Done" button. From here on the user can "Undo", "Redo" and open the config screen as much as he wants.
When the user clicks the "Deploy" button, the updated node config variables will be stored on the server. The updated node config will now also be stored in Node-RED's flow.json file.
The node's server side javascript code will be executed again, based on the updated node config variables.
The node will store the updated node config variables into the manual config files.
The node will also read the automatic config files.
The node can now execute all its javascript processing, with the latest version of both the automatic config files and the manual config files.
Now the most difficult part of this discussion: I need the blessing from @JGKK
