Johannes (@JGKK) and I are creating a suite of nodes to integrate voice2json into Node-RED. These nodes allow the user to setup voice2json (entirely via Node-RED), without having to edit the voice2json configuration files manually. However this means that the properties of our nodes are stored in the external voice2json configuration files, instead of in Node-RED's flow.json file.
An extra requirement is that the voice2json configuration files can be updated by a background process, without our node knowing it ...
We got something running already, but I'm not sure whether we are heading in the right direction...
Let's first have a look at a normal situation (without external files):
When the flow editor is started, the node configuration is transferred from the server to the client.
When the node’s config screen is opened, the configuration is displayed.
The user can change the configuration in the node’s config screen.
When the user clicks the ‘Done’ button (not ‘Cancel’) in the node’s config screen, the configuration is updated (on the client side). And the ‘Deploy’ button will become active automatically (since the node is 'dirty').
When the user clicks the ‘Deploy’ button, the node configuration is transferred from the client to the server (and stored in the flow.json file).
The server-side configuration is now up to date.
The run-time javascript code of our node will be executed, so the node's server-side code can work with the updated configuration.
But have a look what happens when the node's config is stored in the voice2json configuration files:
In our case the configuration is stored in the voice2json configuration files.
We could load those config files from the server (via a http endpoint) into the node’s config screen. This can be done automatically (when the config screen opens), but then the node suddenly becomes dirty (and Deploy button active) which seems weird behaviour since the user hasn't changed anything. So makes more sense to have a "load config" button ...
And when the configuration is changed in the config screen, the updated configuration can be written again to the files. Automatically or via a "Save" button.
This is very straightforward, however this way we entirely circumvent the Node-RED mechanism (no undo, no redo, the node doesn’t become dirty, the deploy button doesn’t become active, nothing is deployed, there is no automatic merge, …). And in the live stream yesterday, it was mentioned that Node-RED could offer in the far future mechanisms to let users see real-time what other users are changing. This kind of cool stuff will never work together with our mechanism...
So I would like to do it the Node-RED way, but don't know how
Suppose we store the configuration data (i.e. the files contents) in the node by adding a node.fileContents=[] array variable.
As soon as the changes have been deployed (via the Deploy button), the node can store the changes into the files (see step 12). So don't think there is a problem here. P.S. the data will be stored both in the flow.json file and in the voice2json files.
But don’t think it is possible to load the configuration from the files (see step 13), as soon as the configuration is loaded (step 1) into the flow editor. Which we really would need, since the configuration in the flow.json file can be out of sync with the voice2json files.
When you deploy changes and somebody else has meanwhile changed the same nodes, Node-RED will show a "Merge" dialog. However when a background process has meanwhile changed the configuration files, those changes will be overwritten by the config of the node. So the user will not be aware that there has been a conflict, and there will be no Merge dialog box .
Not an easy problem. But still hoping for some good advice...
You could load the config file when the node dialog is opened via oneditprepare and an http endpoint.
Then the node dialog can be used to change the config, the node gets dirty and if deployed, the changes are saved to the json file again.
You do not have node-red take control over those values, therefore, the node does not get dirty if the config is just loaded
Hi @cinhcet,
Thanks!!!! Yes indeed that is exactly what we currently do (see step 9). But like you say we can save it via the normal way (so step 4 + 5 + 12), which indeed makes more sense to me compared to saving the data via the http endpoint (step 10 + 11).
But it has some disadvantages:
You open the config screen but you don't change anything, and the Deploy button becomes active. That is weird behaviour.
When the voice2json files have been changed meanwhile by a background process, we will overwrite those changes without a warning.
I do not fully understand why the deploy button becomes active when you just load the configuration into the form elements, which not necessarily need to have an "node-input-blabla" id. And my understanding so far was, but I can be wrong, that if they do not have this id, then changing their values does not make the node dirty, hence loading a configuration would not cause the deploy button to become active.
But I see that the undo and merge problem still would exist
When I want to store them on the server via the Node-RED deploy mechanism, I'm pretty sure I need the "node-input-blabla" id. Because then Node-RED will store the input of those screen fields in my node's config. Node-RED sees that my node config has changed, which means my node will be marked as dirty. As soon as there are dirty nodes, the "Deploy" button will become active. Otherwise I cannot deploy my changes to the server...
So it is good that our node becomes dirty, but it is very confusing for users since they haven't changed anything manually.
On the other hand in that case we can deploy, undo and redo (repetitive) those changes. Therefore I like this much more compared to saving the config via htttp endpoint.
But if the voice2json files have been change meanwhile, our users will never get a Merge dialog popup.
I quickly checked if you do not have the property "blabla" in the "default" property of the node, then if your form element has an id of "node-input-blabla" and gets updated in oneditprepare, the node does not get dirty. So then you could manually check in oneditsave whether something has changed and then save the file. However, as you say, you would not get node reds undo etc. behavior.
What about maintaining the config both in the flows file and also in the external file and then in oneditprepare you load the external file, do a deep compare (I do not know how complex this json file is) and if it differs, you update the node-config, which in this case then makes even sense to become dirty?
The config is not a json or something.
Its a description of voice commands and intents written in a simplified jsgf grammar. So doing a compare in oneditprepare would be a big undertaking.
When saving back to the file from oneditsave we would again have the problem what is if you have the config screen open in two sessions at once for some reason.
The user opens the node's config screen, so the config file needs to be loaded via the http endpoint.
The user changes the values in the config screen and presses the 'Done' button.
The users *opens the node's config screen again multiple times. Now we must not load the config files again, otherwise the changes (from the previous) step would be overwritten.
Sound like you need to so server side or client side comparison for your own "dirty" flagging Bart.
I know @JGKKSaid it would be a big undertaking but if it's doable, might be a solution. Perhaps have a checksum and/or modified date and/or version number in the configs (both client UI side and server/current running config side) for simpler comparison if necessary.
could you maybe give more details about what exactly is stored and what is loaded? Because I assumed you load the complete config file via the http endpoint, but if it is big or what ever, it seems you do not do that.
Morning Steve,
Suppose we could easily check ourselves whether our node is dirty or not. It is not clear to me how (client-side or server-side) this could solve one of our problems? After our oneditsave has been called Node-RED will determine by itself whether our node is dirty, to enable the 'Deploy' button. So where does our dirty checking fit into this picture then?
Or do you mean that we somehow can trigger - with a server side dirty check - something like a Merge dialog (so we can merge our config node data with the current data in the voice2json files). But I'm pretty sure that isn't possible in Node-RED...
Damn I would really like to have these problems solved in a Node-RED compatible way. Because now I am becoming the bottleneck of these voice2json nodes, and @JGKK needs to wait until this techie has found a setup which fits my needs ...
Have a hidden field & matching property (like a number or text field) that you change when you determine "dirty". Then the normal mechanisms of node-red take over.
Thought that hidden fields were not stored by Node-RED into the node. Woukd be nice if you could describe it in more detail.
P.S. I'm not on my computer today, so cannot try anything related to this discussion... I'm preparing a swimming pool, since we have to stay home this summer...
NOTE: the above really only needs to be <input type="hidden" id="node-input-last_modified"> but i wanted you to understand what I meant by putting it into context
Hey Steve,
Thanks for your time !!!
Now I remember that (for my xterm node) I had used RED.nodes.dirty(true); to specify that 'some' node has been changed. Worked well, but not sure anymore if it was a public API.
Yes indeed. When we keep our configuration as a node variable, and update the voice2json configuartion files in our server-side node (after a deploy) then we can use the deploy/undo/redo mechanisms of Node-RED. But it is a pity that we cannot do the same while loading the node from the server to the client: don't think we can read the voice2json configuration files, as soon as our node is read to the client...
ok, another try:
You have the configuration as node variables and part of the node config editor.
When the editor is opened, you load the voice2json configuration files and compare them to the state in the config editor (I know you said that this is difficult, but I fear that there is no other way).
Only those that are different are changed, meaning you set the values of the node editor forms. Then the node gets dirty automatically, when a user hits deploy, the node is restarted on the server side and there you save the config to the voice2json config file.
This would mean that the undo mechanism would work, and it would also work if the user opens the editor multiple times before a deploy, since you could not only compare to the node.config object, but also to the current editor state (which is the recommended way).
.