I need to save a pile of JSON in a node I'm publishing (this is for FlexDash and the data is the configuration of the dashboard). I'm a bit at a loss about the best method for saving. I've contemplated three options:
Use a persistent context store. The mechanics are fairly simple, but rather user unfriendly because there is no persistent context store by default and my node needs to know the name given to any existing store. So when someone installs my node, I first need to instruct them to locate and edit the Node-RED settings.js file to add a persistent context store and then restart Node-RED. Then, when they instantiate a node, I need to have a field for the name of the persistent context store and they need to fill that in correctly. In the end, my node has no way to verify that all these steps were performed correctly (the context store will undetectably? use the default store if the named store does not exist).
Use a file. Again, the mechanics are fairly simple and the file is presumably best placed in the data directory. However, I seem to remember some issues with some Node-RED installs not having a writable filesystem, or a writable data directory, or something like that. I can't find the details.
Store the data in the flows.json by using the Node-RED API to update a flow. The mechanics of this are unclear to me, but with enough digging I'm sure I can make it work. However, my node is actually a config node and thus doesn't belong to any specific flow. With projects and all that I believe there are multiple flow files. It all sounds fraught with problems but maybe it's just my ignorance.
If it is a real config node that its then used by other (widget) nodes then I think option 3 is the way to go.... it is similar to what the existing dashboard does - the ui-base node is that config node that all others use - if you look at a flow using the dashboard then you will see the ui-base node there with all the theme and site setup in it. Similar to any (for example) mqtt-broker definitions etc...
Config nodes are a "known" thing so when you export a flow from a tab then we make sure that any related config nodes also gets exported.
There is (currently) only one overall flow file... - it just has all the flow tabs and config nodes within it...
(there is some discussion about possibly splitting it but that's not on the cards right now so don't worry about it as we would need to migrate existing flows over anyway)
It largely depends on what the lifecycle of that data is and when it needs to be written.
The general principle is that all persistent flow configuration should be in the flow file. But another important point is the flow file should only get updated when a user hits Deploy - nodes should not unilaterally deploy changes to the flow configuration themselves.
If this is runtime state that is generated by the node, then it absolutely should not be in the flow file. Context is what we provide for that - and yes, that relies on the user having setup persistence.
We are currently looking at the Storage API of the Node-RED runtime ahead of the 3.0 release. There is a requirement around being able to persist node state (such as oauth tokens that have to be periodically refreshed). An expanded Storage API could provide a means for nodes to store runtime state in a way that doesn't rely on context, and can be adapted to whatever storage provider the runtime is using.
Is there a way for a node to mark itself as modified, i.e., needing a redeploy? Or a way to tell the flow editor that a node should be shown as such and that the deploy button should be activated?
Aren't you saying that there is a flow API to modify the flow, but that the flow should should not actually be modified unless the user hits deploy, and that it's not possible to cause the deploy button to be activated? Or are you saying that anything that modifies the flow needs to display its own deploy button?
The user owns the flow configuration and owns how/when it is updated. That might be via the actions they take in the editor, or it might be via some remote administration via the HTTP Admin API.
Nodes should not modify the running flow configuration.
You didn't mention that each element in the tab layout (ui_tab and ui_group) is also a config node and that you manage all the relationships (and the way the relationships are represented) so export/import works... That's quite a pile of code...
Yup! And as far as I know (very little...) there's no way to hook into the import process, e.g. to provide a choice about which endpoint/site to merge the flexdash config nodes into. So effectively this means that when the flow is redeployed the code that "discovers" these newly imported config nodes needs to somehow make sense of them and provide tools to reconfigure later. Somewhat similar to multiple ui_tabs or ui_groups having the same order value, just a bit more complicated...
Hi @tve,
Then I have misinterpreted this until now. Because in the original discussion you wrote:
You insert data into a data structure (a tree) on the Node-RED side. That is reflected to a tree in the front-end. A widget is then configured to look at specific nodes in that tree to get its values. The configuration itself is also in that tree
So I thought that your tree contained 3 types of information:
The static config, which is entered manually by the user in the fd nodes config screens and then deployed by clicking on the 'Deploy' button.
The dynamic config, which is the static config that is overwritten by params via input messages.
The data, i.e. the state of what you want to display.
I understood that this way you can avoid having to replay messages, and you are still able to sync multiple clients even after a browser window refresh.
But now it is not clear to me anymore which of those 3 types of information are inside your tree, and which of those needs to be persisted? Your static config is already persisted in the flow.json file via the node properties.
And why do you want to persist that entire tree, while you static config is already persisted? To make sure that the dynamic config and the data survive a restart and redeploy?
Isn't that a problem when you make changes in the config screen, and deploy those changes? Because if your tree also contains the static and dynamic config, that might be obsolete and conflict with the new static config in the node?
Would be nice if you could explain this a bit, so I can follow again after a long hard day at work...
Thanks!
Well, all the data is in one big json tree in memory, but that doesn't mean the entire tree gets persisted. The configuration sub-tree needs to be persisted, otherwise, if you restart everything you see an empty dashboard (more or less). But the dynamic data being visualized is really akin to the messages whizzing around in your node-red flows, they should not be persisted, so the tree as a whole is not saved away (in-memory only, just like messages in flows). Does that answer your question?
Right now things are a bit messy... It's something like:
The edit form in NR is populated with the values in the flow.json
You make changes and hit deploy
NR persists the changed config map to flow.json
The fd-whatever node constructor is called with the config map, it possibly adjusts fields, and after calling RED.nodes.createNode(this, config) calls fd.initWidget(this, config), where fd is a handle onto the appropriate FD config node.
initWidget inserts the config into the config portion of the tree, that causes the changes to be broadcast to all connected dashboards, which immediately update
since the config portion of the tree was changed, after a short delay, it is persisted
If you now restart NR, the config node is instantiated first. It restores the config from persistent store, and when a dashboard connects it gets the persisted config. Concurrently, the fd-whatever node is instantiated, which means the constructor is called with the flow.json config, it calls initWidget, which overwrites the config in the FD config node with the same values.
Ideally the FD config node wouldn't persist the config of the widgets since it's already mostly persisted in the flow.json but that's best tackled at the same time as the rest of the config is also stored there.
The only thing this description doesn't cover is what happens when a fd-whatever node receives a message. The code in testgauge describes the first part of that pretty well, I believe:
// handle flow input messages, basically massage them a bit and update the FD widget
this.on("input", msg => {
// prepare update of widget params
const params = typeof msg.params === 'object' ? Object.assign({}, msg.params) : {}
// msg.payload is interpreted as setting the gauge value, i.e. params.value
if ('payload' in msg) params.value = msg.payload
// send the params to the widget
fd.updateWidget(this, params)
})
Now what updateWidget does is a bit convoluted 'cause of the way FD currently works, but the net effect is that the params passed to updateWidget override the config passed to initWidget without erasing the latter. Or put differently, updateWidget stores the dynamic values in an in-memory portion of the tree that overrides the config.
Feel free to start painting on it if something is not correct or missing!
If I understood everything correctly, the red part is what you need to persist in this discussion: a mix of the static config (which is also in the flow.json), and the dynamic config from the input messages.
Yup!
Only the static config gets persisted.
And I wouldn't use "overwrite static config" but "override static config". Overwriting = replacing. Override = having priority, that's an important difference.
For example, suppose a gauge with config min = 0. If you send a message with msg.params.min = -20 then the gauge changes to start at -20. If you now send a message with mgs.params.min = null then the gauge changes back to start at 0. If you redeploy it also changes back to min=0.
Am I correct that you only store the static config (from the config screen) into the persistent store (and not the dynamic config set by the input message params)?
And about the Node-RED restart order:
First your config node loads the static content from the persistent store into the empty json tree
Afterwards the fd_xxx node constructor overwrites that static config again by the static config loaded by Node-RED from the flow.json file
I don't think I understood this correctly, because then the persistent context store would not be useful.
Please illuminate my brain
Yeah, there's duplication for sure... However, the node constructor only overwrites the static config of the widget params, i.e., "inputs" of the widget. There's also size & position of the widgets, and then the config of panels, grids, tabs, plus that of widgets that do not have corresponding NR nodes...
I thought that this was info that you needed to calculate on the fly, without storing it... And isn't that dependent on the client (screen size,...) in a responsive framework?
Ah yes, I had completely forgotten about those. Of course you need to store that info somewhere. Personally it seems to me that this can be stored - as config node properties - in the flow.json file. Because the config nodes are 1-to-1 related to a dashboard.
And I keep forgetting how users can change this kind of config, without a layout manager...
That's quite a pile of code...
