I think I said the same thing, perhaps not as clearly.
Right, but with the option selected, it survives in those cases too.
It seems like there is no legal way to get information about the module configured for context store(s) in contextStorage if asked from nodes configuration page. (RED.settings.context)
Making smart decisions how to offer the persistence is not possible.
Have you been able to get access from the nodes configuration page Julian.
I do agree that this has been a lively and potentially useful discussion -- thanks for starting it, Paul...
One distinction that I have NOT seen made is what the purpose of the stored data would be -- the initial request was a simple checkbox to save the previously received data points, so the chart could be redrawn easily (i.e. for display purposes). I would not be worried if there were a few dropped points, since it would not influence any downstream systems or controllers...
On the other hand, things like SCADA data that are used to control physical systems need to be handled very carefully to ensure out-of-sync signals do not trigger "bad things" -- but I don't think that's what is being requested here.
Probably 1.5 years ago (on the Google groups, i think), we discussed developing "Caching" nodes that could be used inline to cache msg data (like a ring buffer) and play it back (maybe on some signal, or redeploy). IIRC, it was more for regression testing, but it may also be useful in front of the ui display nodes for just the purpose that the OP envisions. So no changes needed to the chart node itself, and the author can add that behavior by dropping a node in front of the ui node, maybe? I've not looked, but there may well be some contrib-* nodes like this already...
A simple replay from caching node would not work for the chart as it also uses the time at which the message is received.
The chart node in particular does send out its complete array already. There is an example how to use it in the flows library
I have an example caching function for uibuilder that I use. I also have a repo waiting to be written for something more robust as a custom node.
In uibuilder though, I control the cache via control messages that let you do a cache replay or cache clear.
Yes 
Sort of.
You can use the settings property of the registerType function to pass information back to the configuration panel:
In your js file:
RED.nodes.registerType(uib.moduleName, nodeGo, {
credentials: {
jwtSecret: {type:'text'},
},
settings: {
uibuilderNodeEnv: { value: process.env.NODE_ENV, exportable: true },
},
})
Example use in your html file:
if (window.location.protocol !== 'https') {
if (RED.settings.uibuilderNodeEnv !== 'development') {
console.error('HTTPS NOT IN USE BUT SECURITY REQUESTED AND Node environment is NOT "development"')
} else {
console.warn('HTTPS NOT IN USE BUT SECURITY REQUESTED - Node environment is "development" so this is allowed but not recommended')
}
// TODO: Add user warnings
}
Not sure if standard Node-RED settings are accessible directly via RED.settings in the html file.
1 Like
Yes, searching on "buffer" and "queue" produces a few candidates. As @dceejay and @TotallyInformation point out, charting nodes are (or should be) designed with this functionality in mind. More general kinds of message replay are possible with the contributed nodes, and (self-promotion) the q-gate node is easily configured to retain an arbitrary number of the most recent messages, replay them singly or all at once in the order received, and use persistent context storage to restore its state after a shutdown or restart (/self-promotion). As you said, using this kind of replay requires great care. Designing systems to fail safe is an art practiced by too few and mastered by hardly any.
So we are back at the main question. Is the state recovery safe enough to provide from nodes or should it left for users to figure out how to manage if needed.
Currently the custom node can not do it reliably. This can be changed. But I can't see any reason to do it if there isn't agreement about the way it goes.
Can we conclude some midpoint agreement on main topic? Then the solution proposals may be coming in well more targeted.
Indeed, which is why I was pushing to get to a msg.timestamp field added to every msg as it was generated... but it never materialized. Seems that node-red is designed primarily for real-time message passing, and any temporal attributes need to be handled within custom flow logic, not the nodes themselves...
That said, I just checked the ui_chart.js code from my old install, and it does indeed check each incoming msg for a timestamp property, which it uses if found but defaults to now()
if (node.chartType === "line") {
var time;
if (msg.timestamp !== undefined) { time = new Date(msg.timestamp).getTime(); }
else { time = new Date().getTime(); }
Granted, that was probably a local customization I made to verify that it could work, before suggesting it to Dave.
Yes we do seem to be drifting off-topic
IMO there are probably 3 options (in descending order of preference);
- Create a mechanism within the code to read what storage options are available, and let the user choose from a populated dropdown. I realize that you have been doing a lot of work on this @hotNipi (thank you), but to achieve this maybe we need better specific support from other developers, especially the Leaders.
- Follow @drmibell example, and save the context to the default storage, with a brief explanation in the readme.
- In respect of
node-red-contrib-state-trailyou could always add an array of the data as a node output, so that context could be saved/restored in pretty much the same way as the chart node does.
Could we clarify the question we would like to answer? I think the original proposal was that we encourage developers of contributed nodes to add (when appropriate) a feature to restore the state of the node after re-deployment, restart, or reboot. The discussion has highlighted a possible technical limitation in managing multiple, non-default context stores. That is clear. The possibility that even attempting to do this could be "unsafe" or "unreliable" seems greatly exaggerated to me. Unless I'm mistaken, the simple-minded feature in the gate and q-gate nodes works exactly as advertised and should not cause problems for anyone who understands node context. Whether it is useful is up to the user.
1 Like
Use the RED.settings.context property that is already available in the editor:
This is how the TypedInputs already know about what stores are available. It doesn't not provide any information about the nature of the stores - just the list of store names and which one is considered the default store.
I would suggest the next step is to identify some specific nodes (more than the sample of one or two we have) that would benefit from this and to creating some examples that show that benefit and how it would actually work.
My concern is more on the unexplored edge cases this introduces. For example, if a user wants to explicitly reset the stored state of the node, how can they do that in a consistent way?
From that we can see if there needs to be a consistent way to present the option(s) in the UI - especially if it needs to provide a choice of context stores. We can then see if there is something more concrete the Editor could provide to help nodes be consistent.
4 Likes
Thank you Nick, I hope that helps @hotNipi
Could that be achieved by passing msg.reset into the node, which seems to be a common theme amongst other nodes.
I was thinking more of the scenario where they are developing the flow, make a change and want to clear any stored state so when the flow starts, it's starting with a clean slate. So more of an explicit developer-triggered action, rather than some piece of application logic.
How about: select the node, go to the context panel in the sidebar, select and refresh the node scope, and delete any or all of the state variables saved by the node that you want to reset.
May be there is chance to introduce another deploy option with cleanup procedure called if node has dedicated callback for that event?
node-red-contrib-ui-state-trail has now been updated to restore it's chart data after a full deploy, node-RED restart or system reboot. I've just been trying it and it works great, thanks @hotNipi.
The changes implemented have also moved development potential further forward, as the node now has access to the node-RED's setting.js file, and therefore it's contextStorage settings, so rules have be introduced to ensure that Data Storage is only implemented whenever a persistent data store is available.
Knowing which storage modules are available, opens up the future potential to maybe being able to select different storage option from within the node config (as @TotallyInformation commented above), and also ensure that orphaned data is cleaned up if a different storage option is selected.
Also as commented above, edge cases also need to be managed, but hopefully the clever guys here will eventually build this into a viable model for node data persistence.
2 Likes
This whole discussion reopened my thoughts on a caching node to complement uibuilder allowing people to avoid having to write their own function node.
This will now use Node-RED's context store features to store the cached entries. I've taken loads of ideas from this thread to start to flesh out the design so thanks a lot to everyone.
As usual, I somewhat sidetracked myself today by going down a related rabbit hole for creating better node documentation - settling on using VuePress as it integrates well with both a docs folder and GitHub pages.
If you want to follow the progress of the design, the docs for the node-red-contrib-infocache node are here.
2 Likes
