Even by having developed one node using context storage (strictly, not even optional) - I'm still on the side of NO. It should not be widely common. Too much unpredictability.
History is full of readmes that aren’t :). I’m not saying it’s not possible, just not trivial and not necessarily required universally and could confuse naive users. Keep it simple. If a particular node does require it it can be implemented.
So going back to yesterdays post - Save data so that it is not deleted how would you advise Rene to ensure that data is persisted between reboots, node-RED restarts, and full deploys in node-red-contrib-state-trail or are you advocating that users should expect no persistence, just a clean start, and all previous data is lost?
State trail has option to show the stored data (since yesterday). The way to store the data is user decision. My suggestion is to use some dedicated database.
That's exactly what I'm trying to achieve.
But just think from a new user's perspective, how do they create chart persistence? It's currently much more complicated than what is suggested here.
What about some middle ground? Perhaps the concept of context in NR could be made a little easier and more "automatic" than it is now?
Could everything contextual be wrapped up in its own node for example?
That sounds like a lot of work for users to do, and not everyone is confident about creating databases & associated queries.
But as I said above, it's your node, and your decision how it will work.
This might once have been true but there is a well recognised set of circumstances that allow the SD-card file systems to be rock solid. I've been running 2 pi's for well over 3 years now on the same cards with nothing done to reduce writes. I have DB's on the cards that are updating every few seconds as well as all of the logs.
A good brand card (I use Samsung EVO or EVO-Pro) that is oversized (I use 32GB) is all it takes.
If a user of Node-RED wishes to persist data, they can use the file-based context store as they are in control of setting it up (or at least their admin will be in an enterprise context). They can also do it manually by writing something to file.
If a custom node wants to offer persisted data then I think it should provide its own persistence because, as Dave say's, that author of the node will never know the configuration of the system.
That's what I did in uibuilder. Since uibuilder needs to persist changes (newly installed front-end libraries) across restarts and reboots, it has to save that data to a file and read it when NR starts, restarts and a few times in-between for various reasons.
However, I'd expect this to be an exception rather than a rule since Node-RED is primarily about message flow. As such, things like MQTT retained messages or DB queries would be expected to be the start of a flow that needs historic data and would include any required data. Retained context variables are a great addition because it adds another option - I use them for static reference data for example and it saves me creating and manipulating a SQLite DB or JSON file.
I've noticed that some 'contrib' nodes do not pass on the contents of the incoming msg, they build a new one. So If I added msg.foo and sent it thru the node, msg.foo would not exist in the out going message. Should this be considered a 'bug' in that node?
Yes absolutely.
1 Like
Not necessarily, depends on the node. Of course, the node should make it clear what it is doing. Personally, I generally try to pass through data as long as it makes sense. There is an option in uibuilder to do that for example and I always try to pass through the msg.topic at least.
Obviously some nodes are specifically designed to replace the msg.payload and that is one of the reasons why it is helpful to have a convention around .payload and .topic so you have 2 common properties that most nodes use.
Julian, I understand nodes can/will change msg.payload but I'm talking about things in the msg other than msg.payload. If I want to keep something that is in msg.payload, I might create msg.original_payload and copy msg.payload to it before entering another node. The output of that node should still contain msg.original_payload along with what ever it sticks in msg.payload.
@zenofmud @TotallyInformation let's not take this topic off topic. It really is simple - a node should not arbitrarily drop message properties it doesn't specifically use/set. There are very particular scenarios where that might be okay to do, but in the vast majority of cases, nodes should preserve message properties that pass through.
1 Like
Yes, got that. Sorry, I should have been clearer.
I still think "it depends" is the right answer overall but as Nick has indicated, generally you would expect a node to pass data though - unless it has a good reason not too. If it has such a reason, it should make it clear in the documentation.
We've seen examples for instance, where nodes haven't passed through the msg.topic which is really annoying. But passing of other properties is harder to spot and harder for node authors to remember.
In addition, you wouldn't necessarily expect a node to pass all data through to a 2nd/3rd/etc output.
So I don't believe there is a one-size-fits-all answer to this. That is the beauty of Node-RED as well as its complexity. 
No, but to reiterate the message I want to get across, one-size-fits-99%.
2 Likes
I'm not feeling the love, so I guess I must be the 1% in this case. I'll make my last pitch and leave it at that.
I've read the comments above, but apart from dismissing the idea, nobody has come up with a good reason why not...
I thought node-RED was all about making programming easier for users (otherwise we would all be writing javascript!), and yet this suggestion;
- is backwards compatible, users could choose to use it or not
- users could use their own solution instead whether it be a database, MQTT etc, whilst I suspect the majority would use file based persistence
- it reduces the number of nodes in a flow
- new users would find it much easier to use
- because the data would be saved in the node's context scope, there is less likelyhood of user problems caused by mispelling context keys, or using the same key for more than one node
- there is already a template (which has been tested for 12 months without issue)
1 Like
I've looked at the implementation and here is my "why not" (I think these have all been mentioned already):
-
The default context store is used, beginners in particular will find this very confusing since the retention they expect will not happen. They would need to know to go and manually change
settings.jswhich is not necessarily a beginner task. Even then, they would have to chose the file store as the default which many people (myself included), would not want to do.At the very least, the "standard" code to support this would have to provide an option for choosing a different store type which adds even further complexity.
-
It adds hidden complexity which is never going to be a good thing for the majority of nodes. In this case, as indicated above, it adds considerable hidden complexity.
Restoring persistence using a flow may look more messy but it is easier to follow the logic.
-
Most nodes will never need it.
Nothing stopping a custom node from using it though as in your case. I considered using it myself but came to the conclusion of the first bullet above. I originally used the settings.js file for a number of settings related to uibuilder but came to the conclusion that it wasn't a good fit for what I needed to achieve so moved them to their own, dedicated files in the uibRoot folder.
Sorry @Paul-Reed , my comments were nothing to do with your discussion on node context - it was on the side topic of whether a node should pass through all message properties or not.
Back on topic, I think there is a place for nodes to use node context to store state. That is one of the reasons we introduced context. But it does depend on the requirements of individual nodes - it makes more sense for some than others.
I heavily disagree that nodes should create their own schemes for persisting data. There are of course exceptions, but in general, using Context makes sense to me - it's what it was designed for. Otherwise you end up with every node doing its own thing which becomes a nightmare to manage. Plus it ignores the scenario where you are running in an environment where you haven't got a filesystem you can write to - this is why the main storage mechanism and the context store mechanism are pluggable. If nodes start making assumptions they can write to the filesystem, then they will no longer work in lots of scenarios we support.
In response to Julians' points:
That is true of Context as a whole concept. The challenge is getting the UI design right so users are not surprised.
But nor is building a flow that can fully persist all state and manage its restoration properly.
The alternative is you push the complexity into the open by forcing a user to design flows to handle it. That is absolutely not a beginner's task.
The whole ethos of node-red is the nodes hide the complexity of the task they are designed to achieve.
I agree. I don't see this being a case of opening the flood gates for every node to start storing state. But I suspect there will be some key nodes that, if they supported this in some form and the user experience was well designed and consistent, it would go a long way to helping the most common use cases.
But also that they do one thing and do it well.
We seem to be agreeing that state persistence is complex at least. I suspect that a longer discussion would show that we are simply seeing the issue from different perspectives and that, in reality, our views are relatively similar.
Certainly I agree with most of your points.
The point you make about the lack of a filing system is a very good one. Two of my custom nodes are dependent on having a filing system anyway so it is easy to forget this point.
I think a lot of your (good) reasons against concept are due to the persistent setting being in settings.js and not a simple GUI option in the main editor.
If which persistent option to use was a dropdown next to the tick box to use it then I think it becomes a good option
Of course there are many good options that we all want and priorities need to be made 
1 Like
