A guide to understanding 'Persistent Context'

#1

This is something I wrote up after playing with the context options. I've filed a PR with some of this added to the Node-RED documentation but thought I'd release it here also. I hope people find it useful.
Paul


A guide to understanding 'Persistent Context'

Prior to Node-RED v0.19.0, data could be stored in context as a global, flow or node. This was stored in memory and would be reset with a restart of Node-RED. As of v.0.19.0 you can now store node, flow and global context data in memory OR in a file which will persist over restarts. (the data will be stored in the userDir, which is normally $HOME/.node-red, in a folder 'context' with subfolders for each flow or node and one folder for all globals.)

In order to store the data, you must make a change to settings.js - without the change, context stores will always be in memory and will not be persistent.

The contextStorage property in settings.js is used to configure how context data is stored:

    contextStorage: {
    	storeName    : { module: "storeModule" }
    },

storeName: The storeName used in get/sets

storeModule: Node-RED provides two built-in store modules: memory and localfilesystem. It is also possible to create custom store plugins. You must specify localfilesystem (or your own custom module) to make the data persistent. For details on creating custom modules, see the api pages.

If you only have one option in contextStorage, it will always be used. If you have two options and one has the storeName default (order doesn't matter) it will be used if the get/set storeName option is not used. If you try to get or set using a location storeName that does not exist, it will use the default and you will see a one time warning in the log.

Example: say these are your entries in setting.js:

    contextStorage: {
		storeInFile: { module: "localfilesystem"},
    	default    : { module: "memory" }
    },

And you use the following set's (this applies to any node that can access context directly like the change, inject, or change nodes):

flow.set("count", 123);               // stored in memory, count = 123
flow.set("count", 234, "default");    // stored in memory, count now - 234
flow.set("partid", "b301", "storeInFile"); // stored in file, partid = "b301"
flow.set("color", "red", "InFile");   // invalid storeName 
                                      // - stored based on 'default' rules
                                      // - a 'storeName' of `default` exists and is used
                                      // - stored in memory, color = 'red'

Note: Having multiple entries in settings.js can lead to confusion. If you have:

    contextStorage: {
    	default    : { module: "memory" },
		storeInFile: { module: "localfilesystem"},
		memoryOnly : { module: "memory" }
    },

and run the following code:

	flow.set("count", 123);               // the value is stored in memory
	flow.set("count", 234, "default");    // the value is stored in memory
	flow.set("count", 345, "memoryOnly"); // the value is stored in memory

the first line stores '123' in default:count.
the second line replaces '123' with '234' in default:count
the third line stores '345' in memoryOnly:count

If you forget to specify the location in a get or set, you might end up with the wrong value.

SUGGESTION: If you want have all your context data be persistant, setup your settings.js file with the following:

    contextStorage: {
    	default    : { module: "localfilesystem"}
    },

Here are some more examples

Example 1: default to memory, require a name for storing in a file

    contextStorage: {
    	default    : { module: "memory" },
		storeInFile: { module: "localfilesystem"}
    },
	flow.set("count", 123);              // the value is stored in memory
	flow.set("count", 234, "default");   // the value is stored in memory
	flow.set("ID", 345, "storeInFile");  // the value is stored in a file
	flow.set("date", 345, "somewhere");  // since there is no storeName "somewhere", 
	                                     // and there is a storeName "default",
	                                     // the value is stored in memory

Example 2: default to persistant require a name for storing in memory

    contextStorage: {
		storeInFile: { module: "localfilesystem"}
    	memoryOnly : { module: "memory" },
    },
	flow.set("total", 123);  // since no option is provided and there is no "default"
							 // option, the first option "storeInFile" is used so
							 // the value is persistant and stored in a file
	
	flow.set("count", 234, "memoryOnly"); // the value is stored in memory
	flow.set("ID", 345, "storeInFile");   // the value is stored in a file
	flow.set("date", 345, "somewhere");   // since there is no storeName "somewhere" and 
	                                      // there no storeName "default" the first 
	                                      // storeName is used ('storeInFile') so the 
	                                      // value is persistant and stored in a file
13 Likes

Best way to collect, store and retrieve a small (10-25) set of sensor values
How to get global variable properties
User Defined Evironmental Variables
Storing an index by a counter
Automatic fish feeder
Building an API interface for airconditioning automation
#2

Nice work ! but is below correct ? - or is the comment wrong ? or...

0 Likes

#3

Oops :scream: the comment was wrong (corrected) - this is why everyone needs an editor :grin:

0 Likes

#4

Just one little thought about persistent context & writing...to sd cards

Does this new elegant feature increase the wear on the sd card? I can see that files are being updated rather frequently. We learned we should write as few times as possible, turned off disk write access as much as possible. I'm currently using MQTT with retain flag saving all statuses in an object to be able to recover at init. A second thought, most likely MQTT writes to disk anyway when you use the retain flag

What is your opinion on this? Maybe OT but still related...

0 Likes

#5

@krambriw any sort of persistent storage needs to write to the SD card, unless you use some external storage mechanism (USB hard drive, cloud storage api...). It's hard to get around that.

To help minimise the wear, the default mode for the file persistence is to batch up writes to the card every 30 seconds. This does mean if Node-RED crashes unexpectedly, you may not have up to the last 30 seconds worth of data stored. That's the trade-off between writing every update to disk as it happens and trying to minimise wear.

0 Likes

#6

I would not worry too much about writing small amounts of data to the card. Provided you use good quality cards it should not be an issue unless you are repeatedly writing megabytes of data.
Even the lowest spec cards quote thousands of write cycles, which means writing the whole card several thousand times. If you want a life of, say, 10 years that is 3650 days (give or take a couple of leap days) so you could rewrite the whole card (maybe 8GB) every day and still expect it survive many years. In practice it is not that simple but it gives a feel for the figures.

0 Likes

#7

Also, it's certainly possible to write persistent context to a non-wearable media like a network hard drive or something ?

0 Likes

#8

Great write up. I have one question however. When I wish to check on the stored value (file store) the directories under 'context' use obscure node numbering. Would it be possible to use the tab name, (or node name) which is what one might expect to look under? Just a thought. I only care when I'm coding because that's when I'm checking to see things are doing what I expected. But I have to say, I do love the persistent context. Have done a lot of rewriting to incorporate it but the previous solutions I created were not 1,000% reliable and certainly not as elegant as this.
ST

0 Likes

#9

The file store isn't really intended for the user to access directly - its an internal implementation detail. The naming is based on the id of the nodes and flows as they are guaranteed to be unique and don't change. Node names can be changed by the user and you can have many nodes named the same thing.

0 Likes

#10

yes, that's what I meant in my reply where I said:

use some external storage mechanism (USB hard drive, cloud storage api...).

1 Like

#11

OK, I understand. Thank you.

BTW, I think I read it's not on the list for development however, just for some feedback, I would vote in support of being able to create a utility function somewhere that I can globally call upon in any flow / node rather than repeat code in many function nodes. (NOTE: creating a custom / private NR node for this would be overkill and I think less elegant). I have succeeded in achieving this by assigning a function to a context.global variable (NR complains when it starts) but I read this is not recommended and that it may be policed one day. I took it out when I read that because I didn't want to have to go back deep into all my code at some future date when I forgot all the tiny little details should the ability to do this change. Just so you know .... some input from the cheap seats.

0 Likes

#13

Cool!

But were do i place this?

contextStorage: 
	storeName    : { module: "storeModule" }
}

When i put it in the end before the last } i get an error on starting node-red.

Do i understand well that when i put this in settings.js that the global are persistent?

0 Likes

#14

It’s not that it will be policed. Most storage mechanisms only really support serialisable objects, and functions aren’t serialisable. So now we do have persistable context, depending on the storage mechanism, functions (and circular objects) won’t be stored.

0 Likes

#15

Solved! There was an { missing after contextStorage:

0 Likes

#16

'storeName' and 'storeModule' are the place holder names. Did you look at the examples?

0 Likes

#17

As @zenofmud mentions, he's raised a pull-request to get this integrated into our proper documentation - I'm keen we have good docs that can be linked to for these sorts of questions, rather than rely on linking to forum posts.

The pull-request is here: https://github.com/node-red/node-red.github.io/pull/86

Please take a look at the proposed changes and provide any feedback on the PR - and also marvel at how easy it is to contribute to the documentation... :wink:

2 Likes

#18

agree, but i really dont understand how to do that, but instead of doing nothing i post it here in the hope it helps some one...

It's all ready difficult enough for me to get it working :wink:
Because when i stop start NR now the global.get is still empty

0 Likes

#19

@RogierQ - your right, the opening '{' was missing - fixed now.

1 Like

#20

What is not clear to me now, is when it is saved

Doing some tests the results of saved global dont show up directly in the json file
Or is it saved when deploying?

@zenofmud thanks!

0 Likes

#21

It may help if you share exactly what you have put in your settings file and show us how you then set a value in context - just so we can check you're doing it right.

0 Likes