Problems understanding and implementing Persistent Storage

General
1

I'm having trouble understanding how to achieve the following:
I have a set of 15 global variables that I'd like to have restored after a system restart. I have the following section added to the settings,js file:

contextStorage: {	
   default: {	
      module: "localfilesystem",	
      config: {	
         cache: true,	
         flushInterval: 1800,	
      },	
   },	
},

I watched the file system and I've seen the global.json file updating its timestamp every 30 minutes, so I assume the persistent store is being formed and updated.

When I have the digital Ocean droplet restart, and Node-Red is back up, none of the globals are available to a var X = global.get("XYZ")

It seems I am missing/misunderstanding something crucial here.
Do I have these settings correct? Do I have to add a flow that runs on restart to somehow restore the globals?

2

See if this helps you. It's something I wrote awhile ago but still hold up.

and the here is a link to the documentation:
https://nodered.org/docs/api/context/

and also read this:
https://nodered.org/docs/api/context/store/localfilesystem
and pay attention to the options!!! (hint hint)

3

Does the droplet have persistent file storage? Do you lose flows upon restart too?

Check your settings.js file - are your settings still present?

4

One other thing, flushInterval is in seconds and you have:

1800 / 60 = 30 minutes.

5

Yes, the droplet has persistent storage. On a start, the flows are all restored and I see the global.json file in the ...\.node-red\context\global folder being updated for time every 30 minutes. So the saving is happening. What isn't happening is the restoring of the globals to memory

6

Yes, the 30 minutes is a compromise, and perhaps a result of a lack of understanding. In my case, the globals get set once and then don't change unless a certain flow is manually started which updates all the globals. I'm using NR as a cache of sorts and a REST API for a NoCode front end.

7

I have read (but perhaps not completely understood) all of your references, and they were what I used to come up with my solution. But it seems I am missing something still.

I saw this in your one post:

SUGGESTION: If you want have all your context data be persistant, setup your settings.js file with the following:

    contextStorage: {
    	default    : { module: "localfilesystem"}
    },

Then I added the config options to allow for the cache.

Is there something amiss here? Sorry for being a bit unsure here.
Or do I need to do a function that runs on startup to restore my globals?

8

Hmmm, you may have found a bug. I was testing with flow context and it works fine, but when I switched to global context I've been able to reproduce your error.

Let me play a bit more and see what I can find.

1 Like
9

Of course now I can't recreate it. I may have deleted something between stoping and starting NR.

After stopping NR, can you open the global.json file in a editor and see what it contains.

10

Interesting!

global.json contains only {}

11

Check the time stamp of the global.flow and then start NR again and do something to create a global variable, In the NR sidebar, select the contact data tab and click the refresh button so you can see the global data.

Then check the global file again. If the data is there, stop NR and check it again.

I'm wondering it stopping the droplet is causing the global info to be destroyed.

12

Was that after stopping node red, but before restarting it?

13

It was after I stopped node-red with sudo systemctl stop node-red and confirmed it was stopped with sudo systemctl status node-red

14

What does it contain before you stop node-red (but after long enough has elapsed to make sure it has been written to file).
An unrelated point about the flush interval. It will only save it to file if the values have changed, so if they do not change often there is no harm in having a faster update rate.

1 Like
15

@zenofmud and @Colin

I have done a global.set("Persons", global.get("Persons")) at the end of global "creation" process and I am now waiting for the 30 mins to elapse to see if global.json was updated

16

Is that the exact code you use? Are you doing that in a function node? If so I'm not sure if that is going to work and if it does, it is just going to set 'Persons' with the value that is already in it.

Just what are you trying to accomplish?

17

I think it should also trigger the code that says something has (potentially) changed and so cause a context flush at the next timeout. I doubt if that logic checks whether anything has actually changed, though I have not checked the code.

1 Like
18

Also, since JavaScript passes objects by reference, if something "clears" that object out before the write occurs, it will write the changed value - depending on the life cycle of the variable, that may well be an empty object.

The test to make is - write something you know can never be externally affected e.g...

Function 1

const testObject= {
   prop1: 123
   prop2: "check me"
}
global.set('test_value', testObject)

then reboot and test it...

Function 2

const testObject = global.get('test_value') 
node.warn([ 'What do you see', testObject)

Additionally, as said earlier, you should set the flush time to something sensible (e.g. 30 secs). Flush only occurs when there is something to write. I.e., never change a value, never write to global.

1 Like
19

Yes, this is a function node. In this case, I was trying to trigger the write-to-file function of node-red according to my incomplete knowledge of persistent storage, but not change anything. I've seen others use this, and not sure if it is a good practice, but I wanted to show you what I did

Exactly. I only need this write operation to happen once and on demand. Thereafter, I will be reading only from the persistent storage, with memory being the preferred routine use, but from disk on restart

This is good information as well, and I will try this too.

Thanks to all three of you!

20

And thanks @Colin, @Steve-Mcl for the workings of the update process. I have returned the flushInterval to 30 seconds

1 Like