Problems understanding and implementing Persistent Storage

General
21

I have some data storing in global.json now (it's too big to completely see all of its contents so I don't know if it has everything). And that something recreates globals but not as expected.

There are 2 main issues.

  1. If I run the refresh globals flow and look at the globals, they are full of the data they should have. Then within 10-15 seconds all of those globals get overwritten. It seems that the global.json on disk is overwriting the newly created globals from the flow, instead of the other way around.

  2. If I delete one of the globals using a change node, I see it disappear (after a refresh) in the ContextBrowser's listing of globals (after a refresh) as expected. I see after 2 minutes the size of global.json has decreased somewhat which also is expected. Now I run the full global refresh flow and the situation in point #1 occurs. The global I deleted is missing in the ContextBrowser, though I could see data there 20 seconds previously.

I feel I am close here, just not sure what to change to get the behavior I am expecting: that the global.json file should update to the full set of globals on a full data refresh.

22

Have you added a new object using the technique described by @Steve-Mcl? That will prove whether it is a fundamental problem, or something to do with your flows.

23

Yes - forgot to mention that @Steve-Mcl and @Colin. I wrote the current date/time to a global variable, then checked it was there after a node-red restart. It's hard to say, but the value was not there when I checked. But given the replacement of globals with the current globals.json I mentioned above, it might be that I didn't check quickly enough to catch it before the overwrite.

I'm thinking that I do need to write all of the globals (via global.set) with their current values in the globals creation flow at the very end of their augmentation with extra properties (keys). Just so as to have the global.set force the cache to flush to disk.

24

Was that a new, completely independent global set (not part of another structure for example), so something simple like
global.set("testVar", new Date())
Did that var suffer from symptom 1 that you posted earlier - that is that it appears in the globals, but then disappears after a short while?
Assuming it does not disappear, if you stop node red, then before restarting it look in the globals file is it there?

25

Yes, @Colin that was a brand new key "DateRefreshed" and it is not used anywhere else.

DateRefreshed did make it into a global properly

const _time = new Date()
const Now = `${_time.toDateString()} ${_time.toTimeString()}`
node.warn({Now})
global.set("DataRefreshed", Now)
node.warn(global.get("DateRefreshed"))

results in
{"Now":"Sun Aug 13 2023 21:04:02 GMT+0000 (Coordinated Universal Time)"}
{ DateRefreshed: "Sun Aug 13 2023 21:04:02 GMT+0000 (Coordinated Universal Time)" }

now to see if it survives a node=red restart.

26

I don't understand what you mean by that statement. Where are you seeing those values.

Did both warn statements in the function show correctly?
It would have been much easier to use
const Now = new Date().toISOString()

Can you post the node red startup log, the one that starts Welcome to node red. Copy/paste please, not screenshot.

27

I edited my answer significantly just before yours as I noticed a typo DataRefreshed instead of DateRefreshed. The global did save properly . Stand by for whether DateRefreshed survives a restart.

(and also, I don't know where to find the node-red log)

28

One thing I can tell you is that the datetime stamp on global.json and its size do not change throughout the globals creation / global.set flow. Even after waiting the current 2 mins flushInterval in settings.js

29

No DateRefreshed did not survive the restart of Node-Red. That seems consistent as global.json does not change on disk. It seems that something I've done is preventing the memory versions of my globals from being flushed to disk

30

You need to find that. I don't know how to do that on Digital Ocean.

What versions of node-red and nodejs are you running?

31

node.js (v14.18.2)
node-red (v2.1.4)

Still looking into where the log is kept on D.O.

32

That is quite old, I don't know whether there have been context fixes since then. I suggest upgrading to the latest, and also upgrade nodejs to 18.

How do you stop and start node-red?

33

Those upgrades are in the works

I use sudo systemctl stop/start/status/restart node-red

34

Does this show the log?
journalctl -f -u node-red

If it does then leave that terminal open and in another window restart node-red. In the first window you should see it restarting node-red. Post the startup here, from the Welcome to Node-RED message.

35

Yes, I get output from the journalctl -f -u node-red

I am working out how to capture the output as you described (I'm not a linux user) and paste it here

36

To copy from a command window you should be able to use CTRL+SHIFT+C after marking with the mouse.

You are now.

2 Likes
37

Unfortunately, the client is at a conference demoing the software, so I cannot make changes until after hours. I will try tonight

38

You must always check the log for errors. For example, is the context folder and its files owned correctly? If not, that could make the writes fail. That should be in the log. If not, that's probably a bug but the write failure should probably in the syslog as well.

1 Like
39

I need to hold off on any changes whatsoever on the droplet until the weekend. So, I will report back then

40

Here's my update: SUCCESS

After a long hiatus where I wasn't able to make any changes, I have now upgraded Node.js to v20.5.1 and upgraded node-RED to v3.0.2. I also created another droplet on Digital Ocean and reset the node-red service on both to take ~1.5 GB of memory to use. Not sure how much of that it can access, but it represents 75% of a 2GB Memory allotment.

I then set up contextStorage in the settings.js as follows:

contextStorage: {
    default: {
        module: "localfilesystem",
            config: {
                cache: true,
                flushInterval: 30,
            },
      },
},

I then restarted the node-red service, and then ran the script that populates all the globals and does a global.set() for each one in order to start the save-to-disk process.

Next, I restarted the service and checked if the globals were restored to memory. They were!

Thanks to everyone for your suggestions! You got me to where I wanted to be.