# Problems understanding and implementing Persistent Storage

**URL:** <https://discourse.nodered.org/t/problems-understanding-and-implementing-persistent-storage/80527>\
**Category:** General\
**Created:** [13 August 2023 01:09 UTC](https://discourse.nodered.org/t/problems-understanding-and-implementing-persistent-storage/80527 "2023-08-13T01:09:07Z")\
**Posts on this page:** 20\
**Page:** 2

<div class="post-metadata">

**Author:** ![UmbraLimi](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/umbralimi/32/64111_2.png) [@UmbraLimi](https://discourse.nodered.org/u/UmbraLimi)\
**Post date:** [13 August 2023 17:13 UTC](https://discourse.nodered.org/t/problems-understanding-and-implementing-persistent-storage/80527/21 "2023-08-13T17:13:59Z")

</div>

I have some data storing in global.json now (it's too big to completely see all of its contents so I don't know if it has everything). And that something recreates globals but not as expected.

There are 2 main issues.

1. If I run the refresh globals flow and look at the globals, they are full of the data they should have. Then within 10-15 seconds all of those globals get overwritten. It seems that the global.json on disk is overwriting the newly created globals from the flow, instead of the other way around.

2. If I delete one of the globals using a change node, I see it disappear (after a refresh) in the ContextBrowser's listing of globals (after a refresh) as expected. I see after 2 minutes the size of global.json has decreased somewhat which also is expected. Now I run the full global refresh flow and the situation in point #1 occurs. The global I deleted is missing in the ContextBrowser, though I could see data there 20 seconds previously.

I feel I am close here, just not sure what to change to get the behavior I am expecting: that the global.json file should update to the full set of globals on a full data refresh.

---

<div class="post-metadata">

**Author:** ![Colin](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/colin/32/17040_2.png) [@Colin](https://discourse.nodered.org/u/Colin)\
**Post date:** [13 August 2023 18:05 UTC](https://discourse.nodered.org/t/problems-understanding-and-implementing-persistent-storage/80527/22 "2023-08-13T18:05:48Z")

</div>

Have you added a new object using the technique described by @Steve-Mcl? That will prove whether it is a fundamental problem, or something to do with your flows.

---

<div class="post-metadata">

**Author:** ![UmbraLimi](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/umbralimi/32/64111_2.png) [@UmbraLimi](https://discourse.nodered.org/u/UmbraLimi)\
**Post date:** [13 August 2023 20:20 UTC](https://discourse.nodered.org/t/problems-understanding-and-implementing-persistent-storage/80527/23 "2023-08-13T20:20:51Z")

</div>

Yes - forgot to mention that @Steve-Mcl and @Colin. I wrote the current date/time to a global variable, then checked it was there after a node-red restart. It's hard to say, but the value was not there when I checked. But given the replacement of globals with the current globals.json I mentioned above, it might be that I didn't check quickly enough to catch it before the overwrite.

I'm thinking that I do need to write all of the globals (via global.set) with their current values in the globals creation flow at the very end of their augmentation with extra properties (keys). Just so as to have the global.set force the cache to flush to disk.

---

<div class="post-metadata">

**Author:** ![Colin](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/colin/32/17040_2.png) [@Colin](https://discourse.nodered.org/u/Colin)\
**Post date:** [13 August 2023 20:57 UTC](https://discourse.nodered.org/t/problems-understanding-and-implementing-persistent-storage/80527/24 "2023-08-13T20:57:25Z")

</div>

> [@UmbraLimi](#):
>
> I wrote the current date/time to a global variable

Was that a new, completely independent global set (not part of another structure for example), so something simple like  
`global.set("testVar", new Date())`  
Did that var suffer from symptom 1 that you posted earlier - that is that it appears in the globals, but then disappears after a short while?  
Assuming it does not disappear, if you stop node red, then before restarting it look in the globals file is it there?

---

<div class="post-metadata">

**Author:** ![UmbraLimi](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/umbralimi/32/64111_2.png) [@UmbraLimi](https://discourse.nodered.org/u/UmbraLimi)\
**Post date:** [13 August 2023 21:06 UTC](https://discourse.nodered.org/t/problems-understanding-and-implementing-persistent-storage/80527/25 "2023-08-13T21:06:47Z")

</div>

Yes, @Colin that was a brand new key "DateRefreshed" and it is not used anywhere else.

DateRefreshed did make it into a global properly

```auto
const _time = new Date()
const Now = `${_time.toDateString()} ${_time.toTimeString()}`
node.warn({Now})
global.set("DataRefreshed", Now)
node.warn(global.get("DateRefreshed"))

```

results in  
`{"Now":"Sun Aug 13 2023 21:04:02 GMT+0000 (Coordinated Universal Time)"}`  
`{ DateRefreshed: "Sun Aug 13 2023 21:04:02 GMT+0000 (Coordinated Universal Time)" }`

now to see if it survives a node=red restart.

---

<div class="post-metadata">

**Author:** ![Colin](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/colin/32/17040_2.png) [@Colin](https://discourse.nodered.org/u/Colin)\
**Post date:** [13 August 2023 21:14 UTC](https://discourse.nodered.org/t/problems-understanding-and-implementing-persistent-storage/80527/26 "2023-08-13T21:14:22Z")

</div>

> [@UmbraLimi](#):
>
> results in  
> `{"Now":"Sun Aug 13 2023 21:04:02 GMT+0000 (Coordinated Universal Time)"}`  
> `{ DateRefreshed: undefined }`

I don't understand what you mean by that statement. Where are you seeing those values.

Did both warn statements in the function show correctly?  
It would have been much easier to use  
`const Now = new Date().toISOString()`

Can you post the node red startup log, the one that starts Welcome to node red. Copy/paste please, not screenshot.

---

<div class="post-metadata">

**Author:** ![UmbraLimi](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/umbralimi/32/64111_2.png) [@UmbraLimi](https://discourse.nodered.org/u/UmbraLimi)\
**Post date:** [13 August 2023 21:18 UTC](https://discourse.nodered.org/t/problems-understanding-and-implementing-persistent-storage/80527/27 "2023-08-13T21:18:54Z")

</div>

I edited my answer significantly just before yours as I noticed a typo `DataRefreshed` instead of `DateRefreshed`. The global did save properly . Stand by for whether DateRefreshed survives a restart.

(and also, I don't know where to find the node-red log)

---

<div class="post-metadata">

**Author:** ![UmbraLimi](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/umbralimi/32/64111_2.png) [@UmbraLimi](https://discourse.nodered.org/u/UmbraLimi)\
**Post date:** [13 August 2023 21:22 UTC](https://discourse.nodered.org/t/problems-understanding-and-implementing-persistent-storage/80527/28 "2023-08-13T21:22:08Z")

</div>

One thing I can tell you is that the datetime stamp on `global.json` and its size do not change throughout the globals creation / global.set flow. Even after waiting the current 2 mins `flushInterval` in `settings.js`

---

<div class="post-metadata">

**Author:** ![UmbraLimi](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/umbralimi/32/64111_2.png) [@UmbraLimi](https://discourse.nodered.org/u/UmbraLimi)\
**Post date:** [13 August 2023 21:28 UTC](https://discourse.nodered.org/t/problems-understanding-and-implementing-persistent-storage/80527/29 "2023-08-13T21:28:31Z")

</div>

No `DateRefreshed` did not survive the restart of Node-Red. That seems consistent as `global.json` does not change on disk. It seems that something I've done is preventing the memory versions of my globals from being flushed to disk

---

<div class="post-metadata">

**Author:** ![Colin](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/colin/32/17040_2.png) [@Colin](https://discourse.nodered.org/u/Colin)\
**Post date:** [14 August 2023 07:55 UTC](https://discourse.nodered.org/t/problems-understanding-and-implementing-persistent-storage/80527/30 "2023-08-14T07:55:23Z")

</div>

> [@UmbraLimi](#):
>
> I don't know where to find the node-red log)

You need to find that. I don't know how to do that on Digital Ocean.

What versions of node-red and nodejs are you running?

---

<div class="post-metadata">

**Author:** ![UmbraLimi](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/umbralimi/32/64111_2.png) [@UmbraLimi](https://discourse.nodered.org/u/UmbraLimi)\
**Post date:** [14 August 2023 14:51 UTC](https://discourse.nodered.org/t/problems-understanding-and-implementing-persistent-storage/80527/31 "2023-08-14T14:51:59Z")

</div>

node.js (v14.18.2)  
node-red (v2.1.4)

Still looking into where the log is kept on D.O.

---

<div class="post-metadata">

**Author:** ![Colin](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/colin/32/17040_2.png) [@Colin](https://discourse.nodered.org/u/Colin)\
**Post date:** [14 August 2023 15:02 UTC](https://discourse.nodered.org/t/problems-understanding-and-implementing-persistent-storage/80527/32 "2023-08-14T15:02:47Z")

</div>

> [@UmbraLimi](#):
>
> node-red (v2.1.4)

That is quite old, I don't know whether there have been context fixes since then. I suggest upgrading to the latest, and also upgrade nodejs to 18.

How do you stop and start node-red?

---

<div class="post-metadata">

**Author:** ![UmbraLimi](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/umbralimi/32/64111_2.png) [@UmbraLimi](https://discourse.nodered.org/u/UmbraLimi)\
**Post date:** [14 August 2023 18:02 UTC](https://discourse.nodered.org/t/problems-understanding-and-implementing-persistent-storage/80527/33 "2023-08-14T18:02:11Z")

</div>

Those upgrades are in the works

I use `sudo systemctl stop/start/status/restart node-red`

---

<div class="post-metadata">

**Author:** ![Colin](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/colin/32/17040_2.png) [@Colin](https://discourse.nodered.org/u/Colin)\
**Post date:** [14 August 2023 19:44 UTC](https://discourse.nodered.org/t/problems-understanding-and-implementing-persistent-storage/80527/34 "2023-08-14T19:44:19Z")

</div>

Does this show the log?  
`journalctl -f -u node-red`

If it does then leave that terminal open and in another window restart node-red. In the first window you should see it restarting node-red. Post the startup here, from the Welcome to Node-RED message.

---

<div class="post-metadata">

**Author:** ![UmbraLimi](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/umbralimi/32/64111_2.png) [@UmbraLimi](https://discourse.nodered.org/u/UmbraLimi)\
**Post date:** [15 August 2023 00:59 UTC](https://discourse.nodered.org/t/problems-understanding-and-implementing-persistent-storage/80527/35 "2023-08-15T00:59:44Z")

</div>

Yes, I get output from the `journalctl -f -u node-red`

I am working out how to capture the output as you described (I'm not a linux user) and paste it here

---

<div class="post-metadata">

**Author:** ![Colin](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/colin/32/17040_2.png) [@Colin](https://discourse.nodered.org/u/Colin)\
**Post date:** [15 August 2023 06:39 UTC](https://discourse.nodered.org/t/problems-understanding-and-implementing-persistent-storage/80527/36 "2023-08-15T06:39:10Z")

</div>

To copy from a command window you should be able to use CTRL+SHIFT+C after marking with the mouse.

> [@UmbraLimi](#):
>
> I'm not a linux user

You are now.

---

<div class="post-metadata">

**Author:** ![UmbraLimi](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/umbralimi/32/64111_2.png) [@UmbraLimi](https://discourse.nodered.org/u/UmbraLimi)\
**Post date:** [15 August 2023 15:13 UTC](https://discourse.nodered.org/t/problems-understanding-and-implementing-persistent-storage/80527/37 "2023-08-15T15:13:10Z")

</div>

Unfortunately, the client is at a conference demoing the software, so I cannot make changes until after hours. I will try tonight

---

<div class="post-metadata">

**Author:** ![TotallyInformation](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/totallyinformation/32/31_2.png) [@TotallyInformation](https://discourse.nodered.org/u/TotallyInformation)\
**Post date:** [15 August 2023 15:53 UTC](https://discourse.nodered.org/t/problems-understanding-and-implementing-persistent-storage/80527/38 "2023-08-15T15:53:44Z")

</div>

You must always check the log for errors. For example, is the context folder and its files owned correctly? If not, that could make the writes fail. That should be in the log. If not, that's probably a bug but the write failure should probably in the syslog as well.

---

<div class="post-metadata">

**Author:** ![UmbraLimi](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/umbralimi/32/64111_2.png) [@UmbraLimi](https://discourse.nodered.org/u/UmbraLimi)\
**Post date:** [16 August 2023 13:49 UTC](https://discourse.nodered.org/t/problems-understanding-and-implementing-persistent-storage/80527/39 "2023-08-16T13:49:59Z")

</div>

I need to hold off on any changes whatsoever on the droplet until the weekend. So, I will report back then

---

<div class="post-metadata">

**Author:** ![UmbraLimi](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/umbralimi/32/64111_2.png) [@UmbraLimi](https://discourse.nodered.org/u/UmbraLimi)\
**Post date:** [30 August 2023 17:01 UTC](https://discourse.nodered.org/t/problems-understanding-and-implementing-persistent-storage/80527/40 "2023-08-30T17:01:29Z")

</div>

Here's my update: SUCCESS

After a long hiatus where I wasn't able to make any changes, I have now upgraded Node.js to v20.5.1 and upgraded node-RED to v3.0.2. I also created another droplet on Digital Ocean and reset the node-red service on both to take ~1.5 GB of memory to use. Not sure how much of that it can access, but it represents 75% of a 2GB Memory allotment.

I then set up `contextStorage` in the `settings.js` as follows:

```auto
contextStorage: {
    default: {
        module: "localfilesystem",
            config: {
                cache: true,
                flushInterval: 30,
            },
      },
},

```

I then restarted the node-red service, and then ran the script that populates all the globals and does a `global.set()` for each one in order to start the save-to-disk process.

Next, I restarted the service and checked if the globals were restored to memory. They were!

Thanks to everyone for your suggestions! You got me to where I wanted to be.

[Previous page](https://discourse.nodered.org/t/problems-understanding-and-implementing-persistent-storage/80527.md?page=1)

[Next page](https://discourse.nodered.org/t/problems-understanding-and-implementing-persistent-storage/80527.md?page=3)