# A catch with persistent file stores

**URL:** <https://discourse.nodered.org/t/a-catch-with-persistent-file-stores/21062>\
**Category:** General\
**Created:** [30 January 2020 22:55 UTC](https://discourse.nodered.org/t/a-catch-with-persistent-file-stores/21062 "2020-01-30T22:55:47Z")\
**Posts on this page:** 20\
**Page:** 2

<div class="post-metadata">

**Author:** ![TotallyInformation](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/totallyinformation/32/31_2.png) [@TotallyInformation](https://discourse.nodered.org/u/TotallyInformation)\
**Post date:** [31 January 2020 18:13 UTC](https://discourse.nodered.org/t/a-catch-with-persistent-file-stores/21062/21 "2020-01-31T18:13:40Z")

</div>

Going to take me some time to work through the various options on the different platforms and check but I can't see that it would hurt to add the others. Just not clear to me that it actually does what I thought. ☹

---

<div class="post-metadata">

**Author:** ![dceejay](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/dceejay/32/38_2.png) [@dceejay](https://discourse.nodered.org/u/dceejay)\
**Post date:** [31 January 2020 18:19 UTC](https://discourse.nodered.org/t/a-catch-with-persistent-file-stores/21062/22 "2020-01-31T18:19:29Z")

</div>

Added a PR here [https://github.com/node-red/node-red/pull/2447](https://github.com/node-red/node-red/pull/2447)  
feel free to comment etc...  
(inc added SIGBREAK for ctrl-break on windows)

see here for node.js documentation - [https://nodejs.org/api/process.html#process\_signal\_events](https://nodejs.org/api/process.html#process_signal_events)

---

<div class="post-metadata">

**Author:** ![TotallyInformation](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/totallyinformation/32/31_2.png) [@TotallyInformation](https://discourse.nodered.org/u/TotallyInformation)\
**Post date:** [31 January 2020 18:58 UTC](https://discourse.nodered.org/t/a-catch-with-persistent-file-stores/21062/23 "2020-01-31T18:58:38Z")

</div>

Doh! Well, this explains why I can't process any of those signals while on Windows:

> Windows does not support sending signals

Drat, missed that.

---

<div class="post-metadata">

**Author:** ![dceejay](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/dceejay/32/38_2.png) [@dceejay](https://discourse.nodered.org/u/dceejay)\
**Post date:** [31 January 2020 19:19 UTC](https://discourse.nodered.org/t/a-catch-with-persistent-file-stores/21062/24 "2020-01-31T19:19:55Z")

</div>

I think you mis-read that - I read that as meaning that node.js on Windows can't produce those signals - not that it can't catch them... otherwise the SIGBREAK would not make sense... but then again it's windows so it may not make sense ;-).... (sorry - couldn't stop myself on a Friday)

---

<div class="post-metadata">

**Author:** ![TotallyInformation](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/totallyinformation/32/31_2.png) [@TotallyInformation](https://discourse.nodered.org/u/TotallyInformation)\
**Post date:** [31 January 2020 19:26 UTC](https://discourse.nodered.org/t/a-catch-with-persistent-file-stores/21062/25 "2020-01-31T19:26:45Z")

</div>

> [@dceejay](#):
>
> I think you mis-read that

No, I read it the same way as you. Windows is only partially POSIX compliant. Normally not an issue and it is just annoying here, not the end of the world. Just makes testing stuff based on the persistent file store harder - well slower. I'll just reduce the timeout and wear out my super-expensive SSD, good job I didn't pay for it!

I've learned a lot very quickly. I'll just add it to my prodigious memor .... ooh, squirrel!

---

<div class="post-metadata">

**Author:** ![dceejay](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/dceejay/32/38_2.png) [@dceejay](https://discourse.nodered.org/u/dceejay)\
**Post date:** [31 January 2020 19:35 UTC](https://discourse.nodered.org/t/a-catch-with-persistent-file-stores/21062/26 "2020-01-31T19:35:20Z")

</div>

Adding those handlers should fix it for you. (And nodemon) at least

---

<div class="post-metadata">

**Author:** ![TotallyInformation](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/totallyinformation/32/31_2.png) [@TotallyInformation](https://discourse.nodered.org/u/TotallyInformation)\
**Post date:** [31 January 2020 20:34 UTC](https://discourse.nodered.org/t/a-catch-with-persistent-file-stores/21062/27 "2020-01-31T20:34:55Z")

</div>

BTW, this is a very interesting thread about signal handling issues both in Node.js specifically and JavaScript in general. Of special note are the comments about the inability to handle promises.

> <https://github.com/nodejs/node/issues/29480>
>
> I am not sure whether this is a question, bug report, or feature request, but here we go.
> At the moment, it...

---

<div class="post-metadata">

**Author:** ![kuema](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/kuema/32/6542_2.png) [@kuema](https://discourse.nodered.org/u/kuema)\
**Post date:** [3 February 2020 12:44 UTC](https://discourse.nodered.org/t/a-catch-with-persistent-file-stores/21062/28 "2020-02-03T12:44:04Z")

</div>

Just for reference, the same or similar problem exists with PM2 on Windows. It just kills the processes.  
Funny thing: there is a documented way to perform a graceful shutdown on Windows by using `process.on('message')`, but that doesn't work either. 😐 Plus there are several outstanding issues for this problem.

- [https://pm2.keymetrics.io/docs/usage/signals-clean-restart/#windows-graceful-stop](https://pm2.keymetrics.io/docs/usage/signals-clean-restart/#windows-graceful-stop)
- [https://github.com/Unitech/pm2/issues/3254](https://github.com/Unitech/pm2/issues/3254)
- [https://github.com/Unitech/pm2/issues/3651](https://github.com/Unitech/pm2/issues/3651)

Note: I haven't checked since mid. 2019, but there have been some updates on this matter in October 2019.

---

<div class="post-metadata">

**Author:** ![dceejay](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/dceejay/32/38_2.png) [@dceejay](https://discourse.nodered.org/u/dceejay)\
**Post date:** [3 February 2020 12:48 UTC](https://discourse.nodered.org/t/a-catch-with-persistent-file-stores/21062/29 "2020-02-03T12:48:22Z")

</div>

which SIGNAL does pm2 use ?

---

<div class="post-metadata">

**Author:** ![kuema](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/kuema/32/6542_2.png) [@kuema](https://discourse.nodered.org/u/kuema)\
**Post date:** [3 February 2020 13:05 UTC](https://discourse.nodered.org/t/a-catch-with-persistent-file-stores/21062/30 "2020-02-03T13:05:41Z")

</div>

It's using the `SIGINT` signal. That one is already caught by the `red.js` and is working properly under **Linux**.

As for Windows, they are sending a process message which can be caught by `process.on('message')`. At least that is what their documentation says. It never worked for me. 😅

---

<div class="post-metadata">

**Author:** ![kuema](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/kuema/32/6542_2.png) [@kuema](https://discourse.nodered.org/u/kuema)\
**Post date:** [3 February 2020 13:32 UTC](https://discourse.nodered.org/t/a-catch-with-persistent-file-stores/21062/31 "2020-02-03T13:32:37Z")

</div>

I have just checked their change from October 2019 on my Windows system.

I can confirm that the documented workaround using process messages is working now.  
However, none of the signals are getting through. I listened to `SIGBREAK`, `SIGINT`, `SIGUSR2`, `SIGQUIT`, `SIGTERM`.

---

<div class="post-metadata">

**Author:** ![TotallyInformation](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/totallyinformation/32/31_2.png) [@TotallyInformation](https://discourse.nodered.org/u/TotallyInformation)\
**Post date:** [3 February 2020 20:07 UTC](https://discourse.nodered.org/t/a-catch-with-persistent-file-stores/21062/32 "2020-02-03T20:07:06Z")

</div>

I think there are 2 issues potentially here and they are both referred to in the earlier links.

Firstly, Windows is not fully POSIX compliant and doesn't in fact issue all of the signals.

Secondly, you cannot necessarily stack signal processing because the first function that is hit is the one that will exit and so other functions won't be triggered.

---

<div class="post-metadata">

**Author:** ![drmibell](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/drmibell/32/8424_2.png) [@drmibell](https://discourse.nodered.org/u/drmibell)\
**Post date:** [3 February 2020 23:15 UTC](https://discourse.nodered.org/t/a-catch-with-persistent-file-stores/21062/33 "2020-02-03T23:15:16Z")

</div>

> [@TotallyInformation](#):
>
> Node-RED, I think, should flush its cache when stopping but it doesn't.

If I could step back to the original issue, I have two thoughts.

First, as far as context is concerned, failing to flush during what is supposed to be an orderly shutdown is no worse (and possibly better) than the consequences of a system crash or power cut. The saved context will be out of date by at most 30 seconds (or the `flushInterval`), and the user will have to decide whether there is any point in trying to use it. Short of actually solving the problem, the question becomes whether the behavior is predictable.

Second, and more important, I think the issue can be finessed entirely (at a cost) by configuring an additional context store with `module: "localfilesystem"` and `config: {cache:false}`. If I understand correctly, this will force every change in the context variables using that store to be written to the file system immediately. Obviously, this makes the most sense if the file system sits on something more durable than an SD card, or if the size or frequency of updates is low.

---

<div class="post-metadata">

**Author:** ![kuema](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/kuema/32/6542_2.png) [@kuema](https://discourse.nodered.org/u/kuema)\
**Post date:** [4 February 2020 05:32 UTC](https://discourse.nodered.org/t/a-catch-with-persistent-file-stores/21062/34 "2020-02-04T05:32:50Z")

</div>

> [@TotallyInformation](#):
>
> Firstly, Windows is not fully POSIX compliant and doesn't in fact issue all of the signals.

Right, in fact the only reliable signal I found so far was `SIGINT` when pressing CTRL-C in a console.  
I use that one to stop my services that were setup with NSSM, which can simulate the CTRL-C.

> [@TotallyInformation](#):
>
> Secondly, you cannot necessarily stack signal processing because the first function that is hit is the one that will exit and so other functions won't be triggered.

That was just for testing to see which signals make it through to NodeJS. 😅 But I agree that these should be used with care in a production case.

---

<div class="post-metadata">

**Author:** ![kuema](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/kuema/32/6542_2.png) [@kuema](https://discourse.nodered.org/u/kuema)\
**Post date:** [4 February 2020 05:39 UTC](https://discourse.nodered.org/t/a-catch-with-persistent-file-stores/21062/35 "2020-02-04T05:39:47Z")

</div>

Valid points. I am facing a similar "dilemma" with a current project, keeping some state variables in the file store. Though in this special case it would even be an option to **delete** it completely and start with a clean slate. But to do that you would need to detect such an unclean shutdown condition in the first place.

---

<div class="post-metadata">

**Author:** ![TotallyInformation](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/totallyinformation/32/31_2.png) [@TotallyInformation](https://discourse.nodered.org/u/TotallyInformation)\
**Post date:** [4 February 2020 06:48 UTC](https://discourse.nodered.org/t/a-catch-with-persistent-file-stores/21062/36 "2020-02-04T06:48:27Z")

</div>

There is clearly no way to do this totally reliably across all platforms from within a Node.js service itself.

I've come to the conclusion that if you want more reliability from a persistent data store, you need to hand the data off to a dedicated managed data engine. Either a database engine or something like REDIS. Something that wraps writes into a transaction so that it guarantees validity.

Such an engine will have its own, well developed ways of dealing with crashes but won't be as susceptible to them anyway because it doesn't have to deal with unexpected errors and user generated code issues.

Just to be clear, this is **not** a Node-RED issue, it is a Node.js one.

For the most part, using the file store in Node-RED will be perfectly stable even on Windows in by far the majority of cases. I would never have even noticed except that I began rapid testing of some code changes which meant that I had to keep restarting Node-RED - which happens automatically for me due to nodemon.

You could, of course, also change the timeout or have two file stores, one for less important data set to the default 30s and another for critical data set to 1s or whatever. However, as I say, in that case, you would generally be far better off using a managed data engine instead.

With node-red-contrib-infocache, I will probably eventually implement a secondary persistence feature that uses one of the many databases or REDIS unless someone is able to contribute code to the REDIS backed context store: [https://github.com/node-red/node-red-context-redis](https://github.com/node-red/node-red-context-redis).

Who knows, I might even get time one day to create a context store backed by PouchDB or similar which would be interesting.

---

<div class="post-metadata">

**Author:** ![kuema](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/kuema/32/6542_2.png) [@kuema](https://discourse.nodered.org/u/kuema)\
**Post date:** [4 February 2020 06:59 UTC](https://discourse.nodered.org/t/a-catch-with-persistent-file-stores/21062/37 "2020-02-04T06:59:29Z")

</div>

> [@TotallyInformation](#):
>
> Just to be clear, this is **not** a Node-RED issue, it is a Node.js one.

I would even say it's neither. I'd blame Windows. 😁

In my case, the only reason I went for the file context store was it's simplicity by _not_ requiring an extra service and that the data is not mission critical. I'd definitely go another route if I needed a reliable solution.

A Redis-backed context store sounds good, I had that idea myself some time ago. 🙂

---

<div class="post-metadata">

**Author:** ![dceejay](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/dceejay/32/38_2.png) [@dceejay](https://discourse.nodered.org/u/dceejay)\
**Post date:** [4 February 2020 07:10 UTC](https://discourse.nodered.org/t/a-catch-with-persistent-file-stores/21062/38 "2020-02-04T07:10:29Z")

</div>

> [@TotallyInformation](#):
>
> one day to create a context store backed by PouchDB or similar

Now that indeed would be useful and interesting.

---

<div class="post-metadata">

**Author:** ![TotallyInformation](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/totallyinformation/32/31_2.png) [@TotallyInformation](https://discourse.nodered.org/u/TotallyInformation)\
**Post date:** [4 February 2020 07:10 UTC](https://discourse.nodered.org/t/a-catch-with-persistent-file-stores/21062/39 "2020-02-04T07:10:44Z")

</div>

> [@kuema](#):
>
> I would even say it's neither. I'd blame Windows

Well, the issue I linked to makes it clear that isn't the only issue. The main debate being whether Node.js should create its own solutions or whether that should be handed off to the JavaScript standards committee. The Windows issue makes this problem harder but it isn't the only underlying problem, only the one that highlighted this.

> [@kuema](#):
>
> the only reason I went for the file context store was it's simplicity by _not_ requiring an extra service and that the data is not mission critical. I'd definitely go another route if I needed a reliable solution.

Yes, and that is really what I'd like this discussion to end with I guess - a clear understanding of why the Node-RED persistent context store is good and useful - AND its limitations and potential issues. The "catch" from the title.

> [@kuema](#):
>
> A Redis-backed context store sounds good, I had that idea myself some time ago

It was mentioned in the original design documentation I think but hasn't been implemented. I'm sure it is on the list. Don't know that it would be very difficult to do actually since the actual API interface seems pretty simple. I don't use REDIS so, as I say, I'd probably use PouchDB or something similar myself. However, I really shouldn't take on another project until I've finished some other stuff.

---

<div class="post-metadata">

**Author:** ![TotallyInformation](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/totallyinformation/32/31_2.png) [@TotallyInformation](https://discourse.nodered.org/u/TotallyInformation)\
**Post date:** [4 February 2020 07:12 UTC](https://discourse.nodered.org/t/a-catch-with-persistent-file-stores/21062/40 "2020-02-04T07:12:11Z")

</div>

> [@dceejay](#):
>
> Now that indeed would be useful and interesting.

Oh dear, Dave. Now you've gone and done it! I sense another distracting project bubbling up!

[Previous page](https://discourse.nodered.org/t/a-catch-with-persistent-file-stores/21062.md?page=1)

[Next page](https://discourse.nodered.org/t/a-catch-with-persistent-file-stores/21062.md?page=3)
