Global, Flow and Node Context File Storage Interval

[davidz]

If there are global, flow and node context file storage enabled, are the global, flow and node context files saved at the same time, or they are saved at different time?

There is only one context file save interval setting in the "settings.js" file. How does Node-RED handle multiple types of context file storage? Right now, the files are saved at different sub-directories under /context directory.

If the context files are saved at different time, then the probability of getting the context files corrupted increases when the number of context files increases.

This question is related to the context file corruption issue, which causes Node-RED restart failure:

Context File corrupt/truncated - node-red auto-restart failure

When my device restarts abruptly - the context file get corrupt because of the truncated json. Automatic restart of node-red fails due to this. One way is to go with a DB based context storage instead of file but MySQL or SQLLite could be little heavyduty for the device sometimes. Have anyone solved this problem using something like untruncate-json or equivalent ? TIA.

We can duplicate this Node-RED restart failure due to context file corruption for the latest Node-RED version 3.0.2.

[knolleary]

Whenever a write is made to any context scope (node/flow/global) it starts a timer (if one isn't already running). When that timer expires, it will flush out to disk any context scopes that have had changes made since the last flush happened.

[Colin]

@davidz

We can duplicate this Node-RED restart failure due to context file corruption

How are you duplicating the corruption?

[davidz]

Assume that there are 8 flow context files, and 16 node context files, does it mean that there are 25 timers recording the write time (1 global, 8 flow, and 16 nodes)? Or there is just one overall timer?

If there is a timer for each context file node, then we should keep the number of context files low?

In a function node, if we have a command at line 1:
Line 1: flow.set("A", true);
...
Line 10: flow.set("A", true);

So at both lines 1 and10, the timer for context file "A" will be updated?

[davidz]

We use several sensors to send data, and use flow and node context to save the sensor information. The sensor data arrives in less than 1 second interval. When turn on/off an RPI multiple times quickly without using proper "shutdown", then the context file could be corrupted (with the 30s default saving interval). RPI can always boot normally, but Node-RED will fail to load.

[knolleary]

There is a single timer for the whole store.

@davidz

In a function node, if we have a command at line 1:
Line 1: flow.set("A", true);
...
Line 10: flow.set("A", true);

So at both lines 1 and10, the timer for context file "A" will be updated?

Assuming no other activity, line 1 will start the timer. Assuming Line 10 happens within the flush interval, it won't start a new timer because one is already scheduled.

This is why the interval is described as the maximum time between flushes when there is data to flush.

[davidz]

Got it. Thanks much for the explanation!

[Colin]

@davidz

When turn on/off an RPI multiple times quickly without using proper "shutdown", then the context file could be corrupted

OK, it wasn't clear from the initial post that the problem appeared after pulling the plug.