# Context File corrupt/truncated - node-red auto-restart failure

**URL:** <https://discourse.nodered.org/t/context-file-corrupt-truncated-node-red-auto-restart-failure/79897>\
**Category:** General\
**Created:** [18 July 2023 07:12 UTC](https://discourse.nodered.org/t/context-file-corrupt-truncated-node-red-auto-restart-failure/79897 "2023-07-18T07:12:38Z")\
**Posts on this page:** 20\
**Page:** 2

<div class="post-metadata">

**Author:** ![jakonnode](https://avatars.discourse-cdn.com/v4/letter/j/3ec8ea/32.png) [@jakonnode](https://discourse.nodered.org/u/jakonnode)\
**Post date:** [21 July 2023 08:50 UTC](https://discourse.nodered.org/t/context-file-corrupt-truncated-node-red-auto-restart-failure/79897/21 "2023-07-21T08:50:01Z")

</div>

> [@Colin](#):
>
> A node red command line option to treat context read failures as a warning might be worth while.

Are you suggesting a command line check before starting node-red for the context read failures?

---

<div class="post-metadata">

**Author:** ![TotallyInformation](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/totallyinformation/32/31_2.png) [@TotallyInformation](https://discourse.nodered.org/u/TotallyInformation)\
**Post date:** [21 July 2023 12:46 UTC](https://discourse.nodered.org/t/context-file-corrupt-truncated-node-red-auto-restart-failure/79897/22 "2023-07-21T12:46:33Z")

</div>

> [@jakonnode](#):
>
> Current problem is that in some sites, such issue happens where there is no UPS, and the device won't come back and we don't know for few hours and some cases even days. Wanted to fix that case also.

If using a Pi and SD-Cards, you MUST use a UPS and a well featured power supply otherwise you risk corrupting the whole card. This used to be a major issue. It isn't so bad on newer Pi's but is still a risk. At least have sufficient local power to let the Pi shut down correctly. This isn't a Node-RED issue, it is a Pi hardware issue.

---

<div class="post-metadata">

**Author:** ![bobcroft](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/bobcroft/32/52443_2.png) [@bobcroft](https://discourse.nodered.org/u/bobcroft)\
**Post date:** [24 July 2023 08:35 UTC](https://discourse.nodered.org/t/context-file-corrupt-truncated-node-red-auto-restart-failure/79897/23 "2023-07-24T08:35:53Z")

</div>

I don't do much with Rpi's but I am certain I saw a UPS 'shield' or similar that fitted a pi and allowed it to finish any writes ans shut down in a controlled manner. It was probably on Tindie and not very expensive. Such a device would help reduce corruption due to random shut downs. The data you are storing seems to have some importance to you, that being the case why not use a better storage medium like at least an SSD?

---

<div class="post-metadata">

**Author:** ![jakonnode](https://avatars.discourse-cdn.com/v4/letter/j/3ec8ea/32.png) [@jakonnode](https://discourse.nodered.org/u/jakonnode)\
**Post date:** [24 July 2023 09:30 UTC](https://discourse.nodered.org/t/context-file-corrupt-truncated-node-red-auto-restart-failure/79897/24 "2023-07-24T09:30:49Z")

</div>

Sorry for being direct - kindly don't suggest - use UPS, use Server type machines, Linux or Windows on a data center or use RDBMS or use a RAID 5 type disks etc. Unfortunately, we don't have any of these and we have hundreds of installations at the industrial site with extremely fragile (mobile) network or RPi type of hardware. And we can't tell our customers that we don't support without these. Can't change these now. ☹

Plus this problem happens in a properly UPS-backed up Windows box so the problem is real.

I am not highlighting any inadequacy or gap in node-red and I understand this is not a sunny day use-case for node-red.

Looks like a database (MySQL) or as Colin suggested writing to two files (journaling) would help - I am yet to try it. Any other similar suggestions on top of node-red are welcome!

---

<div class="post-metadata">

**Author:** ![jbudd](https://avatars.discourse-cdn.com/v4/letter/j/5f8ce5/32.png) [@jbudd](https://discourse.nodered.org/u/jbudd)\
**Post date:** [24 July 2023 10:00 UTC](https://discourse.nodered.org/t/context-file-corrupt-truncated-node-red-auto-restart-failure/79897/25 "2023-07-24T10:00:17Z")

</div>

> [@jakonnode](#):
>
> Plus this problem happens in a properly UPS-backed up Windows box so the problem is real.

What happens in the event of a power outage - Does the Windows box attempt a clean shutdown, yet the context store still gets corrupted?  
Does the UPS provide power for long enough for the shutdown to complete?  
Is the context store device on this Windows box also an SD card, an SSD, hard disk?

---

<div class="post-metadata">

**Author:** ![Colin](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/colin/32/17040_2.png) [@Colin](https://discourse.nodered.org/u/Colin)\
**Post date:** [24 July 2023 10:15 UTC](https://discourse.nodered.org/t/context-file-corrupt-truncated-node-red-auto-restart-failure/79897/26 "2023-07-24T10:15:28Z")

</div>

Further to @jbudd's questions. You started by saying this happens on power fail on the non-ups devices. Have you confirmed that it _is_ on power fail on those devices, or is that just conjecture?

Also on the Windows machine, is it on power fail that you get the problem?

---

<div class="post-metadata">

**Author:** ![jakonnode](https://avatars.discourse-cdn.com/v4/letter/j/3ec8ea/32.png) [@jakonnode](https://discourse.nodered.org/u/jakonnode)\
**Post date:** [24 July 2023 10:28 UTC](https://discourse.nodered.org/t/context-file-corrupt-truncated-node-red-auto-restart-failure/79897/27 "2023-07-24T10:28:38Z")

</div>

In a particular instance on Windows - Event Viewer says it is rebooted without shutting down first - unexpectedly. This is on UPS and likely have crashed for some reason. It has a regular SATA (Toshiba ATA Disk) drive. I don't know whether it is a power trip or windows crash in this case.

(As said, such a scenario frequents in RPi like device)

The reason why I had mentioned RPi device is that - ideally would like a common solution between Windows, Linux & RPi installations - hence can't start with MySQL as this could be heavier.

![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/1/c/1c553af4f34a6b02c6ac710a94a52edd61832fda.png)

---

<div class="post-metadata">

**Author:** ![M-a-r-t-i-n](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/m-a-r-t-i-n/32/52391_2.png) [@M-a-r-t-i-n](https://discourse.nodered.org/u/M-a-r-t-i-n)\
**Post date:** [24 July 2023 11:03 UTC](https://discourse.nodered.org/t/context-file-corrupt-truncated-node-red-auto-restart-failure/79897/28 "2023-07-24T11:03:24Z")

</div>

This may seem like a crazy idea.. but..

why not use a directory in the file system as a FIFO buffer.  
Each data update is wirtten as a (separate) file into this directory. you then have another flow reading these files and sending them on via mqtt. If the mqtt link fails, then the directory slowly fills up. if the pi crashes, then on restarting you either send on the files via mqtt, so you have no data loss, or you may just want to erase them all and start again. by keeping each file to one update, if one becomes corrupt, you still have all the others. Also a smaller file may be less susceptible to becoming corrupt??

---

<div class="post-metadata">

**Author:** ![jbudd](https://avatars.discourse-cdn.com/v4/letter/j/5f8ce5/32.png) [@jbudd](https://discourse.nodered.org/u/jbudd)\
**Post date:** [24 July 2023 11:19 UTC](https://discourse.nodered.org/t/context-file-corrupt-truncated-node-red-auto-restart-failure/79897/29 "2023-07-24T11:19:47Z")

</div>

Well that looks like a problem internal to the PC, such as a dodgy power supply or motherboard issue. Maybe even corruption of an OS file might cause it. Whatever, the UPS was unable to intervene.

Thinking about your SD card based systems, I wonder if something about your flows or the Node-red context storage processes makes this file especially prone to corruption (possibly the slow stage of writing to flash) or if you are perhaps unknowingly getting other corrupt files too.

> [@jakonnode](#):
>
> Yes, it happens fairly few times and we get to know only after few hours that it was done

Are you certain that the device rebooted after this event?

> [@jakonnode](#):
>
> we are sending data to server using guaranteed delivery flow - and this flow writes to the local context first (queue) and the data transfer flow would send them by reading from this context storage

Can you show us this part of your flow?

---

<div class="post-metadata">

**Author:** ![jakonnode](https://avatars.discourse-cdn.com/v4/letter/j/3ec8ea/32.png) [@jakonnode](https://discourse.nodered.org/u/jakonnode)\
**Post date:** [24 July 2023 12:21 UTC](https://discourse.nodered.org/t/context-file-corrupt-truncated-node-red-auto-restart-failure/79897/30 "2023-07-24T12:21:58Z")

</div>

> [@jbudd](#):
>
> Are you certain that the device rebooted after this event?

Yes, this corresponds to a reboot event in the Event Viewer.

> [@jbudd](#):
>
> Can you show us this part of your flow?

Sure, it is taken from the Guaranteed flow submitted by Colin. Sub flow is used almost as-is.  
[https://flows.nodered.org/flow/05e6d61f14ef6af763ec4cfd1049ab61](https://flows.nodered.org/flow/05e6d61f14ef6af763ec4cfd1049ab61)

---

<div class="post-metadata">

**Author:** ![jakonnode](https://avatars.discourse-cdn.com/v4/letter/j/3ec8ea/32.png) [@jakonnode](https://discourse.nodered.org/u/jakonnode)\
**Post date:** [24 July 2023 12:33 UTC](https://discourse.nodered.org/t/context-file-corrupt-truncated-node-red-auto-restart-failure/79897/31 "2023-07-24T12:33:13Z")

</div>

Yeah! this looks like a possible approach I was contemplating as mentioned above.

The problem is to not write the transaction data directly into the context file. I can write it to CSV file (with some minimal buffering - say every 10 seconds or so) and the csv file name and order is mentioned in the context file in the end. i.e context file will have only the filename(s). Minimize the risk of corruption and the context file will remain small and the CSV file is easily recoverable. (I also have the option to write x number of records per CSV.)

> [@jakonnode](#):
>
> I am thinking of an alternative where we write it to a CSV and not to the context file. And keep may be file name(s) of the CSV in the context storage. That way incomplete records/rows can be ignored. Atleast the system won't fail to restart on incomplete records as it does today and there are no semantic issues with the file parsing.

One thing that I wanted to bounce off with this team is that - there is always a possibility of truncation of context file every time someone writes into it when something happens to the node-red process (which is likely in any scenario). There has to be a fallback - like continue on error, recover context file upon restart etc. Today the node-red doesn't restart when it cannot load the context file which is a serious issue isn't?

---

<div class="post-metadata">

**Author:** ![Colin](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/colin/32/17040_2.png) [@Colin](https://discourse.nodered.org/u/Colin)\
**Post date:** [24 July 2023 12:41 UTC](https://discourse.nodered.org/t/context-file-corrupt-truncated-node-red-auto-restart-failure/79897/32 "2023-07-24T12:41:23Z")

</div>

> [@jakonnode](#):
>
> Today the node-red doesn't restart when it cannot load the context file which is a serious issue isn't?

Yes, I think a settings.js option to treat corrupted context as a warning rather than a fatal error might be one possibility.

---

<div class="post-metadata">

**Author:** ![Colin](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/colin/32/17040_2.png) [@Colin](https://discourse.nodered.org/u/Colin)\
**Post date:** [24 July 2023 12:48 UTC](https://discourse.nodered.org/t/context-file-corrupt-truncated-node-red-auto-restart-failure/79897/33 "2023-07-24T12:48:50Z")

</div>

Does anyone know where the context save code is located? I would like to have a look at it.

---

<div class="post-metadata">

**Author:** ![thebaldgeek](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/thebaldgeek/32/93523_2.png) [@thebaldgeek](https://discourse.nodered.org/u/thebaldgeek)\
**Post date:** [24 July 2023 19:18 UTC](https://discourse.nodered.org/t/context-file-corrupt-truncated-node-red-auto-restart-failure/79897/34 "2023-07-24T19:18:23Z")

</div>

My Windows based Node-RED has this exact issue for the past 3+ years.  
Node-RED will crash (usually the screenshot Node (the author is not responsive) due to puppeteer doing something it should not) and not restart due to borked context file.  
I have about 50+ context files and its a real pain to figure out which one it is due to the long random file names.  
Also when it crashes, I am usually in a hurry to get it back up (public facing website) so end up just deleting all the context files and then the site comes up, but all the tables etc are black and the users complain.

But, for a clean stop/start, the context is great.

I use the app 'restart on crash' to monitor Node-RED and restart it 15 seconds after it crashes.

I guess my point is, its not just an issue for a Pi on the edge, but also for a solid Windows 'server class' machine as well. And yes, my PC is on a massive UPS as I cant not have it go down. Its just random Node-RED crashes that cause the issue.

---

<div class="post-metadata">

**Author:** ![Colin](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/colin/32/17040_2.png) [@Colin](https://discourse.nodered.org/u/Colin)\
**Post date:** [24 July 2023 20:27 UTC](https://discourse.nodered.org/t/context-file-corrupt-truncated-node-red-auto-restart-failure/79897/35 "2023-07-24T20:27:30Z")

</div>

> [@thebaldgeek](#):
>
> Its just random Node-RED crashes that cause the issue.

I find this very strange. I would expect the code that saves the context to write it out to a temporary file and only when that was complete would it rename the new file as the current one. If my assumption is correct it is difficult to see how a node-red crash could cause the problem. Are you running reasonably up to date versions of node-red and nodejs?

@jakonnode are you also running reasonably up to date versions of node-red and nodejs in the failing systems?

---

<div class="post-metadata">

**Author:** ![jbudd](https://avatars.discourse-cdn.com/v4/letter/j/5f8ce5/32.png) [@jbudd](https://discourse.nodered.org/u/jbudd)\
**Post date:** [24 July 2023 20:33 UTC](https://discourse.nodered.org/t/context-file-corrupt-truncated-node-red-auto-restart-failure/79897/36 "2023-07-24T20:33:47Z")

</div>

That could be valuable info.

The PC itself does not restart?

You don't happen to use a similar method to write the context variables as the OP?

For what it's worth here's a little shell script to look for invalid JSON files in the context directory.  
Probably it could easily be converted to PowerShell.

```auto
#! /bin/bash
NODEREDHOME=~/.node-red
CONTEXTDIR=$NODEREDHOME/context

for f in $(find $CONTEXTDIR -type f -print)
do
  if python -mjson.tool $f >/dev/null 2>&1 # Exit code non zero if file is not valid json
  then true
  else echo $f is bad
  fi
done

```

---

<div class="post-metadata">

**Author:** ![Colin](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/colin/32/17040_2.png) [@Colin](https://discourse.nodered.org/u/Colin)\
**Post date:** [24 July 2023 20:58 UTC](https://discourse.nodered.org/t/context-file-corrupt-truncated-node-red-auto-restart-failure/79897/37 "2023-07-24T20:58:55Z")

</div>

Found the code. In [https://github.com/node-red/node-red/blob/master/packages/node\_modules/%40node-red/runtime/lib/nodes/context/localfilesystem.js](https://github.com/node-red/node-red/blob/master/packages/node_modules/%40node-red/runtime/lib/nodes/context/localfilesystem.js)

```auto
async function writeFileAtomic(storagePath, content) {
    // To protect against file corruption, write to a tmp file first and then
    // rename to the destination file
    let finalFile = storagePath + ".json";
    let tmpFile = finalFile + "."+Date.now()+".tmp";
    await fs.outputFile(tmpFile, content, "utf8");
    return fs.rename(tmpFile,finalFile);
}

```

It is difficult to see how that can end up with a corrupted file in the case of a node-red crash. If the crash happens before the rename then the original file is still there, if it happens after the rename then the new one should be there complete.  
The above code was introduced in node-red v1.0 I believe ([Use a more atomic process for writing context files · node-red/node-red@10077ae · GitHub](https://github.com/node-red/node-red/commit/10077ae7500431816176840be1549cb5a9c8a2a5))

---

<div class="post-metadata">

**Author:** ![thebaldgeek](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/thebaldgeek/32/93523_2.png) [@thebaldgeek](https://discourse.nodered.org/u/thebaldgeek)\
**Post date:** [24 July 2023 21:34 UTC](https://discourse.nodered.org/t/context-file-corrupt-truncated-node-red-auto-restart-failure/79897/38 "2023-07-24T21:34:37Z")

</div>

My PC currently is at 102 days uptime, so no, the PC does not reboot at all.  
Just Node-RED restarts. Sometimes it will restart cleanly, sometimes it will not.  
Node-RED v3.0.2  
nodeJS v18.15.0

I have a lot of 'template' nodes with the option at the bottom 'Reload last value on refresh'. I suspect that a lot of context files are from those nodes.  
I also have a lot of flow.set and global.set context commands in my function nodes to pass data around and to ensure arrays are loaded up on a restart etc

---

<div class="post-metadata">

**Author:** ![hotNipi](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/hotnipi/32/383_2.png) [@hotNipi](https://discourse.nodered.org/u/hotNipi)\
**Post date:** [25 July 2023 07:06 UTC](https://discourse.nodered.org/t/context-file-corrupt-truncated-node-red-auto-restart-failure/79897/39 "2023-07-25T07:06:25Z")

</div>

> [@thebaldgeek](#):
>
> Node-RED will crash (usually the screenshot Node (the author is not responsive) due to puppeteer doing something it should not) and not restart due to borked context file.

If it is `node-red-contrib-screenshot` - the last update is Nov 24, 2016. Meanwhile the underlying library `screenshot-desktop` got last (windows related) update 2 weeks ago.  
The `puppeteer` has 286 open issues. (well of course not all of them are really issues but...)

For me those alone will make even

> a solid Windows 'server class' machine

kind of irrelevant.

If the running process **forced** to fail hard, I cant see any reason to expect fully functional recovery from it.

---

<div class="post-metadata">

**Author:** ![Colin](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/colin/32/17040_2.png) [@Colin](https://discourse.nodered.org/u/Colin)\
**Post date:** [25 July 2023 07:44 UTC](https://discourse.nodered.org/t/context-file-corrupt-truncated-node-red-auto-restart-failure/79897/40 "2023-07-25T07:44:37Z")

</div>

> [@hotNipi](#):
>
> If the running process **forced** to fail hard, I cant see any reason to expect fully functional recovery from it.

Given the code I posted earlier, can you see how a node-red crash could cause a context file to be corrupted?

@thebaldgeek can you post the section of .node-red/settings.js relevant to context storage please?

**[Edit]**@thebaldgeek also how often are you seeing the corruption?

[Previous page](https://discourse.nodered.org/t/context-file-corrupt-truncated-node-red-auto-restart-failure/79897.md?page=1)

[Next page](https://discourse.nodered.org/t/context-file-corrupt-truncated-node-red-auto-restart-failure/79897.md?page=3)