# Durable and reliable contextStorage - DB/mysql/mariadb based

**URL:** <https://discourse.nodered.org/t/durable-and-reliable-contextstorage-db-mysql-mariadb-based/95085>\
**Category:** General\
**Created:** [3 February 2025 12:31 UTC](https://discourse.nodered.org/t/durable-and-reliable-contextstorage-db-mysql-mariadb-based/95085 "2025-02-03T12:31:54Z")\
**Posts on this page:** 17\
**Page:** 3

<div class="post-metadata">

**Author:** ![Colin](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/colin/32/17040_2.png) [@Colin](https://discourse.nodered.org/u/Colin)\
**Post date:** [4 February 2025 21:00 UTC](https://discourse.nodered.org/t/durable-and-reliable-contextstorage-db-mysql-mariadb-based/95085/41 "2025-02-04T21:00:58Z")

</div>

> [@jakonnode](#):
>
> FileStorage can solve this may be using a dual file write method where atleast one file is complete/non-broken at any point in time.

If I remember correctly, the context file storage writes a new file with a temporary name then renames the old file and the new one, or something along those lines, so the window for failure is very small.

---

<div class="post-metadata">

**Author:** ![jakonnode](https://avatars.discourse-cdn.com/v4/letter/j/3ec8ea/32.png) [@jakonnode](https://discourse.nodered.org/u/jakonnode)\
**Post date:** [6 February 2025 14:27 UTC](https://discourse.nodered.org/t/durable-and-reliable-contextstorage-db-mysql-mariadb-based/95085/42 "2025-02-06T14:27:27Z")

</div>

FYI! I am running into a very basic issue with Redis. ☹

Created a ticket separately to avoid cluttering with a different discussion in this...

> [@Global.get fails while using contextStorage(redis)](https://discourse.nodered.org/t/global-get-fails-while-using-contextstorage-redis/95183):
>
> While using redis contextStorage as default and use global.get() it results in an error as below. "Error: Callback must be a function" Using the below redis package. [GitHub - node-red/node-red-context-redis: A Node-RED Context store plugin backed by Redis](https://github.com/node-red/node-red-context-redis) global.set() goes through and my redis-cli has stored the value too that I can query and check. This doesn't happen with default being filesystem (and obviously memory) I cannot make redis context as default. I would like both durability …

---

<div class="post-metadata">

**Author:** ![jakonnode](https://avatars.discourse-cdn.com/v4/letter/j/3ec8ea/32.png) [@jakonnode](https://discourse.nodered.org/u/jakonnode)\
**Post date:** [6 February 2025 14:28 UTC](https://discourse.nodered.org/t/durable-and-reliable-contextstorage-db-mysql-mariadb-based/95085/43 "2025-02-06T14:28:11Z")

</div>

Unfortunately, they don't...they write to the same file hence all this issues. Unless it is resolved in a later versions.

---

<div class="post-metadata">

**Author:** ![ziga](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/ziga/32/98138_2.png) [@ziga](https://discourse.nodered.org/u/ziga)\
**Post date:** [6 February 2025 14:47 UTC](https://discourse.nodered.org/t/durable-and-reliable-contextstorage-db-mysql-mariadb-based/95085/44 "2025-02-06T14:47:12Z")

</div>

I decided to write a custom context storge that will suport different databases. I currently have redis working now I'll look into sql dbs.

@jakonnode what will you be using, sqlite, postgre, mysql?

---

<div class="post-metadata">

**Author:** ![Colin](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/colin/32/17040_2.png) [@Colin](https://discourse.nodered.org/u/Colin)\
**Post date:** [6 February 2025 15:07 UTC](https://discourse.nodered.org/t/durable-and-reliable-contextstorage-db-mysql-mariadb-based/95085/45 "2025-02-06T15:07:29Z")

</div>

> [@jakonnode](#):
>
> Unfortunately, they don't...they write to the same file hence all this issues

I am sure I checked the code at one point. How do you know it is writing to the same file?

---

<div class="post-metadata">

**Author:** ![jakonnode](https://avatars.discourse-cdn.com/v4/letter/j/3ec8ea/32.png) [@jakonnode](https://discourse.nodered.org/u/jakonnode)\
**Post date:** [6 February 2025 15:26 UTC](https://discourse.nodered.org/t/durable-and-reliable-contextstorage-db-mysql-mariadb-based/95085/46 "2025-02-06T15:26:06Z")

</div>

I am checking redis with persistence but it is running into some issues. If I can't make it work I will use MariaDB/MySQL.

Thanks ziga. Otherwise I plan to write one.

JAK

---

<div class="post-metadata">

**Author:** ![Colin](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/colin/32/17040_2.png) [@Colin](https://discourse.nodered.org/u/Colin)\
**Post date:** [6 February 2025 15:33 UTC](https://discourse.nodered.org/t/durable-and-reliable-contextstorage-db-mysql-mariadb-based/95085/47 "2025-02-06T15:33:05Z")

</div>

> [@jakonnode](#):
>
> Unfortunately, they don't...they write to the same file hence all this issues

Looking at the code that appears to be incorrect. [node-red/packages/node\_modules/@node-red/runtime/lib/storage/localfilesystem/util.js at master · node-red/node-red · GitHub](https://github.com/node-red/node-red/blob/master/packages/node_modules/%40node-red/runtime/lib/storage/localfilesystem/util.js)

---

<div class="post-metadata">

**Author:** ![jakonnode](https://avatars.discourse-cdn.com/v4/letter/j/3ec8ea/32.png) [@jakonnode](https://discourse.nodered.org/u/jakonnode)\
**Post date:** [7 February 2025 17:37 UTC](https://discourse.nodered.org/t/durable-and-reliable-contextstorage-db-mysql-mariadb-based/95085/48 "2025-02-07T17:37:11Z")

</div>

You're right Colin. Even the old version has rename from a tmpFile to contextFile. I wonder how come it gets corrupt with such atomic operation.

It is easily reproducible when the power to the device goes and it reboots. It is not reproducible when we kill node-red though.

```auto

async function writeFileAtomic(storagePath, content) {
    // To protect against file corruption, write to a tmp file first and then
    // rename to the destination file
    let finalFile = storagePath + ".json";
    let tmpFile = finalFile + "."+Date.now()+".tmp";
    await fs.outputFile(tmpFile, content, "utf8");
    return fs.rename(tmpFile,finalFile);
}

```

---

<div class="post-metadata">

**Author:** ![jakonnode](https://avatars.discourse-cdn.com/v4/letter/j/3ec8ea/32.png) [@jakonnode](https://discourse.nodered.org/u/jakonnode)\
**Post date:** [7 February 2025 17:46 UTC](https://discourse.nodered.org/t/durable-and-reliable-contextstorage-db-mysql-mariadb-based/95085/49 "2025-02-07T17:46:07Z")

</div>

Atlast, I found a way to deal with this problem of contextFile getting corrupt and the node-red is unable to restart itself.

After evaluating redis context module, realized that all the global.get & context.get will have to be modified as they respond in async manner hence it can't be default contextStore. Same will be the case if we come up with database based contextStore too. Since they are an external service, it is only best to do get as async.

The solution I came up with is that I wrote a javascript function that will find the context file corresponding to the node that writes to the context, checks for likely corruption of the context file. If found corrupt, it traverses through the file and remove the last/corrupt array element.

This jsscript is called before invocation of node red so that it takes care of any possible corrupt file before starting in the node-red.service file.

```auto
ExecStartPre=/usr/bin/env node /usr/local/bin/fix-node-red-context-file.js
ExecStart=/usr/bin/env node-red-pi $NODE_OPTIONS $NODE_RED_OPTIONS

```

Hope this approach works fine. Thanks everyone for prompting me on possible solution(s)

---

<div class="post-metadata">

**Author:** ![dceejay](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/dceejay/32/38_2.png) [@dceejay](https://discourse.nodered.org/u/dceejay)\
**Post date:** [7 February 2025 18:09 UTC](https://discourse.nodered.org/t/durable-and-reliable-contextstorage-db-mysql-mariadb-based/95085/50 "2025-02-07T18:09:13Z")

</div>

feel free to also attach the fixup file as an example for others 🙂

---

<div class="post-metadata">

**Author:** ![ziga](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/ziga/32/98138_2.png) [@ziga](https://discourse.nodered.org/u/ziga)\
**Post date:** [8 February 2025 17:35 UTC](https://discourse.nodered.org/t/durable-and-reliable-contextstorage-db-mysql-mariadb-based/95085/51 "2025-02-08T17:35:29Z")

</div>

Here are couple of context storage plugins for Node RED: [GitHub - sebenik/node-red-context](https://github.com/sebenik/node-red-context?tab=readme-ov-file)

Atm there's support for Redis, MySSQL/MariaDB, PostgreSQL and SQLite.

---

<div class="post-metadata">

**Author:** ![jakonnode](https://avatars.discourse-cdn.com/v4/letter/j/3ec8ea/32.png) [@jakonnode](https://discourse.nodered.org/u/jakonnode)\
**Post date:** [10 February 2025 05:21 UTC](https://discourse.nodered.org/t/durable-and-reliable-contextstorage-db-mysql-mariadb-based/95085/52 "2025-02-10T05:21:55Z")

</div>

Absolutely! More than for others...the team here can review and point me whether it is in the right direction.

Attaching the script that looks for the in-context file, see whether it is corrupt, cleans-up the last corrupt element from the array to make the in-context file (context:localfilesystem). And this is called just before starting the node-red as mentioned above.

Attached the JS file as .txt.  
[fix-node-red-context-file.js.txt](https://discourse.nodered.org/uploads/short-url/ldOibbyMGFqUKmWiPZDK0JJbJtT.txt) (5.2 KB)

---

<div class="post-metadata">

**Author:** ![krambriw](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/krambriw/32/5429_2.png) [@krambriw](https://discourse.nodered.org/u/krambriw)\
**Post date:** [10 February 2025 07:45 UTC](https://discourse.nodered.org/t/durable-and-reliable-contextstorage-db-mysql-mariadb-based/95085/53 "2025-02-10T07:45:28Z")

</div>

Maybe this functionality/sanity check should/could be integrated as an init function of Node-RED itself??

---

<div class="post-metadata">

**Author:** ![jakonnode](https://avatars.discourse-cdn.com/v4/letter/j/3ec8ea/32.png) [@jakonnode](https://discourse.nodered.org/u/jakonnode)\
**Post date:** [10 February 2025 09:01 UTC](https://discourse.nodered.org/t/durable-and-reliable-contextstorage-db-mysql-mariadb-based/95085/54 "2025-02-10T09:01:43Z")

</div>

#1. Not sure everyone will have a use-case like ours - Store & Forward that uses a persistence store like localfilesystem  
#2. If they do, they may not be fine to let go the last corrupt element silently and proceed with.  
#3. They may not have the device restarting causing this frequently like ours. (many in this thread itself pointed out that it is very unusual)  
Hence I would recommend/request to keep this outside of the localfilesystem implementation.  
Edit: We can always keep this property driven by explicitly agreeing to something like this and keep it in init and not a default behavior.

---

<div class="post-metadata">

**Author:** ![jakonnode](https://avatars.discourse-cdn.com/v4/letter/j/3ec8ea/32.png) [@jakonnode](https://discourse.nodered.org/u/jakonnode)\
**Post date:** [10 February 2025 13:11 UTC](https://discourse.nodered.org/t/durable-and-reliable-contextstorage-db-mysql-mariadb-based/95085/55 "2025-02-10T13:11:53Z")

</div>

> [@ziga](#):
>
> Here are couple of context storage plugins for Node RED: [GitHub - sebenik/node-red-context](https://github.com/sebenik/node-red-context?tab=readme-ov-file)
> 
> Atm there's support for Redis, MySSQL/MariaDB, PostgreSQL and SQLite.

Ziga, many thanks. I did try the MySQL. Broadly it works especially it makes the node-red survive the restart without getting into trouble. Found few issues... All the context values are stored in one row - it is getting updated instead of inserting. That makes it worse on performance, running count for showing the status etc. As I said earlier, I have to modified the context.get and context.set with callbacks. Somewhere I see that the set/update is getting called frequently - may be it is querying for status (count). This is very detrimental for performance on a RPi-SD Card. I will also try redis & sqllist.

---

<div class="post-metadata">

**Author:** ![ziga](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/ziga/32/98138_2.png) [@ziga](https://discourse.nodered.org/u/ziga)\
**Post date:** [10 February 2025 13:47 UTC](https://discourse.nodered.org/t/durable-and-reliable-contextstorage-db-mysql-mariadb-based/95085/56 "2025-02-10T13:47:36Z")

</div>

Make sure to use latest `1.0.5` version as per bug that was reported by @gregorius

> [@Custom context storage plugins](https://discourse.nodered.org/t/custom-context-storage-plugins/95249/2):
>
> Good work on creating a bunch of storage adapters, will make many folks happy. I was looking through the [codebase](https://github.com/sebenik/node-red-context/blob/master/packages/base/index.js) and I have a question about the set functionality. I assume that the BaseContext [set method](https://github.com/sebenik/node-red-context/blob/55659138d198a5bdf9a0c847479709a4c2a767ef/packages/base/index.js#L73) ... set(scope, key, value, callback) { return this.queue.enqueue(async () =\> { if (!key) { throw new Error('Key is not defined'); } if (!callback || typeof callback !== 'function') { throw new Error('Callback is not a function'); } if (!Array.isArray(key)) { key = [k…

and in 1.0.5 additional bug was fixed when retrieving keys that don't exist.

> [@jakonnode](#):
>
> All the context values are stored in one row - it is getting updated instead of inserting.

This is true for "nested" keys yes, e.g if you are storing a value to object a.b = 12, in database everything stored under object a gets stored as a value, so above becomes key: `a`, value: `{ "b": 12 }`

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/1X/d073cd938eafa2e558d7c2cd59003b3ef4963033.png) [@system](https://discourse.nodered.org/u/system)\
**Post date:** [11 May 2025 13:48 UTC](https://discourse.nodered.org/t/durable-and-reliable-contextstorage-db-mysql-mariadb-based/95085/57 "2025-05-11T13:48:18Z")

</div>

Closed on 11 May 2025 13:48 UTC

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.

[Previous page](https://discourse.nodered.org/t/durable-and-reliable-contextstorage-db-mysql-mariadb-based/95085.md?page=2)