Durable and reliable contextStorage - DB/mysql/mariadb based

[Colin]

@jakonnode

FileStorage can solve this may be using a dual file write method where atleast one file is complete/non-broken at any point in time.

If I remember correctly, the context file storage writes a new file with a temporary name then renames the old file and the new one, or something along those lines, so the window for failure is very small.

[jakonnode]

FYI! I am running into a very basic issue with Redis.

Created a ticket separately to avoid cluttering with a different discussion in this...

Global.get fails while using contextStorage(redis)

While using redis contextStorage as default and use global.get() it results in an error as below. "Error: Callback must be a function" Using the below redis package. GitHub - node-red/node-red-context-redis: A Node-RED Context store plugin backed by Redis global.set() goes through and my redis-cli has stored the value too that I can query and check. This doesn't happen with default being filesystem (and obviously memory) I cannot make redis context as default. I would like both durability …

[jakonnode]

Unfortunately, they don't...they write to the same file hence all this issues. Unless it is resolved in a later versions.

[ziga]

I decided to write a custom context storge that will suport different databases. I currently have redis working now I'll look into sql dbs.

@jakonnode what will you be using, sqlite, postgre, mysql?

[Colin]

@jakonnode

Unfortunately, they don't...they write to the same file hence all this issues

I am sure I checked the code at one point. How do you know it is writing to the same file?

[jakonnode]

I am checking redis with persistence but it is running into some issues. If I can't make it work I will use MariaDB/MySQL.

Thanks ziga. Otherwise I plan to write one.

JAK

[Colin]

@jakonnode

Unfortunately, they don't...they write to the same file hence all this issues

Looking at the code that appears to be incorrect. node-red/packages/node_modules/@node-red/runtime/lib/storage/localfilesystem/util.js at master · node-red/node-red · GitHub

[jakonnode]

You're right Colin. Even the old version has rename from a tmpFile to contextFile. I wonder how come it gets corrupt with such atomic operation.

It is easily reproducible when the power to the device goes and it reboots. It is not reproducible when we kill node-red though.

async function writeFileAtomic(storagePath, content) {
    // To protect against file corruption, write to a tmp file first and then
    // rename to the destination file
    let finalFile = storagePath + ".json";
    let tmpFile = finalFile + "."+Date.now()+".tmp";
    await fs.outputFile(tmpFile, content, "utf8");
    return fs.rename(tmpFile,finalFile);
}

[jakonnode]

Atlast, I found a way to deal with this problem of contextFile getting corrupt and the node-red is unable to restart itself.

After evaluating redis context module, realized that all the global.get & context.get will have to be modified as they respond in async manner hence it can't be default contextStore. Same will be the case if we come up with database based contextStore too. Since they are an external service, it is only best to do get as async.

The solution I came up with is that I wrote a javascript function that will find the context file corresponding to the node that writes to the context, checks for likely corruption of the context file. If found corrupt, it traverses through the file and remove the last/corrupt array element.

This jsscript is called before invocation of node red so that it takes care of any possible corrupt file before starting in the node-red.service file.

ExecStartPre=/usr/bin/env node /usr/local/bin/fix-node-red-context-file.js
ExecStart=/usr/bin/env node-red-pi $NODE_OPTIONS $NODE_RED_OPTIONS

Hope this approach works fine. Thanks everyone for prompting me on possible solution(s)

[dceejay]

feel free to also attach the fixup file as an example for others

[ziga]

Here are couple of context storage plugins for Node RED: GitHub - sebenik/node-red-context

Atm there's support for Redis, MySSQL/MariaDB, PostgreSQL and SQLite.

[jakonnode]

Absolutely! More than for others...the team here can review and point me whether it is in the right direction.

Attaching the script that looks for the in-context file, see whether it is corrupt, cleans-up the last corrupt element from the array to make the in-context file (context:localfilesystem). And this is called just before starting the node-red as mentioned above.

Attached the JS file as .txt.
fix-node-red-context-file.js.txt (5.2 KB)

[krambriw]

Maybe this functionality/sanity check should/could be integrated as an init function of Node-RED itself??

[jakonnode]

#1. Not sure everyone will have a use-case like ours - Store & Forward that uses a persistence store like localfilesystem
#2. If they do, they may not be fine to let go the last corrupt element silently and proceed with.
#3. They may not have the device restarting causing this frequently like ours. (many in this thread itself pointed out that it is very unusual)
Hence I would recommend/request to keep this outside of the localfilesystem implementation.
Edit: We can always keep this property driven by explicitly agreeing to something like this and keep it in init and not a default behavior.

[jakonnode]

@ziga

Here are couple of context storage plugins for Node RED: GitHub - sebenik/node-red-context

Atm there's support for Redis, MySSQL/MariaDB, PostgreSQL and SQLite.

Ziga, many thanks. I did try the MySQL. Broadly it works especially it makes the node-red survive the restart without getting into trouble. Found few issues... All the context values are stored in one row - it is getting updated instead of inserting. That makes it worse on performance, running count for showing the status etc. As I said earlier, I have to modified the context.get and context.set with callbacks. Somewhere I see that the set/update is getting called frequently - may be it is querying for status (count). This is very detrimental for performance on a RPi-SD Card. I will also try redis & sqllist.

[ziga]

Make sure to use latest 1.0.5 version as per bug that was reported by @gregorius

Custom context storage plugins

Good work on creating a bunch of storage adapters, will make many folks happy. I was looking through the codebase and I have a question about the set functionality. I assume that the BaseContext set method ... set(scope, key, value, callback) { return this.queue.enqueue(async () => { if (!key) { throw new Error('Key is not defined'); } if (!callback || typeof callback !== 'function') { throw new Error('Callback is not a function'); } if (!Array.isArray(key)) { key = [k…

and in 1.0.5 additional bug was fixed when retrieving keys that don't exist.

@jakonnode

All the context values are stored in one row - it is getting updated instead of inserting.

This is true for "nested" keys yes, e.g if you are storing a value to object a.b = 12, in database everything stored under object a gets stored as a value, so above becomes key: a, value: { "b": 12 }