Custom context storage plugins

Share Your Nodes
1

As per debate started here: Durable and reliable contextStorage - DB/mysql/mariadb based

there are now several plugins available to store Node RED context in either Redis, SQLite, PostgreSQL or MySQL/MariaDB.

More here: GitHub - sebenik/node-red-context

4 Likes
2

Good work on creating a bunch of storage adapters, will make many folks happy.

I was looking through the codebase and I have a question about the set functionality.

I assume that the BaseContext set method ...

 set(scope, key, value, callback) {
    return this.queue.enqueue(async () => {
      if (!key) { throw new Error('Key is not defined'); }
      if (!callback || typeof callback !== 'function') { throw new Error('Callback is not a function'); }

      if (!Array.isArray(key)) { key = [key]; }
      if (!Array.isArray(value)) { value = [value]; }

      if (key.length !== value.length) { throw new Error('Number of values don\'t match number of keys'); }

... is called when I do something similar to ...

flow.set("responses", content, () => { node.send(msg) })

... in a function node?

What I don't understand is the conversion to arrays of key and value in this case.

In my example above, the key is "responses" (i.e. string) and value is content which is an array. Now your code would convert the key to ["responses"] and content remains an array. Then the lengths are compared but since content is an array of length > 1, an error would be thrown.

But what I was trying to do (and what works with the memory storage of NR) is assign the responses key an array as value and not assign a collection of keys a single value each.

I understand that the underlying storage may or may not support arrays as values but that should not cause an error or if it does, then it should be something like "array values not supported by storage mechanism XYZ".

What am I missing here?

EDIT:

Having a looking at the existing redis store:

Redis.prototype.set = function (scope, key, value, callback) {
    if (callback && typeof callback !== 'function') {
        throw new Error('Callback must be a function');
    }
    try {
        if (!Array.isArray(key)) {
            key = [key];
            value = [value];
        } else if (!Array.isArray(value)) {
            // key is an array, but value is not - wrap it as an array
            value = [value];
        }

the difference seems to be that if the key isn't an array, then it becomes an array and the value is wrapped in an extra array.

So in my example it becomes ["responses"] and [[... content array ...]] - which do have the same length of one.

1 Like
3

Hey @gregorius , tnx for having a look. You're definitely right ... seems that atm there's a bug here. I'll make a change asap.

4

Above fixed in version 1.0.4.
Also fixed a redis context store bug where keys would not be displayed in context panel.

1 Like
5

Latest version is 1.0.5 where another bug was fixed in case retrieving key that does not exist.

6

I had another look and was going to ask about stringification of values but then realised you're using unstorage which does the stringification automagically.

Overall a good approach to delegate the heavy lifting to another library - you're basically just abstracting the context storage API of Node-RED to an "interface" (to use Java speak) that then is implemented by instanciable classes that represent various data storage engines which, in turn, delegate to unstorage!

Perhaps one thing I did wonder about is why the keyprefix comes before the scope:

scope = this.options.keyPrefix ? `${this.options.keyPrefix}${scope}` : scope;

And then another question, why #getKey as method name? Why is there a '#' (I've never seen that before in JS)?

And now another question :wink: - shouldn't you be extending the scope with keyPrefix the set method as well, i.e., somewhere here? I admit I haven't look into what the keyPrefix option does, so ignore if this is a stupid question.

As I see it, the keyPrefix option is only supported in delete and keys methods - is that meant to be?

And one more question, the options is set here but you make a reference, not duplication, of the options hash passed in. Isn't doing an delete on those options then deleting them from the options hash passed in? Meaning this would be a side-effect for anyone passing in options?

7

It's to denote that method is private. It should not matter in this case, could be public as well but I tend to expose only methods defined in a context interface.

It's a Redis specific thing - I should move the logic from base to redis class to be more clear. Thing is, unstorage provides it's own configuration to prefix redis keys caled base. But that can clash with redis configuration keyPrefix, thus I decided to disregard the base configuration and use keyPrefix instead. Unfortunately it seems this one is taken into an account when calling storage.get and set functions, but not when calling keys, thus I had to prepend that to the scope.

Why pefix can or should be used in redis is to not collide with other potential keys one might store. It should not be a problem with flow and node scope as those are represented with IDs, but a global scope can more easily clash with something already in redis. It's also beneficial to have a key prefix when manually querying keys in redis and searching for node red specific ones.

Unstorage has 2 sets of options, one for its interface and one for database driver. Former are predominant and only option for unstorage I care about is tableName (where applicable).
I didn't want to further break down context configuration into 2 sets, thus tableName is first copied to it's own prop then deleted from main options object, so I can pass whole options object to db driver.
I noticed options also include settings key by default (attached by NR), which I don't need or use, thus it's also deleted.

1 Like
8

I made it a bit more clear in the code, and locked the dependency versions to prevent unwanted bugs in case unstorage changes the behavior of their functions.

Latest release: 1.0.6

1 Like