Saving in global context parses key, breaking on "." shown in JSON

[TotallyInformation]

@jbudd

And where the unique key is an email address I would probably use a "sanitised" string as the key (as suggested by Julian above)

Just note that I don't believe that the deep key approach is any more (or less) efficient than getting/setting the whole object since it has to be read into memory anyway. Certainly I never bother and I've never noticed any impact. I believe this has been discussed before on the forum.

@jbudd

I wonder if anyone has done a comparison between memory / filesystem context and database storage as record numbers increase.

I've not seen one.

Some DB's are better suited to rapid updates than others. But unless you are using a DB engine backed by a service (e.g. Postgres or MariaDB), they will have very similar issues to Node-RED's file-backed context stores. Using a DB is more about how you want to access/process the data.

Any data that fits into memory is almost certain to be faster to read/write in a context variable than a DB I believe.

@jbudd

And if using context, my personal choice would be to use flow context in preference to global

Yes, as with any data, keeping its access to a minimum is always sensible.

[knowledgeshark]

@Colin

tes the whole flow or global context whenever anything changes, at the next 30 second boundary (configu

Thanks for discussion.

It seems the issue is writing out to disk and how much is written? If I get a large array, and modify it - I would think the entire array is written to disk.

But if I modify or add a new global variable (eg. global.set(unique, value)) - will this be the only thing written out (merged with the existing global.json)?

Also, with the global variables, as it scales to thousands, is this using the sqlite DB to get the elements?

Lastly, for a sanitized key, is there a list of the special characters or do I just look at javascript BNF?

Thanks

[dceejay]

@knowledgeshark

I would think the entire array is written to disk.

In the simplest case of filesystem context store then yes, but it could also depend on other operating system factors - eg using an overlay file system, (such as in docker), can mean that only changes are written to disk, so again that can improve efficiency.

[knowledgeshark]

Without that docker overlay issue, just to clarify:

This saves to the filesystem (assuming on), index by "value1".

global.get("value1");
global.set("value1",<stuff>);

Then I add another value. (Assuming value1 already saved)

global.get("value2");
global.set("value2",<stuff>);

Will only the "value2" get written to disk?

If so, then the large array would be efficient assuming all unique (and optionally sanitized keys).

thanks.

[TotallyInformation]

@knowledgeshark

Will only the "value2" get written to disk?

Actually, this is worse than I'd realised. If you are using the file store, ALL GLOBALs seem to be written to a single file!

~/.node-red/context/global/global.json

This is not at all how I thought it would work. Not sure how I missed this before, too many assumptions I guess.

[bakman2]

the flow/global variables are evaluated as javascript objects if i am not mistaken, not as JSON, thus they will break on the dot notation.

Use a simple approach:

const obj = {
users: [
 {
  email: "some.email@gmail.com", category: "uncle", "uuid":123
 }
]
}

global.set("somevar", obj)

to return results:

const g = global.get("somevar").users

// return all uncles
const uncles = g.filter(x => x.category == 'uncle')

// return single object
const someemail = g.find(x => x.email == 'some.email@gmail.com')

Not tested, but that is the idea

[knowledgeshark]

That javascript object notation is a strange constraint for a key. I assume there are implementation simplicities, but we don't access that object by name outside of the "global.get" as far as I know, so why not keep it as JSON doesn't mind.

As the storage is JSON, I think it should mirror what the user is setting (without breaking apart my query item). I grab these stored JSON's and show in a pretty JSON tools, manipulate them, prototype them, and sometimes update the stored JSON (after node-red stopped/flushed). Thus this key breakage isn't a feature I like.

So I'm back to options (key == query).

1. change my key name (replace . and @) - ever time I reference the object
2. use a JSON array with my own unchanged keys - but maybe take the HIT of file storage of the entire array
3. use unchanged key names and let the context break them apart
4. use a JSON_db - like I've been doing for 5 or more years.

But it would be nice to get clarification on when the filesystem is updated and at what granularity. Obviously if the JSON has to be stored on disk, and it's a single file, the entire file must be written (even the JSON_db does that).

[bakman2]

I think there is a misunderstanding here. Json is a string, a javascript object contains variables and functions. Ie you can store/use functions from a global /flow variable

The flow variables are written to disk every 30s.

[knowledgeshark]

Json is a string, a javascript object contains variables and functions. Ie you can store/use functions from a global /flow variable

Can you show an example? The JSON in the context store is just a string (and what we have been discussing). Are there use cases where "functions" are stored in that same context (like a function pointer?) - or I miss what you are saying for a "JavaScript object".

[TotallyInformation]

@knowledgeshark

As the storage is JSON, I think it should mirror what the user is setting (without breaking apart my query item).

We seem to be hitting a repeating understanding issue here. The left-hand argument of a context set function is a JavaScript identifier, not JSON. The data on the right-hand side is stored in memory as a JavaScript variable. But, on writing to file, it is saved as a JSON string.

For this reason, if you are using an in-memory context var, you can put functions in it. But if you write it to a file-based store and restart Node-RED, the function will have gone.

[{"id":"0fcbe41248c8a4b1","type":"inject","z":"b2f18a716bd20f99","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"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","payloadType":"str","x":290,"y":1720,"wires":[["6b51232d72a6fbc7"]]},{"id":"6b51232d72a6fbc7","type":"function","z":"b2f18a716bd20f99","name":"function 27","func":"function fnTest() {\n    return 42\n}\nglobal.set('TEST', {\n    ans: 42,\n    fnTest: fnTest,\n})\n\nglobal.set('TEST2', {\n    ans: 42,\n    fnTest: fnTest,\n}, 'file')","outputs":1,"timeout":0,"noerr":0,"initialize":"","finalize":"","libs":[],"x":430,"y":1720,"wires":[[]]},{"id":"9132acdb9e6301f4","type":"inject","z":"b2f18a716bd20f99","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"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","payloadType":"str","x":290,"y":1780,"wires":[["8fbaca02b552f8bc"]]},{"id":"8fbaca02b552f8bc","type":"function","z":"b2f18a716bd20f99","name":"function 28","func":"const TEST = global.get('TEST')\nconst TEST2 = global.get('TEST2', 'file')\n\nreturn {\n    test1: {\n        fnRes: TEST.fnTest(),\n        payload: TEST,\n    },\n    test2: {\n        fnRes: TEST2.fnTest(),\n        payload: TEST2,\n    },\n}\n","outputs":1,"timeout":0,"noerr":0,"initialize":"","finalize":"","libs":[],"x":430,"y":1780,"wires":[["a87ddcc2d1c78d1c"]]},{"id":"a87ddcc2d1c78d1c","type":"debug","z":"b2f18a716bd20f99","name":"debug 23","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":620,"y":1780,"wires":[]}]

Similarly if you use the REDIS-backed store.

That's because you cannot write a function to JSON. If you check the global.json file, you can confirm this.

So to repeat, the 1st argument to a context set is a JavaScript identifier, it should be a valid JS variable name. However, there is some extra processing that lets you read from or write to a deep property. This is not standard javascript or json. It is a "fix" that is node-red specific.

If you want to use context for storing an Object, pick a valid name and keep everything in the object itself.

For simplicity I strongly recommend only using simple variable names for context set/get - always get/set the whole variable - you loose nothing and gain nothing by trying to use the "clever" deep references in the name. Do that in standard JavaScript - if you are using a function node. In any case, always use JavaScript/JSON valid variable and property names to save yourself a lot of head scratching.

[knowledgeshark]

Ok, I'm sorry to beat this. I'm a stickler for "language" issues, and I don't get it.

So to repeat, the 1st argument to a context set is a JavaScript identifier, it should be a valid JS variable name. However, there is some extra processing that lets you read from or write to a deep property. This is not standard javascript or json. It is a "fix" that is node-red specific.

Any function can take a variable name, so the following is the same as passing a constant "scott".
It's using the value of the variable - at runtime - as the key to this global.set.

var _scottsYourUncle = "scott";
global.set(_scottsYourUncle, mergedJSON);

But since it's a variable, versus a string (like "scott") as the KEY to the set/get, then you can now get RUNTIME errors. The following is after setting my variable to a JSON string (actually that mergedJSON variable.

var _scottsYourUncle = mergedJSON;
global.set(_scottsYourUncle, mergedJSON);

RUNTIME error:

22 Apr 17:41:11 - [error] [function:Merge global.qrJSON] Error: Invalid context key

I had never heard of accessing a "deep property" except in normal JSON processing.

So maybe the node-red specific "fix" should be re-examined, or documented:-)

ps.

mergedJSON = {"category":"cat","uuid":"UUID","name":"Name","mapping":"https://SemanticMarker.org","date":"2026-04-23T00:54:32.384Z","ownerIP":"127.0.0.1","imageURL":"https://KnowledgeShark.me/images/balloonLabyrinth.jpg","qrcode":"http://localhost:1880/qr/cat/Name/UUID","count":"1","access":[{"date":"2026-04-23T00:54:32.384Z","ip":"127.0.0.1"}]}

[TotallyInformation]

@knowledgeshark

Any function can take a variable name, so the following is the same as passing a constant "scott".
It's using the value of the variable - at runtime - as the key to this global.set.

OK, crossover terminology is confusing.

global.set('fred', someObject)

Here fred is the name of a context variable. It is passed as a string because the name of the variable is not, itself, variable.

If I have a context variable fred that contains this:

{
   jim: 42,
   dora: 24,
}

I can do this global.get('fred.dora') which will give the result 24. The .dorapart is a deep reference to thefred` variable. But that deep reference is a cludge written into Node-RED, it is not normal JavaScript. What's more, it has no real advantage at all in a function node since the whole context store is already in memory. At best it might save you a line of code but, as we are seeing, it can be confusing.

const fred = global.get('fred')
node.warn(fred.dora)

Is identical in terms of memory and processing as

const dora = global.get('fred.dora')
node.warn(dora)

@knowledgeshark

RUNTIME error:

Yes, because you have tried to pass the object value in the JavaScript variable mergedJSON as a context variable name. Since mergedJSON contains a string that is NOT A VALID JavaScript entity name, you get an error.

If mergedJSON, for example, has the value {"fred": 42} or fred@example.com - this CANNOT be a valid context variable name and you get the error. If it were to contain the value hello_there-stranger-things, that IS a valid context variable name and it will work.

KISS - Keep it simply simple!

@knowledgeshark

the node-red specific "fix" should be re-examined,

It is what it is and lots of things would break if it were changed now. The same process is doubtless used in change nodes as well where it makes more sense.

@knowledgeshark

or documented:-)

As already said, the devs are always looking for people to create improved code and documentation.

@knowledgeshark

mergedJSON

Try this:

const mergedJSON = {"category":"cat","uuid":"UUID","name":"Name","mapping":"https://SemanticMarker.org","date":"2026-04-23T00:54:32.384Z","ownerIP":"127.0.0.1","imageURL":"https://KnowledgeShark.me/images/balloonLabyrinth.jpg","qrcode":"http://localhost:1880/qr/cat/Name/UUID","count":"1","access":[{"date":"2026-04-23T00:54:32.384Z","ip":"127.0.0.1"}]}
const name = JSON.stringify(mergedJSON)
const myObj = {}
myObj[name] = 42

Pretty obviously, this would not work. Similarly:

global.set(name, 42)

will also not work - for the same reasons.

[knowledgeshark]

I can do this global.get('fred.dora') which will give the result 24. The .dorapart is a deep reference to the fred` variable. But that deep reference is a cludge written into Node-RED, it is not normal JavaScript.

Ok. If the "get" KEY is a deep query into the JSON that's new documented information on the context capabilities.

I assumed the "get" returns the JSON .. and you do that deep stuff in the JSON.
I assume the "set" behaves the same way ('fred.dora',42).

It is what it is and lots of things would break if it were changed now.

I'm sure..

Thanks for the info.

ps. that dot notation to access the JSON looks valuable (as I was just playing with it). Now we're onto something. thanks to your team..

Also: the only documented reference to the "." notation is this, and it's talking about the hierarchy of the context (versus something deep in the object).

var colour = flow.get("$parent.colour");

[Colin]

@knowledgeshark

I assumed the "get" returns the JSON

You are misusing the term JSON here. The get will return the javascript object, not JSON. JSON is always a string.

[TotallyInformation]

To round that info off. If your variable is in the file store, Node-RED startup reads the JSON and loads it into memory as JavaScript objects which is what you get. Similarly, a set, writes the JS Object into the memory store and a few seconds later, Node-RED converts it to JSON and re-writes the store file.

I do still think that it would be good if someone were to write a DB-backed store, DuckDB would, I think be a good DB to use. That could potentially allow even larger data to be handled and could open up additional processing and event notification potential. But sadly, I just don't have the time. Retirement seems to be keeping me at least as busy as when I was working full-time!

[christopher-lambe]

Custom context storage plugins

As per debate started here: Durable and reliable contextStorage - DB/mysql/mariadb based there are now several plugins available to store Node RED context in either Redis, SQLite, PostgreSQL or MySQL/MariaDB. More here: GitHub - sebenik/node-red-context

[knowledgeshark]

I started this thread looking at context and learned a lot.
As for a JSON DB, as I mentioned I still use this version:

I've been using a JSON-DB successfully for years.

Thanks.