Saving in global context parses key, breaking on "." shown in JSON

General
1

So I finally started to use the "context" feature. I've been using a JSON-DB successfully for years.

I used global.set(query, data) where data is JSON.

This works. But when the "query" has a "." inside, the key is broken up in the resulting JSON database (the filesystem global/global.json) will break up the query into sub elements (3 in this example) .

My query is:

scott.your@gmail.com_123_uncle

The resulting stored JSON (in global.json). Notice the storage has broken my query on the "." (as far as I know).

{
  "scott": {
    "your@gmail": {
      "com_123_uncle": {
            "category": "uncle",
            "uuid": "123",
            "name": "scott.your@gmail.com"
      }
    }
  }
}

Data retrieval works so with the same query, it will put it back together to get my data object.

Does anyone know more about this? It makes updating the JSON manually harder.

2

This is by design. The "query" as you call it is the context variable you wan to write to. However, Node-RED is "clever" and allows you to read/write a specific property deep inside an object variable.

Not quite sure what you were expecting really.

3

Node-RED is "clever" and allows you to read/write a specific property deep inside an object variable.

I don't see how this is "clever" - it took apart a STRING variable which isn't deep inside the object. It's at the root of my object. I wouldn't write to the your@gmail portion of my data storage.

I was expecting:

"scott.your@gmail.com_123_uncle":
      {
                "category": "uncle",
                "uuid": "123",
                "name": "scott.your@gmail.com"
       }

I use the email address as a key for my storage. The above example tacked on a couple more items.

The storage items should show up like I set it.

"scott@gmail.com": { },
"bob@gmail.com": {},
"another.mail@another.com:" {}

The power of JSON and stored on the file system, is it can be grabbed, shown in JSON tools, exported, imported, edited (with node-red stopped).

Note: that a JSON array would be as shown in my last example, and something I did first. That array isn't modified by node-red. But I was hoping the sqlite DB would be more efficient if I used the query approach versus the array approach (as it scales).

Are there other special characters that node-red was "clever" is breaking my query?

4

A JavaScript variable/constant name containing period's is not directly a valid name. What is "clever" is that the context set function recognises this and takes apart the name and creates any missing (sub)properties and puts the value in the result.

As I say, this is not a valid JavaScript identifier.

You cannot do const scott.your@gmail.com_123_uncle = 42. The context set function uses the same naming restriction. The context variable name has to be a valid JS var name.

your@gmail would not be valid either. However, it is possible to have that as a property name.

You should convert the periods and @ symbol to different, acceptable symbols such as - or _.

Otherwise, use a different storage library and not Node-RED's context. DuckDB for example.

5

Intersting. Thanks for the constraints that the query has to be valid javascript. That's not how JSON works, but if that's the design - nice to know.

thanks,

6

Indeed, but the 1st argument to that function is not JSON nor is it meant to be. The 2nd argument can, of course be JSON, or, more simply a JavaScript Object.

7

I just tried the "change" node setting `global.scott@gmail.com" -- and there are no error messages.

I also looked at the Context document and it doesn't mention this either. I realize that isn't valid javascript - but I would think it would be documented somewhere.

thanks

8

Nobody has ever raised this before so possibly not a common requirement. However, the devs are always looking for people to contribute - would be a good addition to the docs I think.

9

What exactly is it that you want to do that doesn't work? If you do global.set("scott.your@gmail.com_123_uncle", something) and later let myVar = gobal.get("scott.your@gmail.com_123_uncle") then you should get back what you set.

10

Thanks for the mention.

I had thought that setting a value was just a string and the object. Versus a "variable name" and an object, especially as the "context" was touted as the JSON Database replacement.

global.set("scott@gmail.com", jsonObject);

Thanks.

11

I'm sure you have considered this but wouldn't an array of objects like this be more approachable:

[
{
                "email": "scott.your@gmail.com_123_uncle",
                "category": "uncle",
                "uuid": "123",
                "name": "scott.your@gmail.com"
       },
      {...},
      ...
]

You can extract the required object by matching the email value, either javascript or jsonata.

12

I actually started my "context" use as an array, which worked great, and indexed by my email syntax. eg: global.get("myarray") which was keyed by the email.

But then I thought global.get and global.set of a full array would be very inefficient as the array grew. So I though the more global.set by key would utilize the sqlite query. That's where I'm at.

Now: if there is a smart array query/retreival I'm all for that. Do you know even if either would leverage the sqlite query? (It seems their breaking my query up wouldn't be able to query quickly, or even slower).

For many years I've been using the jsonDB I mentioned earlier - and it uses the msg.datapath which are my email syntax.

So I'm looking at the most efficient solution, and the arrays are clean, but seem inefficient.

13

What is this sqlite query that you speak of, is it the same thing as a jsonDB?

I suppose it depends how many elements there are, and also whether you will be deleting elements.
If there were a lot of "records", I would most likely put them in an external database, but then I do have one on my Node-red server.

14

When you do a context set all it does is save a reference to the object, it does not copy the object, so it makes no difference how large the object is.

15

That's good to know. But it still seems the access of this array (in javascript) would be less efficient than as a database query as the array gets into the 1000's or many more.

I'll might try some benchmarks.
thanks.

16

Well that depends on the context library that you are using.

Assuming you are using memory, then it is very efficient. I have a number of very large context variables that use the filing system store. Since these are loaded into memory when node-red starts, they are just as efficient. I can't vouch for the REDIS store. Those very large variables have no measurable impact when I manipulate them.

A context store backed by something like DuckDB would probably be best for something really complex. I did once start trying to create a store based on DuckDB but it didn't get very far and I ended up with other priorities.

So unless your arrays are truly massive and complex, the context store is actually very efficient indeed. So much so, that I have stopped using databases for most things.

17

Storing as an object can help with performance, especially when retrieving a value as you only need the key eliminating the need to traverse an array.

I concur with @TotallyInformation - the context stores are very capable, I have objects with more than 1 million elements sitting in flow/global context, no performance issues at all, extremely fast retrieval.

18

Are those in persistent storage using the file context? If so I would be a bit worried about writing that out every 30 seconds. I think it writes the whole flow or global context whenever anything changes, at the next 30 second boundary (configurable).

19

That's a good point. If there are many records then an object is clearly better.
And where the unique key is an email address I would probably use a "sanitised" string as the key (as suggested by Julian above)

{
   "scott-your_gmail-com": {
                "email": "scott.your@gmail.com_123_uncle",
                "category": "uncle",
                "uuid": "123",
                "name": "scott.your@gmail.com"
       },
      "someone-else_gmail-com": {...},
      ...
}

Since I run Node-red on Raspberries with just 512MB memory, I shy away from holding large amounts of data in context but no doubt it's perfectly fine provided you can smoothly handle a restart.

I wonder if anyone has done a comparison between memory / filesystem context and database storage as record numbers increase.

And if using context, my personal choice would be to use flow context in preference to global

20

My largest file-based store is around 200,000 and it has no problems with that though it doesn't really update any more. I have one around 75,000 that updates (extends) weekly, again no issues.

But yes, I wouldn't want to do this with something that is updated very rapidly. Especially as my write period is set to 10sec. I do have several though that update once a minute. They are around 800-1,500 entries each.