Json-db-node-red keeping object that wasn't stored persistently

I've hit a strange issue with json-db-node-red. It seems the lazy caching is keeping my object that I've modified and returning it, instead of going to the DB (file) to get a clean copy.

From their info:

A Lazy-caching is implemented to only load the collection when needed.

Basically a JSON object is successfully returned from a DataOut in all it's values. I include a password in that data. But I check the password against one provided by the user (on a web interface, eg. webAddress/:NAME/:PASSWORD )

If they don't match, the "status" is set to "Invalid Password", and the full JSON is returned to the caller (so they can check status).

But since this is a full object, the password is valid (it just doesn't match). So that information is returned to the user. Obviously not good.

So I set "password" to "undefined" or something.

HERE is the issue: The next DataOut retrieval of the same object finds the password = undermined. And now my passwords definitely don't match.

NOTE: the text file storing the data has NOT been changed.

Does anyone know how to stop this in-memory caching?

json-db-node-red v0.5.3
node-red v2.0.5

thanks,

Does that node offer you anything you need over using persistent global storage as provided by node-red?

Not a core node so please raise an issue on the node's GitHub site.

What do you mean "not a core node"?

Isn't this a discussion about using node-red?

I'm not sure what global storage you mean, but the json DB stores in a file in json format and protects its integrity. Then you query with a data path to access and update fields. It's powerful.

My issue is after retrieval, I modify a field but don't persist (dataIn). The next retrieval finds my modified version in-memory, versus grabbing from the persistent file - so the values are wrong.

I'll look at creating a copy and modifying the copy instead.

I can also provide a sample flow.
Thanks

I believe Julian was simply stating, if you have a specific question about a node which isn't a core node-red node, you might have more luck hitting up the developer on its repository.

And - you did post this in the "Creating Nodes" section :man_facepalming:

Of course there is no reason not to ask on the forum - especially if it is a popular node - but searching the forum for "json-db-node-red" reveals it is not I'm afraid.

It may be a reference thing - if the json-db caches the data in memory.

Try using const copy = RED.util.cloneMessage(data); before you save or before you modify.

It is written by a contributor and isn't part of Node-RED directly, it is an external, optional extension.

You need to speak to the author about getting it fixed if it needs to, we cannot directly help, only provide some guidance.

Yes, and I've provided guidance as a node-red fan.

By the way, that node hasn't been updated in 4 years which is quite a long time. Node-RED itself has moved on a long way in that time.

It is one of the most used features of node-red. Working with context : Node-RED

I suggest you watch this playlist: Node-RED Essentials. The videos are done by the developers of node-red. They're nice & short and to the point. You may find about other useful features that you are not currently aware of.

The node you refer to has not been updated for over four years and it's github page has not got an Issues section where you can post questions. The author has not made updates to any of his repositories for several years so I rather suspect the node is not supported.

I'll try that.
What nodes do others use for file persistence (vs sqlite)?

Many use file backed context (as Colin states)

Others use databases or even contrib nodes e.g. node-red-contrib-key-value-store (node) - Node-RED

Or even write to file / restore from file using JSON and FILE nodes.

Ultimately depends on the overall solution

You can use the File nodes for reading and writing files, but for saving run time data that has to be persistent over node-red restart then persistent global context is the most used I think.
Also you can use Retained MQTT Topics to achieve the same thing.

I'll look at context. The json-DB node was perfect for my needs and has worked nicely for a couple years. I'll do the const copy which should fix my issue.

I've used node-red for 7+ years and it's amazing. Some are older versions, but they run 24/7.

Thanks

1 Like

So I hope it is ok - if I use this thread.
I installed the nodes - but I am not satisfied, as the nodes do not implement the delete function which is offered by the node-json-db library.

So I tried to use this library within a function node instead of using the nodes, but I fail with some basics.

So can someone explain why I cannot use this library in a function node?

I already checked - and the library has been correctly installed by node-red:

image

According this description - I should be able to create a db object:

import { JsonDB } from 'node-json-db';
import { Config } from 'node-json-db/dist/lib/JsonDBConfig'

// The first argument is the database filename. If no extension, '.json' is assumed and automatically added.
// The second argument is used to tell the DB to save after each push
// If you put false, you'll have to call the save() method.
// The third argument is to ask JsonDB to save the database in an human readable format. (default false)
// The last argument is the separator. By default it's slash (/)
var db = new JsonDB(new Config("myDataBase", true, false, '/'));

So I did the following:

image

However when I try to create the db object - I get the error - that I cannot use the constructors.

So can please someone explain what I did wrong?

Steve, the cloneMessage() definitely worked. Thanks.

I'll post an issue to their GitHub too.
-scott

Which nodes? You don't need any nodes for persistent context.

I already deinstalled the node-red-contrib-json-db nodes as they have not implement the delete funktionality. So the nodes are not my problem they worked as they should do.

So I want to use a function node with the library, but I am not able to use the constructors.

I'd have to check, but I'm not sure we support loading internals from a module via node-json-db/dist/lib/JsonDBConfig

Does the top level module not expose that in some way you can access?

@knolleary I don't understand your statement of loading internals.

As for persistent storage of JSON data, the only node I see is

json-db-node-red

Am I missing some cool functionality?

@knowledgeshark my response was to @mickym2 and how he was trying to use the underlying npm module directly in a function node.