Best practices to store data persistently?

Developing Nodes
1

Hi, I was wondering what are the best practices to allow a config node to store some data persistently so that it's preserved across restarts?

I thought I could use node.context() with the filesystem persistence but it doesn't look like I can force my specific node context to use it unless the user configures it.

2

Typically, you store values in a property of the defaults object and it is saved as part of the flow when deployed. But I assume you are talking about some dynamic runtime data?

3

Yes the data is generated after the node is deployed, it's like a unique id

4

Obviously you could write it to the file system but this will be problematic in docker and cloud environments.

If you can explain when/why this is value is generated/assigned & where/how it is used, I will try to think of a good solution.

5

I'm writing a Matter node and I have a configuration node that generates a virtual Matter device, when the device is generated I have several ids and a small json object I need to store in order to restore the device anytime Node-RED starts.

6

If this is done in the config node, can't you store it in the flow as an array or object property?

Again, if you can explain the finer detail like "when" and "how" it is used, we might be able to better advise.

7

@Steve-Mcl I would like the storage of this information to be transparent for the user, I don't want them to remember to store the details and restore them manually. Is this what you are suggesting?

I'm not sure what additional details I could provide to make my use case clearer, I'll try to describe the flow more in detail:

  1. Create a new config node
  2. Select an option from a dropdown (what kind of device to create)
  3. Deploy the node
  4. The node.js side starts the new device and generates a set of metadata that needs to be persistent
  5. Next time Node-RED restarts, the same metadata is read by the node and is provided to the logic that starts the device.
8

So, since the user is already in the config node edit panel, can't you simply create the device and store whatever is necessary in the config nodes defaults object. Then it will persist as part of the flow. The advantage is it will work on systems without a file system or persistent context.

Edit:
So in the config edit form, where the user generates a device, can't that be a device description that, when deployed creates the device?

I was asking for details so I can pinpoint where and when data needs persisting.

9

That's not correct Steve. A standard Node-RED docker install would have no problems with writing to the filing system. Since you have to add a /data/ folder which is writable from Node-RED otherwise NR won't work at all.

It is only a problem for services like FlowForge that don't provide a full filing system.

If you want the best of all worlds, you can put the choice of storage into the flow authors hands by allowing them to choose from the possible context stores.

Check out uibuilder's uib-cache node for an example of how to do this. The choice of store is set in the node's config in the Editor and the user is offered all of the available stores.

10

Ok, I'll be more specific. In a k8s type environment where containers are orchestrated without a filesystem. Or, if you don't have a mapped data directory in docker.

The point was: writing to filesystem can be problematic but if you can store data in the flow, then you don't need to work around it.

11

My problem is that I need to write this data both at startup and at runtime so I don't think this can work. What @TotallyInformation describes seems more viable, I'll look into it.

12

For JSON data, I've been having great results using the json-db-node-red JSON file store.

It also caches the values (so just editing the file doesn't work while node-red is running).

13

That would have the same issue as other FS based answers. It uses the FS to do the store and I'm not convinced it gives anything better than a persistent Node-RED context store?

As long as you are in an environment where a persistent store will work (either the FS based one or perhaps the REDIS store), then you should be good.

Though, as always with JavaScript, take a little care over your data objects since there are objects that cannot be serialised or when unserialised, they come out as a slightly different type (dates for example). Node-RED does take some steps to avoid the serialisation problem which most commonly hits when you try to store the res/req objects returned from ExpressJS - these have circular references and so cannot be serialised. Node-RED will also often take care of the date data type change as well (your input is a JavaScript date but your output will be an ISO8601 date string) but not always.

14

Why can't you store on the FS? The node-red uses the FS for config, etc.

I think there are interfaces to real DB's as well but those are too heavy for my needs.

15

For most people, you can. But as Steve mentioned, there are some Node-RED environments that don't have a full FS available. Won't matter to most people running their own or standard cloud environments. Will matter if you use FlowForge or some other environment that does horizontal scaling and so where work might get passed to a different instance.

16

Interesting I'll have to look into FlowForge. The single instance of node-red running in the cloud has raised concerns, but otherwise you hit the issues you are mentioning. thanks.

17

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.