JSON persistence with NodeRed- Redis or PostGreSQL or?

General
1

Dear NodeRed community

Could I ask for views/experiences please?

As JSON is central to NodeRed, APIs etc, plus a pretty common way to store data now, could I ask for views on what database may be best to use for JSON persistence with NodeRed?

The 2 we are considering for a project are
Redis- as I understand there may be nodes that help with Redis setup , plus it can be used for caching
PostGreSQL- as this now has useful JSON type support

Any views/recommendations/advice would be most appreciated

thanks in advance
Tony

(PS have done a search for related thread and didn't find anything)

2

Well, you do know that Node-RED has persistent variables now? They will happily store JSON or anything that serialises to JSON.

That is great as long as you don't want loads of data, especially if it is updating rapidly. At which point, you may wish for a db. Also depends on how critical missing a few seconds of data would be on a crash.

You can also serialise to MQTT but again, this may have performance limitations for large, fast moving data.

So in short, depending on your use-cases, you might not even need a db.

On the other hand, if you are already using something that can store JSON and have sufficient device resources along with sufficient management resources, then potentially a db could give you a performance boost.

You could also look at something like CouchDB and its relatives which uses JSON natively.

Another thing to think about is resilience and horizontal scaling. If you need resilience, the tooling behind a dbms may well be very helpful. If you need horizontal scaling (e.g. replication between dbms instances) then again a dbms would be the way to go probably. It can give you enterprise level performance, security and resilience. But at the cost of resource and admin overheads.

3

thanks Julian @TotallyInformation
I didnt appreciate that and I assume that means its persisting to a local file?

We're certainly aiming at reasonable amount of data and keen on good response times so assumed we'd go down the DB route.
I've looked at CouchDB but it doesnt look like an easy install/ with docker for example.

So again would welcome views on how easy to use Redis/PostGreSQL for this with NodeRed, as they are our 2 preferred DBs for this initial exploration.

Thanks
Tony

4

Yes, for now at least. However, it is designed to have a pluggable interface so others could write an interface - to REDIS for example?

Sorry, not tried that. I'm not really familiar with REDIS or Postgres.

5

thanks again Julian, am aware of other plugins for Redis, is that the same kind of thing?

6

We now have a baseline and very basic Node/Express/Redis tool that does some of what we need it to do.. APIs/JSON persistence & transformation

We're a healthcare non profit foundation looking for someone from this community with good level of NodeRed experience to help us migrate this kind of tool into a NodeRed environment to make it easier for others to reuse.

Anyone who is looking for a bit of funded work please get in touch directly via Message facility in this forum

thanks
Tony

1 Like
7

No problem. I was actually more thinking about a persistent variable handler that used REDIS to complement the two that ship with Node-RED (memory and file). However, the node would work too and doubtless be a bit simpler.

8

Work is happening to produce a Redis storage plugin for Context - https://github.com/node-red/node-red-context-redis/pull/1 - it's just stuck in the queue of things I have to do for 1.0.

2 Likes
9

thanks @knolleary
Could you explain the key difference between the existing contrib node/client for Redis and this context node for Redis in development?

PS if you think the context node fits our needs please let us know how we can help with that "to do"

thanks in advance
Tony

1 Like
10

Hi @tony-shannon

The contrib-redis package provides nodes you can use in your flow. So whoever is creating the flow has to think about how/where they need to get/set data in redis.

The context package does not provide nodes that appear in the palette. It provides a way for flow and global context (https://nodered.org/docs/user-guide/context) to be stored in redis. By default, context is only stored in memory, but node-red does provide a file-system plugin to save context data on disk.

11

Which is why I was recommending that approach as it keeps your flows simple. Both work though of course.

12

thanks both @knolleary and @TotallyInformation
If we want to use this latest context package can we help you by testing it or how else can we try to help?
It appears to offer the JSON persistence we need... sounds like it should offer advantages and flexibility.
I'll be posting request for help with this as some funded work in the forum to get some more attention.
T

13

HI @tony-shannon

the redis plugin has been developed and tested by our friends at Hitachi. In theory it is ready to publish.

The outstanding action is with me to do a proper code review and ensure we have all the bits in place first.

It has been outstanding for a while - I will try to get to it this week.

1 Like
14

Many thanks @knolleary
Please don't feel rushed on our behalf.. we can probably start exploring that fork in the meantime..
thanks again
Tony

15

@tony-shannon one thing of note ... I wouldn't recommend using context to store records. If you have variables like totals, sums, last run time, that change rapidly then context is the place to store them, but if you're storing records I wouldn't store them in context (with or without the Redis context plugin). You should use the nodes for your storage system for storing and updating those nodes.

If you do need store records:

  • Redis is a fast and easy, and it would let you use one system for both record storage and context storage (and queue). Its downside is that it doesn't have some features like secondary indexes.
  • Postgres is an RDBMs with with a rich JSON layer on top ... you can store json but treat values with-inin the JSON as fields for querying ... it also allows for setting up relations between records. The downside, with all those features it's not as fast as Redis or as easy to setup and manage ... Also the Node-Red module for Postgres isn't as robust as the ones for Redis.
16

many thanks @dustinw
Appreciate that sound advice, makes sense.
That helps my understanding of context, seems akin to data that may be cached for shortterm/fast access between nodes, whereas longer term storage/persistence should handled differently.
As you point out Redis is helpful for both, though aware it has its own limitations.
thanks again
Tony

17

I’ve been working on this problem from another angle. You might be interested in looking at this consul context plugin
—Stu