# JSON persistence with NodeRed- Redis or PostGreSQL or?

**URL:** <https://discourse.nodered.org/t/json-persistence-with-nodered-redis-or-postgresql-or/11511>\
**Category:** General\
**Created:** [24 May 2019 16:40 UTC](https://discourse.nodered.org/t/json-persistence-with-nodered-redis-or-postgresql-or/11511 "2019-05-24T16:40:49Z")\
**Posts on this page:** 17\
**Page:** 1

<div class="post-metadata">

**Author:** ![tony-shannon](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/tony-shannon/32/7378_2.png) [@tony-shannon](https://discourse.nodered.org/u/tony-shannon)\
**Post date:** [24 May 2019 16:40 UTC](https://discourse.nodered.org/t/json-persistence-with-nodered-redis-or-postgresql-or/11511/1 "2019-05-24T16:40:49Z")

</div>

Dear NodeRed community

Could I ask for views/experiences please?

As JSON is central to NodeRed, APIs etc, plus a pretty common way to store data now, could I ask for views on what database may be best to use for JSON persistence with NodeRed?

The 2 we are considering for a project are  
Redis- as I understand there may be nodes that help with Redis setup , plus it can be used for caching  
PostGreSQL- as this now has useful JSON type support

Any views/recommendations/advice would be most appreciated

thanks in advance  
Tony

(PS have done a search for related thread and didn't find anything)

---

<div class="post-metadata">

**Author:** ![TotallyInformation](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/totallyinformation/32/31_2.png) [@TotallyInformation](https://discourse.nodered.org/u/TotallyInformation)\
**Post date:** [24 May 2019 20:52 UTC](https://discourse.nodered.org/t/json-persistence-with-nodered-redis-or-postgresql-or/11511/2 "2019-05-24T20:52:44Z")

</div>

Well, you do know that Node-RED has persistent variables now? They will happily store JSON or anything that serialises to JSON.

That is great as long as you don't want loads of data, especially if it is updating rapidly. At which point, you may wish for a db. Also depends on how critical missing a few seconds of data would be on a crash.

You can also serialise to MQTT but again, this may have performance limitations for large, fast moving data.

So in short, depending on your use-cases, you might not even need a db.

On the other hand, if you are already using something that can store JSON and have sufficient device resources along with sufficient management resources, then potentially a db could give you a performance boost.

You could also look at something like CouchDB and its relatives which uses JSON natively.

Another thing to think about is resilience and horizontal scaling. If you need resilience, the tooling behind a dbms may well be very helpful. If you need horizontal scaling (e.g. replication between dbms instances) then again a dbms would be the way to go probably. It can give you enterprise level performance, security and resilience. But at the cost of resource and admin overheads.

---

<div class="post-metadata">

**Author:** ![tony-shannon](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/tony-shannon/32/7378_2.png) [@tony-shannon](https://discourse.nodered.org/u/tony-shannon)\
**Post date:** [28 May 2019 15:27 UTC](https://discourse.nodered.org/t/json-persistence-with-nodered-redis-or-postgresql-or/11511/3 "2019-05-28T15:27:29Z")

</div>

thanks Julian @TotallyInformation  
I didnt appreciate that and I assume that means its persisting to a local file?

We're certainly aiming at reasonable amount of data and keen on good response times so assumed we'd go down the DB route.  
I've looked at CouchDB but it doesnt look like an easy install/ with docker for example.

So again would welcome views on how easy to use Redis/PostGreSQL for this with NodeRed, as they are our 2 preferred DBs for this initial exploration.

Thanks  
Tony

---

<div class="post-metadata">

**Author:** ![TotallyInformation](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/totallyinformation/32/31_2.png) [@TotallyInformation](https://discourse.nodered.org/u/TotallyInformation)\
**Post date:** [28 May 2019 16:37 UTC](https://discourse.nodered.org/t/json-persistence-with-nodered-redis-or-postgresql-or/11511/4 "2019-05-28T16:37:28Z")

</div>

> [@tony-shannon](#):
>
> I assume that means its persisting to a local file

Yes, for now at least. However, it is designed to have a pluggable interface so others could write an interface - to REDIS for example?

> [@tony-shannon](#):
>
> looked at CouchDB but it doesnt look like an easy install/ with docker

Sorry, not tried that. I'm not really familiar with REDIS or Postgres.

---

<div class="post-metadata">

**Author:** ![tony-shannon](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/tony-shannon/32/7378_2.png) [@tony-shannon](https://discourse.nodered.org/u/tony-shannon)\
**Post date:** [4 June 2019 14:21 UTC](https://discourse.nodered.org/t/json-persistence-with-nodered-redis-or-postgresql-or/11511/5 "2019-06-04T14:21:14Z")

</div>

> [@TotallyInformation](#):
>
> Yes, for now at least. However, it is designed to have a pluggable interface so others could write an interface - to REDIS for example?

thanks again Julian, am aware of other plugins for Redis, is that the same kind of thing?

> **[node-red-contrib-redis](https://flows.nodered.org/node/node-red-contrib-redis)**
>
> Node Red client for Redis with pub/sub, list, lua scripting and other commands support.

---

<div class="post-metadata">

**Author:** ![tony-shannon](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/tony-shannon/32/7378_2.png) [@tony-shannon](https://discourse.nodered.org/u/tony-shannon)\
**Post date:** [4 June 2019 14:26 UTC](https://discourse.nodered.org/t/json-persistence-with-nodered-redis-or-postgresql-or/11511/6 "2019-06-04T14:26:07Z")

</div>

We now have a baseline and very basic Node/Express/Redis tool that does some of what we need it to do.. APIs/JSON persistence & transformation

> **[RippleOSI/Node-JSON-Redis-exp1](https://github.com/RippleOSI/Node-JSON-Redis-exp1)**
>
> experiment with Node, JSON and Redis . Contribute to RippleOSI/Node-JSON-Redis-exp1 development by creating an account on GitHub.

We're a healthcare non profit foundation looking for someone from this community with good level of NodeRed experience to help us migrate this kind of tool into a NodeRed environment to make it easier for others to reuse.

Anyone who is looking for a bit of **funded work** please get in touch directly via Message facility in this forum

thanks  
Tony

---

<div class="post-metadata">

**Author:** ![TotallyInformation](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/totallyinformation/32/31_2.png) [@TotallyInformation](https://discourse.nodered.org/u/TotallyInformation)\
**Post date:** [4 June 2019 18:41 UTC](https://discourse.nodered.org/t/json-persistence-with-nodered-redis-or-postgresql-or/11511/7 "2019-06-04T18:41:30Z")

</div>

> [@tony-shannon](#):
>
> thanks again Julian, am aware of other plugins for Redis, is that the same kind of thing?

No problem. I was actually more thinking about a persistent variable handler that used REDIS to complement the two that ship with Node-RED (`memory` and `file`). However, the node would work too and doubtless be a bit simpler.

---

<div class="post-metadata">

**Author:** ![knolleary](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/knolleary/32/3_2.png) [@knolleary](https://discourse.nodered.org/u/knolleary)\
**Post date:** [4 June 2019 18:50 UTC](https://discourse.nodered.org/t/json-persistence-with-nodered-redis-or-postgresql-or/11511/8 "2019-06-04T18:50:13Z")

</div>

Work is happening to produce a Redis storage plugin for Context - [https://github.com/node-red/node-red-context-redis/pull/1](https://github.com/node-red/node-red-context-redis/pull/1) - it's just stuck in the queue of things I have to do for 1.0.

---

<div class="post-metadata">

**Author:** ![tony-shannon](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/tony-shannon/32/7378_2.png) [@tony-shannon](https://discourse.nodered.org/u/tony-shannon)\
**Post date:** [4 June 2019 20:25 UTC](https://discourse.nodered.org/t/json-persistence-with-nodered-redis-or-postgresql-or/11511/9 "2019-06-04T20:25:09Z")

</div>

thanks @knolleary  
Could you explain the key difference between the existing **contrib** node/client for Redis and this **context** node for Redis in development?

PS if you think the context node fits our needs please let us know how we can help with that "to do"

thanks in advance  
Tony

---

<div class="post-metadata">

**Author:** ![knolleary](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/knolleary/32/3_2.png) [@knolleary](https://discourse.nodered.org/u/knolleary)\
**Post date:** [5 June 2019 07:32 UTC](https://discourse.nodered.org/t/json-persistence-with-nodered-redis-or-postgresql-or/11511/10 "2019-06-05T07:32:54Z")

</div>

Hi @tony-shannon

The contrib-redis package provides nodes you can use in your flow. So whoever is creating the flow has to think about how/where they need to get/set data in redis.

The context package does not provide nodes that appear in the palette. It provides a way for flow and global context ([https://nodered.org/docs/user-guide/context](https://nodered.org/docs/user-guide/context)) to be stored in redis. By default, context is only stored in memory, but node-red does provide a file-system plugin to save context data on disk.

---

<div class="post-metadata">

**Author:** ![TotallyInformation](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/totallyinformation/32/31_2.png) [@TotallyInformation](https://discourse.nodered.org/u/TotallyInformation)\
**Post date:** [5 June 2019 10:15 UTC](https://discourse.nodered.org/t/json-persistence-with-nodered-redis-or-postgresql-or/11511/11 "2019-06-05T10:15:01Z")

</div>

Which is why I was recommending that approach as it keeps your flows simple. Both work though of course.

---

<div class="post-metadata">

**Author:** ![tony-shannon](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/tony-shannon/32/7378_2.png) [@tony-shannon](https://discourse.nodered.org/u/tony-shannon)\
**Post date:** [5 June 2019 12:48 UTC](https://discourse.nodered.org/t/json-persistence-with-nodered-redis-or-postgresql-or/11511/12 "2019-06-05T12:48:26Z")

</div>

thanks both @knolleary and @TotallyInformation  
If we want to use this latest **context** package can we help you by testing it or how else can we try to help?  
It appears to offer the JSON persistence we need... sounds like it should offer advantages and flexibility.  
I'll be posting request for help with this as some **funded work** in the forum to get some more attention.  
T

---

<div class="post-metadata">

**Author:** ![knolleary](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/knolleary/32/3_2.png) [@knolleary](https://discourse.nodered.org/u/knolleary)\
**Post date:** [5 June 2019 13:06 UTC](https://discourse.nodered.org/t/json-persistence-with-nodered-redis-or-postgresql-or/11511/13 "2019-06-05T13:06:04Z")

</div>

HI @tony-shannon

the redis plugin has been developed and tested by our friends at Hitachi. In theory it is ready to publish.

The outstanding action is with me to do a proper code review and ensure we have all the bits in place first.

It has been outstanding for a while - I will try to get to it this week.

---

<div class="post-metadata">

**Author:** ![tony-shannon](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/tony-shannon/32/7378_2.png) [@tony-shannon](https://discourse.nodered.org/u/tony-shannon)\
**Post date:** [5 June 2019 14:46 UTC](https://discourse.nodered.org/t/json-persistence-with-nodered-redis-or-postgresql-or/11511/14 "2019-06-05T14:46:03Z")

</div>

Many thanks @knolleary  
Please don't feel rushed on our behalf.. we can probably start exploring that fork in the meantime..  
thanks again  
Tony

---

<div class="post-metadata">

**Author:** ![dustinw](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/dustinw/32/3426_2.png) [@dustinw](https://discourse.nodered.org/u/dustinw)\
**Post date:** [6 June 2019 04:29 UTC](https://discourse.nodered.org/t/json-persistence-with-nodered-redis-or-postgresql-or/11511/15 "2019-06-06T04:29:35Z")

</div>

@tony-shannon one thing of note ... I wouldn't recommend using context to store records. If you have variables like totals, sums, last run time, that change rapidly then context is the place to store them, but if you're storing records I wouldn't store them in context (with or without the Redis context plugin). You should use the nodes for your storage system for storing and updating those nodes.

If you do need store records:

- Redis is a fast and easy, and it would let you use one system for both record storage and context storage (and queue). Its downside is that it doesn't have some features like secondary indexes.
- Postgres is an RDBMs with with a rich JSON layer on top ... you can store json but treat values with-inin the JSON as fields for querying ... it also allows for setting up relations between records. The downside, with all those features it's not as fast as Redis or as easy to setup and manage ... Also the Node-Red module for Postgres isn't as robust as the ones for Redis.

---

<div class="post-metadata">

**Author:** ![tony-shannon](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/tony-shannon/32/7378_2.png) [@tony-shannon](https://discourse.nodered.org/u/tony-shannon)\
**Post date:** [6 June 2019 08:30 UTC](https://discourse.nodered.org/t/json-persistence-with-nodered-redis-or-postgresql-or/11511/16 "2019-06-06T08:30:06Z")

</div>

many thanks @dustinw  
Appreciate that sound advice, makes sense.  
That helps my understanding of context, seems akin to data that may be cached for shortterm/fast access between nodes, whereas longer term storage/persistence should handled differently.  
As you point out Redis is helpful for both, though aware it has its own limitations.  
thanks again  
Tony

---

<div class="post-metadata">

**Author:** ![thechane](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/thechane/32/7701_2.png) [@thechane](https://discourse.nodered.org/u/thechane)\
**Post date:** [19 August 2019 04:50 UTC](https://discourse.nodered.org/t/json-persistence-with-nodered-redis-or-postgresql-or/11511/17 "2019-08-19T04:50:02Z")

</div>

I’ve been working on this problem from another angle. You might be interested in looking at this [consul context plugin](https://www.npmjs.com/package/node-red-contrib-context-consul)  
—Stu