How to limit persistent context data created by the state-flow node

General
1

Thank you for this guide! My question: Is there a method to clear out records older than X, or when a file size is reached, or some other manner to help ensure that the stored file size does not grow too large? In my use of this, I am only wishing to store 48 hours of persistent data.

Any assistance greatly appreciated!

2

What sort of data are you storing there?

3

You have to do that yourself. How, depends on the structure of the data and how you are putting it into the variable.

4

Great information (although still a bit over my head). My simple use of persistable storage is related to a single UI node (for now at least) of the ui_statetrail node with the persistable storage checkbox chosen.

persist
persist488×680 13.4 KB

I am not aware of any additional settings within that contrib node but the data is currently being written to a json file in the node red context folder

store

Within that file each entry does have a timestamp

data

With this description would you be able to help me get started to develop the function node within node-red to parse/delete records older than timestamp < Now() - 48 hours?

5

I haven't used that node, but I would have assumed that with the period set to 24 hours as you have it that it would only save 24 hours worth of data. Is that not what you are finding?

6

I'm on the phone right now but let's see if I can explain.
In the function node:
The data is an array where every element has timestamp.
You'll need to read that array from context.

var Yourarray=context.get("Yourarray") || []

Then do your stuff with that array
And may be add new entry to array.
Before storing the array back to context filter out all elements where timestamp is older than 48 hours. Note that all calculations are done in milliseconds.

var time = Yourarray[Yourarray.length - 1].timestamp - 172800000
Yourarray = Yourarray.filter(el => el.timestamp > time);

And now the array is ready to be stored back into context.

context.set("Yourarray",Yourarray)

7

This whole write up was dealing with the persistence storage used by the base node-red nodes. How 'contrib' nodes store the data is up to them. But you will still have to deal with the data while NR is running and - most likely - build your own flow to remove things that are older than X numbeer of days.

8

Thank you to all! This was the best thread I could find to help lead me to a solution, understanding that contrib nodes are their own animal, I wanted to increase my understanding to achieve my goal. I will take all this feedback and see if I can craft a solution.

BTW, the contrib node is storing more than the 24 hours of data for some reason. I will keep working.

Kind Regards,

9

Oh, you are talking about making the node not using the context in function. Then my previous reply is just for fun. But the logic is same. So not entirely pointless

10

Is that about the state trail node? Bug you have found?

11

Not certain I would call it a state trail contrib node bug...Still investigating

12

@hotNipi is it expected that the stored data for the state trail node would keep increasing? I would have expected it only to store data for the configured period.

13

Only for period for sure. Expecting bug report if not.

14

NOTE: I've move this conversation out of the A guide to understanding ‘Persistent Context’ thread since is is a bit off topic.

15

Understood...and thank you. I have built a separate small flow with another state trail contrib node to test the duration of the persistent storage. After some time I will review that new storage file and timestamps to determine the period for the stored data.

16

Hello...just reporting back on the investigation of this contrib node storage. Even though the "timestamps" within the file are nonsensical, it does appear to be maintaining the data in a fashion consistent with the chosen duration within the node. No Bug! Sorry for the bother.

1 Like
17

State-trail node does everything what is possible to reduce amount of data to handle and store. In case of option combine similar states is used, the not needed timestamps are thrown away and never stored.

18

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.