Information on memory/buffer

General
1

Hi all, I'm trying to figure out how to use NodeRed's memory.

I acquire data through modbus registers.

The goal of my flow is to collect all the data and aggregate it every 15 min.

I already managed to do this using 2 postgress tables: raw_data_table and 15min_table, by query I delete the raw data table each time.

However, this process is expensive for postgress if there is a lot of data to be calculated and deleted at the same time.

The goal is to create a unique INSERT every 15 min using node red memory:
INSERT INTO 15min_table key, dev, var, SUM(value)

1 Like
2

Have a look in the docs for context storage. This lets you use memory (or some other options) to store variables.

3

I don't think it's useful, I have to record more logs (200 per hour) with a 10 second poling and then do the average every 15 min, then later that data will be sent to a SQL DB

4

You asked how to use Node-RED's memory and I answered that. It is more than capable of handling large amounts of rapidly changing data. The main limit being your device's memory and available compute power.

Had you asked for ideas on using something other than Postgress, nearly everyone in the forum would immediately have said: InfluxDB. It is made exactly for that kind of processing of time-based data and can easily aggregate over time.

With InfluxDB, you simply send all the data into a table and let the engine do the aggregation.

5

Sorry for the lack of clarity. I will still use postgres because I need its table_15min, do you think the InfluxDB+Postgres solution is more performant?

6

I'm thinking of installing postgres on another machine. Create the two tables on influxdb and then copy table_15min_influxdb to table_15min_postgres. Do you think it's a winning choice? I'm thinking of more solutions

7

So if you have to use Postgres, do the aggregation in node-red using context vars and then use a prepared statement to write the 15min data, that will be the most efficient.

8

TotallyInformation meanwhile, thank you for your support. Maybe I don't understand the utility of context variables; as far as I know the Flow or Global variables can be retrieved from other unconnected nodes, but I don't understand how to dynamically load N variables and then make an average. Suppose I need to load 50 values ​​to global.abc variable, I don't want to create global.abc1, global.abc2.... global.abc50

9

You can save them in an array, all 50 in one array variable. Each time a new one comes in then get the current array, add the new one to it, and save it again.

1 Like
10

Thanks, I'll try with the context

11

Yes, that is correct. The confusingly named "context" context variables are only accessible to the node that created them. I've made several attempts in the past to get context vars added to the list of options the same way that global and flow are because I think it would be sensible to offer them as an option in many places. Unfortunately, I've not managed to convince the core devs of that utility. So where I need the option, I've had to craft my own interface.

As Colin points out, you don't need to do that. That would create multiple variables. You only need 1.

In a function node, it would look like this:

// Get the global var or create an empty array if it doesn't yet exist
const abc = global.get('abc') || []

// Push the new value onto the end of the array.
abc.push( msg.payload )

// Restrict the max entries - removes the oldest entry
if (abc.length > 50 ) abc.shift()

// You can recalculate your average or some other aggregation here...
// Try doing an internet search for "mdn reduce".

// Save the updated array
global.set('abc')

You could go much further by making your variable an object where abc.data becomes the array and abc.average contains the average of all the entries. Or whatever you need to do.

14

that's just what I need, your example unfortunately doesn't work.

The final goal is to enter an array of values ​​and calculate the mean, then the mean of object 0 called A, object 1 called B and so on.

The more I use NodeRed, the more I realize its potential.

I created the flow like this following your example:
Screenshot 2023-09-05 141943

EDIT: the last line is

// Save the updated array
global.set('abc', abc)
15

In what way does it not work? If you want to see abc you can look in the context tab in the right hand pane, or you could add the line
msg.payload = abc
before the line
return msg

16

There is one small mistake that you spotted already. I've just tested the correction and it absolutely does work.

17 
	contextStorage: {
		default: "file",
			memoryOnly: { module: 'memory' },
			file: { module: 'localfilesystem' }
	},

I set the contextStorage like this (default: filesystem) is it possible via function (I know you can use the Change node) to set the storage type?

Example can I ask via function to save message X on memoryOnly?

18

Yes, certainly. If you have >1 storage type, you adjust the get and set functions, adding the name ("memoryOnly" or "file" in your case) as an extra argument to the function. That is provided as a string so you can replace the fixed string with a variable.

const store = 'file'
const abc = global.get('abc', store)
19 
var array = msg.payload;
const store = 'localfilesystem'
const data = flow.get('json_spooler', store) || []

data.push(array)
flow.set('json_spooler', data)

return msg.payload;

I reset memoryOnly to default.

I created the function using file or filesystem, but nothing always save in memoryOnly mmmm

20

try:

flow.set('json_spooler', data, store)
1 Like
21

Also make sure that all uses of json_spooler refer to the correct store. If you use different stores then it will refer to different flow variables with the same name, but in different stores.

1 Like
22

I can't, now I'm trying with a Change node in this way:
image

[$flowContext('json_spooler') , msg.payload]

Of course, if I use memoryOnly, it creates the array and updates it, if I use file, it replaces it with the last received message.