Performance get/set variables. different methods

[edterbak]

Hello,

I have a performance question.
There are multiple methods of getting and setting variables. I would like to know what is adviced, or what gives the best performance between them.

Question 1: what method is adviced:

Method 1: single call
var example1 = global.get('storedvariable1',"file");
global.set per variable.

Method 2: multi call
var values=flow.get(["v1","v2"]);
var v1=values[0];
var v2=values[1];

But... method 2 can also be in the form like this:

MQTT.block_active
MQTT.allow_rtc_onoff
MQTT.allow_scheduler
MQTT.allow_solar
MQTT.block_mode
MQTT.message_limit
MQTT.counter
MQTT.messages_yesterday
MQTT.messages_today
MQTT.block_mode_previous

Calling variable MQTT
var MQTT = global.get('MQTT', "file");

In this example, it could be in reality that I only require 2 variables from MQTT in the function after reading.

The questions are:
Q1: what method of getting variables would be best for performance.
Q2: Using method 2 in the form I wrote in the example, would loading the extra variables result in a performance thing?

Store to file vs memoryOnly:

I use 2 storage methods. for live values only use memoryOnly, and for storing graphs or settings store in 'file'.
The settings have a flush timer of 300 seconds.

Q3: when global.getting a variable from "file", does that mean there is a read action done?

Q4: When setting a global variable to 'file' location, does the variable store DIRECTLY to file/disk, or collect it during 300 seconds, and THEN store it to file?
For the amount of writing actions.. I can see there is a significant performance topic here in case the actual writing is done instantly instead of once per 300 seconds.
The things I have read still leave me with this question. I THINK the flush timer is collecting all changed variables and writing it once 300 seconds. But I'd like this confirmed

Many thanks for all the support here.
Great forum.

Kind regards,

[Steve-Mcl]

@edterbak

I have a performance question.
There are multiple methods of getting and setting variables. I would like to know what is adviced, or what gives the best performance between them.

Why? Are you hitting a bottleneck of some kind? If not, then dont worrry (see premature optimisation is the root of all evil )

Your Methods 1 and 2 are pretty much the same in terms of code execution however, there is a 3rd way that is definitely faster...

const example3 = flow.get('myData') || {}  // Get 1 flow context variable (or new new {} is empty)
let storedVar1 = example3.storedVar1
let storedVar2 = example3.storedVar2
// ...
let storedVar98 = example3.storedVar98

let aNewStoredVar99 = storedVar1 + 1
let aNewStoredVar100 = storedVar2 + storedVar3
example3.storedVar99 = aNewStoredVar99
example3.storedVar100 = aNewStoredVar100
flow.set('myData', example3) // Update 1 flow context variable

In other words, you get / set an object that contains all of you stored values.

@edterbak

Q1: what method of getting variables would be best for performance.

See above

@edterbak

Q2: Using method 2 in the form I wrote in the example, would loading the extra variables result in a performance thing?

See above

@edterbak

Q3: when global.getting a variable from "file", does that mean there is a read action done?

No. Data stored in the file is loaded (cached) in memory upon starting node-red. Each subsequent get and set is done against the in-memory cache

@edterbak

Q4: When setting a global variable to 'file' location, does the variable store DIRECTLY to file/disk, or collect it during 300 seconds, and THEN store it to file?

Values are written to the in-memory cache & after the configured time, they are flushed to file.

[TotallyInformation]

I think that, in the past, we discovered that even if you try to load a specific property from a context variable object, the whole object gets loaded to memory anyway and so there are no benefits to trying to get just a single part of an object variable. Since JavaScript objects are passed by reference and not by value, assigning to another variable is the tiniest of overheads if any.

There is another tweak which can be helpful as well. You can use modern spread methods. In particular syntax such as:

const {a, b, c} = flow.get('myObject')

[edterbak]

Hi, thanks Steve-Mcl

The reason I am asking is that I am now at a point of restructuring the variables I have. There are about 180 of them I load / save. Not all at once, but when they are needed.
There are groups of variables that kind of belong together.

Now that I am restructuring making things more logical, better to take along the information about performance now, than later run into issues.
But reading the 2 responses, I see it is not something to worry about.

So; There is no benefit in reading one object containing multiple values vs multiple readings containing 1 variable. (apart from the coding itself)
Also when looking at the CPU load aspect?

[edterbak]

Thanks TotallyInformation; So basically if you define a list of variables in front of the =, you can pupulate them after the = in the right order.

Nice tip Thanks

[edterbak]

@Steve-Mcl

No. Data stored in the file is loaded (cached) in memory upon starting node-red. Each subsequent get and set is done against the in-memory cache

Oh, wait....

Does this also mean there is no benefit of having a variable 'xyz',"file" AND 'xyz',"memoryOnly".
If the "file" variable is used, there is totally no need to have the same variable in "memoryOnly"?

This would reduce some coding lines anyway

When you confirm this, I have some trimming to do

[Colin]

@edterbak

There are about 180 of them

Do you really need that many in global context? I very rarely use flow context and virtually never global. Have you considered whether you might be better to restructure some flows to get rid of some of them?

[edterbak]

Well, I do think most are needed. Some I could get away with flow, and also some are required to be in context.

Not meant as an add or something, but the background (usecase) for this:
I started using NR to communicate with my Panasonic heatpump.
A lot of nice things to show, set and do. Also I have created custom functions for things which are not present in the panasonic equipement.
I have shared this project on github. At first I was the only one. Now after two years it grew and it is being used by I think atleast 30 people. But it might as well be more than 100, I cant tell. Kind of nice for an amateur like me.

GitHub - edterbak/NodeRed_Heishamon_control: a Node Red flow to interact with...

note that I am not happy the way the flow now looks. The GUI needs structuring and polishing to make it intuitive again. But thats my challange.

Anyway;
By the nature of the project, I require settings to be stored so they are still available after reboot.
Most variables need to be available on all tabs. Not all. But from my point of view. There is no benefit of setting a flow compared to a global variable.

[Colin]

@edterbak

Does this also mean there is no benefit of having a variable 'xyz',"file" AND 'xyz',"memoryOnly".
If the "file" variable is used, there is totally no need to have the same variable in "memoryOnly"?

If you do that they are different variables.

[Colin]

@edterbak

By the nature of the project, I require settings to be stored so they are still available after reboot.

Do you use MQTT?

[edterbak]

different variable, with the same value... sounds to me like "overhead". correct?

[edterbak]

yes, all is done through mqtt.

[Colin]

@edterbak

all is done through mqtt.

In that case you could consider using Retained Topics in MQTT rather than context. It has the big advantage that when data changes you immediately get notified about the change.

[edterbak]

No, Yes.. welll.

I could do that, but I have already trapped into that error that values are retained and incorrect causing issues in the logic.
I think I am better off using retain false, qos 0 to make sure I have the correct variables.

For me, not fully understanding the effects and the rules regarding to retain messeges, it is the door to hell. ghehehe.

(the situation I had was that the pump was swittching some settings every couple of minutes due some retained value. After removing the topic, issue was gone. But this is likely my shortcoming)

This would shave off about 30 variables at most. probably less. The rest is in my logic and functions.

[Steve-Mcl]

@edterbak

So; There is no benefit in reading one object containing multiple values vs multiple readings containing 1 variable. (apart from the coding itself)

Not quite.

Loading and Saving 1 object (full of your variables) is far quicker than calling flow.set or flow.get multiple times. This is due to several layers (plugin, path parsing, value extraction etc).

When you get 1 object, it pulls only the reference from memory (not all of the individual value) so is faster.

[edterbak]

@Steve-Mcl

Not quite.

Loading and Saving 1 object (full of your variables) is far quicker than calling flow.set or flow.get multiple times. This is due to several layers (plugin, path parsing, value extraction etc).

When you get 1 object, it pulls only the reference from memory (not all of the individual value) so is faster.

This is very usefull. Mega thanks.
Also nice to read that the derection I was thinking in, looks te be the right one according to your reply.

Thank you for the reply and clear answer..

[TotallyInformation]

@edterbak

Does this also mean there is no benefit of having a variable 'xyz',"file" AND 'xyz',"memoryOnly".
If the "file" variable is used, there is totally no need to have the same variable in "memoryOnly"?

That's right - the file store is really pretty much the same as the memory store but the memory is written to file every NN seconds and read from file on Node-RED startup. (There is actually one really important difference though. When written to file, the data has to be serialised and not everything that JavaScript can store in memory can be serialised. Hence the reason that Node-RED tends to automatically remove the ExpressJS res and req properties for example).

When you do a const xx = ....get(....) from any store, you are simply creating a reference to the memory location.

[Colin]

@edterbak

different variable, with the same value

No, really different variables, that can have different values.

[Buckskin]

@TotallyInformation

There is another tweak which can be helpful as well. You can use modern spread methods. In particular syntax such as:

const {a, b, c} = flow.get('myObject')

The advantage of this approach is (taking your MQTT Object as an example)

let {lmessage_limit, allow_rtc_onoff} = flow.get('loadsOfThings').MQTT

[edterbak]

To write it out completely...

MQTT.block_active
MQTT.allow_rtc_onoff
MQTT.allow_scheduler
MQTT.allow_solar
MQTT.block_mode
MQTT.message_limit
MQTT.counter
MQTT.messages_yesterday
MQTT.messages_today
MQTT.block_mode_previous

getting all variables would be like this:

let {block_active, allow_rtc_onoff, allow_scheduler, allow_solar, block_mode, message_limit, counter, messages_yesterday, messages_today, block_mode_previous} = global.get( 'MQTT', "file");

Correct? or am I interpreting it wrong now

thank you all guys for the guidance. This stuff will probably speed up the flow and responsiveness, and reduce resource footprint.