How Node-red's memory recycling mechanism works

General
1

I installed a Node-red on a Linux embedded device with limited space, and wrote some complex flowcharts.

Through the "top" command in Linux, it was found that Node-red has a very high usage of virtual memory (exceeding 50% of the total).

I extensively use flow. set ("data_name", data_val) in the flowchart, but after using it, I did not explicitly set the variables to null, undefined, or destroy them, so I found that even after redeploying, the values of the variables still exist.

Based on the above situation, I have some doubts:

(1) Will variables set by flow. set() be automatically reclaimed (memory is automatically cleared)?

(2) What method is required to manually clean variables?

(3) If there are multiple places in the flowchart that "simultaneously" call flow.set() to assign values to the same variable, will there be any problems? Do we need to write a mutex mechanism?

(4) Is the working principle of flow.set() and Change Node the same?

(5) What methods are there to reduce the memory usage of Node-red?

2

My understanding of how node-red handles memory is based off of year 2020. I don't think it has changed.

Non context is uses the Node.js runtime rules. It will handle garbage collection at a certain ammount. I don't know the stock value but you can set it "Passing arguments to the underlying Node.js process" ---> Running Node-RED locally : Node-RED <---read about it here.

Context (set.flow / set.global)
is stored in ram unless you setup a context storage yourself ---> Working with context : Node-RED

default is to use ram.

To keep ram usage low you will need to set your flow/global vars to empty or null =, ={}, ="" when you are done using them.

I hope this helps you find what your looking for.

3

Your guidance has been very helpful to me.

That is to say, variables assigned by "set. flow/set. global" must be manually and explicitly released, otherwise they will remain in RAM (and will not disappear even after redeployment), unless Node-red is restarted, is that so?

4

Just like the heap space allocated using malloc() in C language, must it be released through free() after use?

5

Yes .... but I'm sure there is a way in the function node or api to release/clear a set.flow pragmatically but I've always just set them to null and it works fine.

I hope someone smarter than me can answer your question that knows the innards of this better than me. I mainly run node-red on a dockers on servers with 256GB of ram so its not my main concern. I've only learned what i needed to know to keep things/usage from getting out of control.

Sounds like your working in a very memory constrained environment.

7

Yes, the memory space of embedded devices is relatively small,

Thank you very much for your guidance. I have learned a lot~

8

I think it would be Best to set them to undefined to remove the reference to them from memory. flow. set ("data_name", undefined)

9

Thank you for your suggestion.

I just checked and found that I have called flow.set for 400 times. I need to check them one by one and set them undefined when appropriate. This is a major project ::

10

Can confirm:

Once you set it to undefined it will be removed from the object that stores the context.

2 Likes
11

Thank you! That's amazing, just analyze the source code directly

12

This suggests that your design is not ideal. The node red way is to pass data using messages, then when a message is finished with the memory will be automatically reclaimed. Flow and global context should generally only be used when it is not practical to pass the data in messages.

14

I am trying to design a complex industrial control system using Node red,

I have some requirements that may require separation operations to achieve:

For example, using MQTT to send a request to the server, and then subscribing to the processing result. When it is determined to be successful, subsequent operations are triggered.

I used a flow context to mark whether the processing was successful.

15

No, not unless you set them to undefined or use a change node to delete them

See previous

No different to any other application - but based on your post, I suspect you are using context too much and unnecessarily.

Yes - same code under the hood

Avoid context unless necessary. I consider using context akin to adding globals to an application (leads to bugs and difficult to debug). Try to program with PURE principles (pass data in the msg), only store when then there is no other options.

1 Like
16

On the other hand, industrial systems are constantly collecting data, and sensor data is constantly being updated,

I wrote the data into the flow context so that the latest sensor data can be called from anywhere,

So unfortunately, I used a lot of flow context

17

Thank you very much for your detailed answer,

Regarding this point, I would like to ask more, does "No different to any other application" means the need to use mutexes?

18

Since the application is nodeJS (and thus single threaded*) a mutex is not required however you can cause concurrency issues that are just as problematic to debug. Remember how I said using context was akin to using globals in any other type of application? Add to that, the truly async nature of node-red, you can hit concurrency issues when storing / reading context/globals in a flow called repeatedly (in quick sucession). That is why I suggest sticking to PURE principles.

1 Like
19

Fortunately, you reminded me and I will immediately modify my program. Thank you again for your guidance

20

No. It is retained until you either delete the variable or restart node-red.

You can use a change node to delete it.

Timing with node.js apps can be tricky, if you really need to guarantee an order of update, you should build that into your flows. The trickiest thing is if you end up using a node that is asynchronous in operation since knowing when the update actually happens is going to be almost impossible.

Yes.

Almost impossible to answer since it highly depends on what nodes and what data you are using and how.

But minimising the use of context variables is certainly something you can do. Given that in normal operation, Node-RED passes "messages" between nodes and these are transient, that is likely to be a lot more memory efficient. Noting that "messages" aren't really that at all, they are a reference to a memory object that is passed between nodes until the section of flow comes to an end.

So even with messages, if you have a long and complex flow and part of it needs a very complex, large msg object, it might be worth considering, at some appropriate point in the flow, forcing the creation of a new smaller msg to allow the large one to be dropped.

Node-RED uses node.js so understanding the memory model for that is quite important in this situation. Node.js uses a "garbage collection" technique for memory recovery. So even when you've finished using some memory, it isn't immediately recovered, node.js waits until it thinks it needs to tidy up before recovering the memory. This is why you will typically see memory use for a node.js app creep up and then suddenly drop. At the same time, you will likely see a CPU spike. In extreme cases, you may see the node.js app pause while this happens.

21

That part could probably be replaced with another MQTT topic which would let you use subscriptions to manage the flows as event driven.

However, a flag is unlikely to be causing you memory issues.

Ultimately, you are going to have to decide whether:

  1. It is actually worth the effort of re-engineering flows or you should move to a more capable server
  2. Whether some of your processing should, in fact, be moved out of Node-RED into a more resource efficient processing environment.
22

But I am currently developing a communication system based on MQTT, and the published information may not be replied to for various reasons. I must wait for a period of time (enough time to confirm that the reply has been lost) before re publishing the same message. If there is still no reply after sending twice, this message will be discarded.

Meanwhile, if the first message is successfully sent, the same message will not be sent again.

In order to determine whether need to send the message again, I must use the flow context to confirm whether the first message has been successfully received (if received, the process on the mqtt side will set the flag to 1, so that the sending process knows that it does not need to send again).

If this method is not used, is there a better way?

1 Like