# Memory leak or what?....Restarting flow with tf coco ssd node increases memory consumption

**URL:** <https://discourse.nodered.org/t/memory-leak-or-what-restarting-flow-with-tf-coco-ssd-node-increases-memory-consumption/52493>\
**Category:** General\
**Created:** [21 October 2021 06:10 UTC](https://discourse.nodered.org/t/memory-leak-or-what-restarting-flow-with-tf-coco-ssd-node-increases-memory-consumption/52493 "2021-10-21T06:10:38Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![krambriw](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/krambriw/32/5429_2.png) [@krambriw](https://discourse.nodered.org/u/krambriw)\
**Post date:** [21 October 2021 06:10 UTC](https://discourse.nodered.org/t/memory-leak-or-what-restarting-flow-with-tf-coco-ssd-node-increases-memory-consumption/52493/1 "2021-10-21T06:10:38Z")

</div>

Having NR 2.0.6 & Buster full version running on a RPi4 w 4G. A flow that is pretty complex but runs very stable.

A fresh start of NR with the flow occupies roughly 15% of the available memory according to top and ps aux. I'm using Projects and when swithing to a previous version of the project, occupation of the memory increases, in my case with around 5% on the RPi4. Switching back increases occupation with another 5%, totalling in 10% more. Further switching shows the same increases for each switching made

This memory occupation is "confirmed stable". After running the system several hours after switching, even over night, there is no change, I expected the garbage collection to kick in but it seems the memory is not released correctly

Any ideas how I can debug this? I suspect something in the flow causing this since the same test with other flows does not behave like this. Also to mention, the same behaviour with the flow in a RPi3 shows the same behavior

EDIT: Is actually enough to just restart the flow, each restart increases the memory consumption in the same way

---

<div class="post-metadata">

**Author:** ![krambriw](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/krambriw/32/5429_2.png) [@krambriw](https://discourse.nodered.org/u/krambriw)\
**Post date:** [21 October 2021 08:04 UTC](https://discourse.nodered.org/t/memory-leak-or-what-restarting-flow-with-tf-coco-ssd-node-increases-memory-consumption/52493/2 "2021-10-21T08:04:03Z")

</div>

In the flow we are using, the _tf coco ssd_ node is placed in several subflows. This node is causing the memory occupation. Trying with a simple flow with just that node added shows the same behavior

From the Deploy button, select to restart the flow several times and check the memory occupied by node using top and you will see that it increases after each restart and it is never released - only way to release the memory is to completely restart NR or reboot

![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/1/6/1610e337d04cad20399f35e96c74c0e65c7409a7.png)

---

<div class="post-metadata">

**Author:** ![Colin](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/colin/32/17040_2.png) [@Colin](https://discourse.nodered.org/u/Colin)\
**Post date:** [21 October 2021 08:20 UTC](https://discourse.nodered.org/t/memory-leak-or-what-restarting-flow-with-tf-coco-ssd-node-increases-memory-consumption/52493/3 "2021-10-21T08:20:11Z")

</div>

What values in `top` are you using to determine that the total memory in use is going up?

If you keep doing it does node-red eventually crash?

---

<div class="post-metadata">

**Author:** ![TotallyInformation](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/totallyinformation/32/31_2.png) [@TotallyInformation](https://discourse.nodered.org/u/TotallyInformation)\
**Post date:** [21 October 2021 08:26 UTC](https://discourse.nodered.org/t/memory-leak-or-what-restarting-flow-with-tf-coco-ssd-node-increases-memory-consumption/52493/4 "2021-10-21T08:26:23Z")

</div>

I think that you will find that GC doesn't kick in unless it needs to so depending on whether you started Node-RED with a parameter to guide the GC process (--max-old-space-size?) you may find that Node.js doesn't think it needs to do anything. Maybe try temporarily setting the old space size to something small like 128 to see if GC starts kicking in.

Also, check what nodes you have installed. Nodes are "live" for all projects not just the current one. That means that the core code of every node is always executed on startup. It is possible that you have a misbehaving node installed.

If I get time, I'll try to see if this is happening on my dev system which is the only place I use projects.

---

<div class="post-metadata">

**Author:** ![krambriw](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/krambriw/32/5429_2.png) [@krambriw](https://discourse.nodered.org/u/krambriw)\
**Post date:** [21 October 2021 09:18 UTC](https://discourse.nodered.org/t/memory-leak-or-what-restarting-flow-with-tf-coco-ssd-node-increases-memory-consumption/52493/5 "2021-10-21T09:18:37Z")

</div>

> [@Colin](#):
>
> If you keep doing it does node-red eventually crash?

I can see that that it continues to increase after each flow restart if I just have that coco node on the flow. If the flow is empty, no nodes added at all, there is no increase at all. On my RPi4 I did not stress it that far but on my RPi3 I switched projects until it was completel exhausted and did not respond anymore - I waited very long but no GC seemed to happen, I simply had to recycle the power supply

> [@TotallyInformation](#):
>
> Also, check what nodes you have installed

If I add no nodes at all to the flow, there is no increase at all after flow restarts. Add several tf coco ssd nodes to the flow and the % increase will be larger after each restart

---

<div class="post-metadata">

**Author:** ![TotallyInformation](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/totallyinformation/32/31_2.png) [@TotallyInformation](https://discourse.nodered.org/u/TotallyInformation)\
**Post date:** [21 October 2021 09:20 UTC](https://discourse.nodered.org/t/memory-leak-or-what-restarting-flow-with-tf-coco-ssd-node-increases-memory-consumption/52493/6 "2021-10-21T09:20:24Z")

</div>

That's your answer then, almost certainly.

What resources does that node create/allocate for each instance? You can check whether those are cleared out in the on close function.

---

<div class="post-metadata">

**Author:** ![krambriw](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/krambriw/32/5429_2.png) [@krambriw](https://discourse.nodered.org/u/krambriw)\
**Post date:** [21 October 2021 09:24 UTC](https://discourse.nodered.org/t/memory-leak-or-what-restarting-flow-with-tf-coco-ssd-node-increases-memory-consumption/52493/7 "2021-10-21T09:24:03Z")

</div>

It's reading a model for AI object detection, I have no clue what else it is doing or if it is releasing stuff correctly, I have just discovered this behavior yesterday. The author is luckily well known, @dceejay

---

<div class="post-metadata">

**Author:** ![krambriw](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/krambriw/32/5429_2.png) [@krambriw](https://discourse.nodered.org/u/krambriw)\
**Post date:** [21 October 2021 09:41 UTC](https://discourse.nodered.org/t/memory-leak-or-what-restarting-flow-with-tf-coco-ssd-node-increases-memory-consumption/52493/8 "2021-10-21T09:41:17Z")

</div>

> [@Colin](#):
>
> What values in `top` are you using to determine that the total memory in use is going up?

If I check with ps, the result is as below. I have 5 copies of the node in the flow, nothing else. Each time I restart the flow, the memory increases. If I delete all nodes, having then an empty flow, and deploy, the memory is not released

 ![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/3/4/3419c01fa0db3a7fb60614fd1d2e5f84a1365c43.png)  
 ![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/c/4/c4deb032e6f43bfccb8e02966498ebfa39904d94.png)

---

<div class="post-metadata">

**Author:** ![Colin](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/colin/32/17040_2.png) [@Colin](https://discourse.nodered.org/u/Colin)\
**Post date:** [21 October 2021 10:23 UTC](https://discourse.nodered.org/t/memory-leak-or-what-restarting-flow-with-tf-coco-ssd-node-increases-memory-consumption/52493/9 "2021-10-21T10:23:17Z")

</div>

It does look as if the node is not releases its resources on close. Maybe add an issue on the node's github page.

---

<div class="post-metadata">

**Author:** ![dceejay](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/dceejay/32/38_2.png) [@dceejay](https://discourse.nodered.org/u/dceejay)\
**Post date:** [21 October 2021 15:30 UTC](https://discourse.nodered.org/t/memory-leak-or-what-restarting-flow-with-tf-coco-ssd-node-increases-memory-consumption/52493/10 "2021-10-21T15:30:33Z")

</div>

OK - I have pushed 0.5.7 that explicitly clears the model (and font) on close... but the way node.js works the GC still may not kick in until it feels like it...

---

<div class="post-metadata">

**Author:** ![krambriw](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/krambriw/32/5429_2.png) [@krambriw](https://discourse.nodered.org/u/krambriw)\
**Post date:** [21 October 2021 17:29 UTC](https://discourse.nodered.org/t/memory-leak-or-what-restarting-flow-with-tf-coco-ssd-node-increases-memory-consumption/52493/11 "2021-10-21T17:29:44Z")

</div>

Unfortunately, it did not help

I put 12 copies of the node on a flow (nothing else). After each flow restart, memory consumption increases with around 5% on a RPi4 w 4G. When memory consumption increases it takes longer and longer for the models to be loaded into all 12 nodes. When it reaches around 69%, it is not able to load all nodes with the models. After another flow restart, NR is no longer able to load the model into any of the 12 nodes. Node is now occupying 72% of the memory

On a RPi3 it has difficulties to load the models in all 12 nodes already from start, it takes long time, I think memory is not enough so continuing the test on a RPi3 is meaningless

 ![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/f/8/f8fca74cebc6186703310b1e2a7cbc57335d7473.png)  
 ![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/b/0/b01e413d209006f96f4549821aa2bcf6c13a5019.png)

---

<div class="post-metadata">

**Author:** ![krambriw](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/krambriw/32/5429_2.png) [@krambriw](https://discourse.nodered.org/u/krambriw)\
**Post date:** [24 October 2021 06:08 UTC](https://discourse.nodered.org/t/memory-leak-or-what-restarting-flow-with-tf-coco-ssd-node-increases-memory-consumption/52493/12 "2021-10-24T06:08:11Z")

</div>

Made a comparison test, just to have something to compare, using another node for ssd object detection

> **[node-red-contrib-tensorflow](https://flows.nodered.org/node/node-red-contrib-tensorflow)**
>
> TensorFlow.js nodes using pre-trained models

When I do the same tests, repeatedly restarting the flow, the memory consumption does not build up. So it must be something else than loading/offloading the model

I would like to keep using the tfjs-coco-ssd node since it is also much faster in analyzing

 ![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/8/f/8fb935996462daad005d6ad32e732ce886ecb826.jpeg)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/1X/d073cd938eafa2e558d7c2cd59003b3ef4963033.png) [@system](https://discourse.nodered.org/u/system)\
**Post date:** [23 December 2021 06:08 UTC](https://discourse.nodered.org/t/memory-leak-or-what-restarting-flow-with-tf-coco-ssd-node-increases-memory-consumption/52493/13 "2021-12-23T06:08:28Z")

</div>

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.