Memory leak, what am I doing wrong?

For me I noticed there is memory leak when the flow has debug node connected to it, even if it is disabled. I didn't have time to drill into the root cause though...

Hi @wb666greene,
Did you make any progress on this topic?
I'm also experiencing similar issues, see my other (remotely related) topic: Inspecting/Capping (per-Node?) memory usage.
Thank you!

I've not found a solution. Despite updates to node-red, nodejs, and the contrib nodes I use it still dies about once every other day.

But the out of memory killer restarts node-red and everything is running fine again within 10-15 seconds. My system has been running 24/7 since my last post in this thread and I've never noticed the issue save for examining the log files I save daily.

I haven't tried removing the debug nodes, I'm still making a few tweaks and I need the nodes for testing/verifying any changes made. Maybe in a week or two I'll been done with the minor changes suggested by a fw months of actual real-world use, and then try removing the debug nodes.

I see. Thanks for sharing all the details, including your AI project.
(Which sounds pretty pretty cool, BTW!) :grinning:

I've deleted all debug nodes and the node-red memory usage still seems to be monotonically increasing. If it doesn't get OOM killed (I see it in the log files), I'll follow-up, but so far debug nodes or not doesn't seem to be the issue here.

I seemed to have perhaps accidentally solved my memory leak.

In short, I had a serious and mysterious issue with my Lorex security DVR. To clear the problem I had to reformat the DVR and reconfigure from backup. While troubleshooting this, I messed up my node-red flow and had to delete it and restore from my exported json file backup. In the process I decided to add a bunch of extra debugging and clean some things up.

The net result is my flow appears to have stopped leaking memory after these changes.

The only change affecting the flow of the jpg image buffers from the ftp server was to remove an RBE node.

I'd had this in originally because I seemed to be sometimes getting duplicate images from the DVR. Without the RBE node filtering the buffers my flow has been running for two days and the memory usage has remained 14-16% throughout. I removed it because my new extra debugging code never showed any duplicates being sent now.

Pretty much everything node-red has been upgraded once or more since I started this thread. My other major "cleanup" was to remove some duplicate MQTT input nodes and instead pass these states via flow context instead of duplicate subscription nodes.

I hate it when problems mysteriously go away, so I'm wondering if there are any known issues with RBE node and binary buffers as payloads?

The suggestion earlier in this thread that debug nodes were involved in the leaks doesn't seem to be the case as I now have more debug nodes than ever, and the prior instance had none.

1 Like

Well it's never been reported until now :slight_smile:
but yes it would have to hold onto the old image until a new one arrives in order to compare the two.
Also it works across different topics - so if you had changing topics on the msg (maybe for some ID or other) - then there would be an old image per topic... which if they had something like date in (for no good reason I can think of :slight_smile: then it could potentially grow indefinitely.

2 Likes

I believe that this explains it! The topic is the filename from the ftp client (my Lorex security DVR) and it has the time and date embedded as a very hard to read string as part of the filename

Before I discovered the node-red-contrib-ftp-server node I had been trying to use vsftp as the server and a watch node to get the filenames to feed to the AI as they came it. This was giving me duplicates which the RBE mostly eliminated, but there was a race between vsftpd and the watch node that I never could eliminate that sometimes caused incomplete jpegs to be sent on for processing.

Thanks, I feel a lot better now that the mystery seems solved!

Now for future reference, if this situation comes up again, it looks like the reset message could be the solution if done at appropriate intervals. From re-reading the info pane for RBE node, am I correct that if I send a message with an empty topic and msg.reset added to the message object it would clear all the topics? Does msg.reset need any particular value? Presumably this reset would release all the storage.

1 Like

Yes it would. Or just delete or move msg.topic to another property like msg.filename before rbe..

1 Like