Struggling with memoryleak

General
1

Hello!,

I notice i have a memoryleak when listening to a mqtt broker, and putting the payload into a queue:
@ba47/node-red-queue - npm (npmjs.com)

image

It seems like this node always creeps up, but never frees memory.
I have tried several other queue nodes as well with the same issue. Also same issue if i disconnect the output as well.
Is there some core issue i am missing here?

2

Welcome to the forum @RonnyR

For what reason are you queuing the messages? There is probably a better way of doing it.

3

Did you read the Issues section in the link you provided? Could that be describing your problem?

4

Hi, thanks for replying, yes i have read the issues section and its not solving the issue.
About the reason for doing this:
I am trying to create a timeframe of 3 seconds where i store all the messages to create a snapshot of the set of datapackages to detect deviations.

I use a mqttbroker node->decode with a protobuf file->to this queue.

5

Unfortunately the node does not have a link to a repository where you can submit issues, which is often a bad sign. You may have to find an alternative solution.

6

Thank you Colin, ill try some other nodes as well, if people have any suggestions i appreciate it. :slight_smile:

7

What does a "deviation" mean? Can you give an example?

8

Sure!, I want to gather 1000 of payloads, containing a address and a number, if the number deviates i want to collect them into a "bucket" to process them even more as a deviation and create a report. :slight_smile:

9

Do you mean you want to collect all the payloads (just the payloads) over a fixed time and then release them in one message?

10

Correct, i want to collect all payloads over a fixed time and them release them collected in one message for further processing. First stage of a filtering sequence really.

11

There is a Filter node which you can setup to only pass messages where the payload has changed.
You could use that to only report when the number for any address changes.

Another approach I would consider is to save the data to a database and use SQL to report on deviations.
If you accumulate thousands of records in a relatively short time this would need careful housekeeping to keep the db size reasonable.

12

Thank you for your response.
I will look into the filter node as well.
Also I have a database, but I would expect a standard queue handling should be easy to memory manage, not to leak. I have tried several nodes now, also q-gate, here its a lot less, so I guess I have to deep dive more into node.

13

Perhaps something like this will do what you want

image
image796Ă—156 14.3 KB

[{"id":"a4f83620bc9c7846","type":"function","z":"bdd7be38.d3b55","name":"Queue payloads","func":"let payloads = context.get(\"payloads\") || []\nif (msg._release) {\n    // the incoming message includes msg._release true so pass on the payloads\n    msg.payload = payloads\n    payloads = []\n} else {\n    payloads.push(msg.payload)\n    msg = null // stop the return from sending a message\n}\ncontext.set(\"payloads\", payloads)\nnode.status({text: `queued: ${payloads.length}`})\nreturn msg;","outputs":1,"timeout":0,"noerr":0,"initialize":"","finalize":"","libs":[],"x":560,"y":2920,"wires":[["0abb31cae9c20c03"]]},{"id":"aef84022f34c57bd","type":"inject","z":"bdd7be38.d3b55","name":"Simulated incoming messages 1 per second","props":[{"p":"payload"}],"repeat":"1","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"iso","payloadType":"date","x":270,"y":2880,"wires":[["a4f83620bc9c7846"]]},{"id":"fec86de385320a44","type":"inject","z":"bdd7be38.d3b55","name":"Release every 5 seconds","props":[{"p":"_release","v":"true","vt":"bool"}],"repeat":"5","crontab":"","once":false,"onceDelay":0.1,"topic":"","x":300,"y":2980,"wires":[["a4f83620bc9c7846"]]},{"id":"0abb31cae9c20c03","type":"debug","z":"bdd7be38.d3b55","name":"Queued payloads","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":800,"y":2920,"wires":[]}]
14

Absolutely, i will try to test this out, as I had a similar solution.
Thank you :slight_smile:

15

It seems like its my gRPC node which i have no lifecycle control over thats leaking memory.
If the gRPC node is the last node the memory slowly creeps upwards.
If i disconnect it, the memory is stable.
I use:
node-red-contrib-grpc (node) - Node-RED (nodered.org)

Is there anything else you should think of when i am using this node?

I trigger it quite a few times, like 5 times a second. I will try to add a delay to see if that helps as well.

16

I would first ask the developer to update dependencies to see if that helps any. It is using versions of gRPC and protobuf that are ~4y old and have had lots of updates since the node was released.

NOTE: the repo has bump updates sitting waiting since 2022 - so dont hold your breath.

Alternatively, fork it, update the dependencies, install your fork direct from GitHub.

1 Like
17

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.