Hi folks,
Since I'm experimenting quite a lot with audio and video in Node-RED, I have to be cautious all the time that I don't store audio/video chunks somewhere in memory. Indeed I have very few memory on my Raspberry, so I need to use it with caution. Nodes like rbe/delay/... are quite memory-consuming when using this kind of data. Especially when these nodes are used 'topic-dependent'. From this discussion last night, I see that I'm not the only one struggling with this kind of problems...
I have now an experiment with cameras that works well, and after 10 minutes suddenly an out-of-memory and my raspberry goes down. Don't have clue what is going wrong. Would be nice if there was some way in Node-RED to follow up memory (e.g. on node-basis), to make troubleshooting a lot easier.
I don't think this is currently available, but if I am mistaken please correct me!!!! Can anybody give me some ideas how to handle this best?
- In my use case, a visual approach (e.g. somewhere in the flow editor) might help. E.g. in a side-panel or in the context store panel, or on the node itself, or the info panel...
- But it should be nice if I could log the memory usage somewhere in a file, because I don't know when it goes down (and it goes down quickly due to the large amount of data arriving). So perhaps I need some kind of memory logging node??
- ...
And secondly I'm wondering what is the best way (technically) to determine the size of a node??? I'm thinking about shared memory (somehow) or circular references perhaps, a node storing data on flow/global memory ... And not all data is stored in the context, like e.g. in the rbe node (from the discussion in the link above):
Since the data is stored in 'this' node, it won't be visible in the context store panel anyway:
Thanks !!!
Bart
I started to create my own proposal, so everybody can shoot at it 
I created an experimental node-red-contrib-memory-analyser node, that calculates the current memory usage from all selected nodes (to find quickly which node is leaking memory):
Currently nodes can be selected on their type (e.g. 'http request') and their name (e.g. 'myHttpNode'). A few other options should be added to the config screen:
- Perhaps I could also add other filters like 'only active nodes' or 'node id filter' ...
- There should also be an option to calculate flow and global memory size somehow...
- Only show nodes with memory usage above a threshold, both relative (percentage) and absolute (bytes) values allowed.
- Only show nodes with a memory increase (since the previous measurement) above a threshold.
- ...
I assume that a node can have TWO TYPES OF OWN MEMORY (they can also store data in flow or global memory, but I don't know which node has stored which data):
-
Data inside the node instance itself. For example the httpRequest node contains following information:
The size of the node is determined by using the object-sizeof, which 'seems' to be working:
P.S. The 'wires' array seems not to be pointing to the next nodes in the flow, so I'm not calculating the size of the entire node chain (but only of 'this' node):
-
Data stored in node context memory. I thought to calculate that size by looping all the context keys and cumulating the size of all corresponding context values:
var contextSize = 0;
for (var i = 0; i < nodeToAnalyse.context().keys(); i++) {
var key = nodeToAnalyse.context().keys()[i];
analysis.context_size += sizeof(nodeToAnalyse.context().get(key) || {});
}
However I cannot access the context of other nodes 
. Of course that is completely normal, since we don't want nodes to mess with other nodes memory content. So I wouldn't want it to be another way.
However when I want to have a complete memory analysis by node, I'm completely screwed ...
Since I don't have a node.size(), node.context().size() or ... available, I don't know how to continue with this. Can anybody offer me the tip of the day?
My quick response is that if youre putting lots of effort into this please don't rush to publish a node until weve discussed it properly. This ought to be part of the platform and not a node in a flow.
I'm offline today - let's pick this up tomorrow.
2 Likes
Did this ever get anywhere?
Hi Julian (@TotallyInformation),
I'm back in Node-RED world, but my batteries are quite empty at the moment...
Have published node-red-contrib-nodes-memory@0.0.1-beta.3 so we have at least some starting point for our discussion. On the readme page you can find:
- Under "Node Usage" shortly explained my idea about how the memory analysis could be triggered.
- Under "Limitations" there is a list of already known issues. Some of them can hopefully be solved, but some of them I can never solve. But the latter ones are of less interest to me, so they will just become 'known limitations'...
It started as a simple idea to help users that are getting a headache finding which Node-RED node is consuming too much memory. But as always, it is again getting complex 
Therefore all 'constructive' feedback is welcome!!
I had to quit my development this evening, because of limitation nr 2 (which makes my node quite useless)! Due to changes in Node-RED (introduction of context stores some time ago) my trick from last year - to calculate the context size of other nodes - doesn't work anymore (caused by a 'scope' variable that I cannot overrule ...). The easiest solution I can think of, is addition of a node.getSize function in Node-RED which returns the size of a node.
Let's suppose this idea would be accepted (??), I already have some doubts about the implementation of that function:
- As specified in limitation nr 4 the object-sizeof has very limited documentation, so I don't know if it is accurate enough (although it is very popular ...).
- Every node could have both it's own node memory and also context memory. Those could be calculated e.g. like this:
const sizeof = require('object-sizeof');
// Calculate the size of the node.
// The size of the wires could be subtracted, in case wires gets extra properties in the future??
var nodeSize = sizeof(node) /*- sizeof(node.wires)*/;
// Calculate the node context size, as the sum of the sizes of all variables on the node context memory
var contextSize = 0;
for (var j = 0; j < node.context().keys(); j++) {
var key = node.context().keys()[j];
contextSize += sizeof(node.context().get(key) || {});
}
But should the function return those two sizes separately, or only a single total sum of both sizes? In my node I have returned both, so you can analyze even in more detail where the memory is being consumed ...
- As explained in limitation nr 5, some nodes will use memory outside of the V8 engine. With OpenCV for example, we had a problem because the memory used in V8 was entirely negligible compared to the C++ memory being used. Should a node be able to add its own extra memory size to the result of the
node.getSize somehow? Of course not much contributed nodes would implement this feature ... [EDIT] I mean the size that a node can return should be a separate number, to make sure you can easily determine rhat memory is being used outside of NodeJs.
Had hoped to be able to create a custom node without having to touch the Node-RED core, but I don't see anyway at the moment how I could accomplish that ...
Bart
Hi Bart, a brief response I'm afraid as I'm trying to wrap up work so that I can go on holiday.
It looks like you've started to discover some of the things I alluded to I think in another thread. Namely that JavaScript memory utilisation is complex and JS isn't terribly good at introspection.
Yes, of course, especially since Node.js handles external, non-js libraries very easily. Spawn'd processes would probably also need to be accounted for separately?
In uibuilder v2 for example, I'll be spawning npm commands. I can't see any way that you could account for that.
From your readme: " node is based on the object-sizeof node, which doesn't explain how the memory is calculated" - actually it does explain it. It simply walks through a js object until it finds one of a string, number or boolean. Since the sizes of those are known via the standards, it accumulates the sizes until it runs out of object.
You will note that it doesn't include the size of the program code itself.
@TotallyInformation, hope you have a nice holiday!
1 Like
