Analyse Node-RED memory usage

General
5

Hi Julian (@TotallyInformation),
I'm back in Node-RED world, but my batteries are quite empty at the moment...

Have published node-red-contrib-nodes-memory@0.0.1-beta.3 so we have at least some starting point for our discussion. On the readme page you can find:

  • Under "Node Usage" shortly explained my idea about how the memory analysis could be triggered.
  • Under "Limitations" there is a list of already known issues. Some of them can hopefully be solved, but some of them I can never solve. But the latter ones are of less interest to me, so they will just become 'known limitations'...

It started as a simple idea to help users that are getting a headache finding which Node-RED node is consuming too much memory. But as always, it is again getting complex :woozy_face:

Therefore all 'constructive' feedback is welcome!!

I had to quit my development this evening, because of limitation nr 2 (which makes my node quite useless)! Due to changes in Node-RED (introduction of context stores some time ago) my trick from last year - to calculate the context size of other nodes - doesn't work anymore (caused by a 'scope' variable that I cannot overrule ...). The easiest solution I can think of, is addition of a node.getSize function in Node-RED which returns the size of a node.

Let's suppose this idea would be accepted (??), I already have some doubts about the implementation of that function:

  1. As specified in limitation nr 4 the object-sizeof has very limited documentation, so I don't know if it is accurate enough (although it is very popular ...).
  2. Every node could have both it's own node memory and also context memory. Those could be calculated e.g. like this:
    const sizeof = require('object-sizeof');
 
// Calculate the size of the node.  
// The size of the wires could be subtracted, in case wires gets extra properties in the future??
var nodeSize = sizeof(node) /*- sizeof(node.wires)*/;
             
// Calculate the node context size, as the sum of the sizes of all variables on the node context memory
var contextSize = 0;
for (var j = 0; j < node.context().keys(); j++) {
   var key = node.context().keys()[j];
   contextSize  += sizeof(node.context().get(key) || {});
}
    But should the function return those two sizes separately, or only a single total sum of both sizes? In my node I have returned both, so you can analyze even in more detail where the memory is being consumed ...
  3. As explained in limitation nr 5, some nodes will use memory outside of the V8 engine. With OpenCV for example, we had a problem because the memory used in V8 was entirely negligible compared to the C++ memory being used. Should a node be able to add its own extra memory size to the result of the node.getSize somehow? Of course not much contributed nodes would implement this feature ... [EDIT] I mean the size that a node can return should be a separate number, to make sure you can easily determine rhat memory is being used outside of NodeJs.

Had hoped to be able to create a custom node without having to touch the Node-RED core, but I don't see anyway at the moment how I could accomplish that ...

Bart