Hello, I've been looking to measure the amount of memory, CPU cycles, and time used by a few different flows and nodes. I've taken a look at a number of posts on the forum, but have run into a few issues:
Elastic APM or OpenTelemetry could be the solution for what I need, but I've found them too complicated to set up thusfar and the nodes in the flows directory don't profile the current server but connect to an existing Elastic data instance.
Some of the other threads on Elastic/APM are from at least a year ago so I'm hoping someone might have some updates in that time. My plan is to use hooks to filter only events I'm interested in (Messaging Hooks : Node-RED), but I'm unsure of the best way to measure the metrics I'm after so I have some questions:
Is there something like v8-cpu-profiler that can also collect memory usage?
Is it possible to use node-red-contrib-process-resources to monitor specific nodes, how?
Have you got any experience of using Elastic APM or OpenTelemetry in Node? Do you have any examples or simple resources you can point me to for getting set up? Bonus points if you've set up manual instrumentation in NodeRED before.
Are there any other packages I've missed that can do sub-second memory and CPU traces?
That is a bunch of (good) questions. I am afraid I won't have a ready to go answer. Otherwise I would have implemented it myself already as a Node-RED contribution...
But I will try to give as much feedback as possible, and hopefully it can be of any help to you:
I use Dynatrace a lot for my daily job, and that is a very amazing tool. But very expensive. I know that they also allow deep monitoring of NodeJs, but I have never used that.
My ClinicJs discussions were trial and error. I have written down at the time being my experiments on the Node-RED design repo (see here). That is a bit more structured, but unfortunately not a ready-to-implement analysis...
Currently ClinicJs also has a heap analyzer. The output is a flame chart. I don't think I have ever had a look whether it is possible to calculate the sum of all the memory consumed by a specific node. So not sure whether that would be possible. Don't know this. But of course a flame chart is also very interesting to analyze memory usage.
My node-red-contrib-v8-cpu-profiler indeed only creates a cpu profile, via the v8-profiler-next library. As you can see on the readme page of that library, it also supports heap snapshots, but would be a bit weird to integrate this into my node that is named "cpu-profiler"... Don't know if my node-red-contrib-heap-dump still works after all that time? Of course those dumps are only snapshots at a certain moment in time, not continious monitoring (like with Dyntrace). But they can help a lot to troubleshoot memory issues.
My node-red-contrib-process-resources node shows information per process. When a process has multiple threads, you can't even split the information per thread. When you want to have the information per function or node, that would even be one more step deeper. Which means not possible with my node. Dynatrace offers this kind of stuff, called "method hotspots".
You could also try my node-red-contrib-inspector node to activate remote debugging. As you can see in the wiki tutorial, you could use e.g. Chrome developer tools to start a remote debugging session. Then you can start a memory profile from Chrome:
Since you had already done your homework before you started this discussion, I am not sure whether my above explanation offers much new knowledge to you ...
@jackcarey,
I had a quick look at the code of the ClinicJs heap profiler, and it might be that I found something interesting. I need to investigate in depth to make sure it is not a dead end...
Awesome, thanks Bart! I'll take a look through your links now and digest them. I tried using your node-red-contrib-process-resources, set to poll as frequently as possible while my other nodes are executing. This way I can output everything to a JSON file to analyze at least my whole NodeJS instance. Next will be using v8-cpu-profiler alongside heap-dump.
I think Elastic/OpenTelemetry could ultimately be the way forward for individual function/node tracing, but their set-up doesn't seem to fit with the straightforward, low-code ethos of NodeRED!
You can place the node in your flow without any other connections and it will export quite a few stats including cpu and memory.
If you don't use Prometheus, you can still use http to grab the metrics and translate them to whatever format is useful. Or you can use the source as a reference on how to access the stats.
We are using node-red inside Kubernetes, so the prometheus exporter works quite well. And there are some prebuilt Grafana dashboards for node.js that can be used.
@jackcarey,
Unfortunately a dead end, since I have no free time anymore lately to digg further into stuff like this...
To implement memory profiling in Node-RED, you will need somebody with both a lot of time and a lot of Node-RED core knowledge. Otherwise it will never happen...
Anyway I will explain shortly what I tried. If somebody pops up to help, then I might spend some other time on this. And otherwise you could use it to start troubleshooting your memory usage manually:
When you connect with the Chrome debugger to your NodeJs instance (see explanation) you can start a memory profiling.
The first option creates a heap snapshot, which means Node-RED becomes completely unresponsive. Which is not what we want of course... Tools like Dynatrace offers continious profiling, while your application keeps working. You won't even notice it. But they accomplish that by injecting (at application startup) a shared library into the application code, which injects code into your application to deep monitoring. That way they get a huge amount of information out of your application. However that is not possible for us as you can understand. So the only option we have is to start (for some time) a heap sampling profile, which would result in about 10% extra performance loss. But at least you can profile live while Node-RED keeps running, and you could optionally repeat the steps that you suspect to cause the performance troubles:
As soon as you click the "Start" button, the V8 engine (which runs below NodeJs) will start collecting information about which functions are using which amounts of memory. Since I knew which flow was causing my memory issues, I had disabled all other flows in the flow editor.
I started my test, which is reading images from my ip camera (via an mjpeg stream) and execute object detection on those images:
After had executed my test, I clicked the "Stop" button.
When you select the recorded profile (in the left sidebar), you will see the flame chart on a timeline. On the timeline you can see where the most memory is being used:
You can manually zoom in, by selecting this area with your mouse in the timeline window.
However when you don't zoom in (i.e. you watch the entire timeline), you will see the memory being used by each function across my entire test. This way you see immediately that most of the flames represent the nodes in my object detection flow:
Wide bars mean more memory. The upper bars are Node-RED stuff (e.g. processImmediate), i.e. stuff from your core which you can ignore. Some ot the flames can also be ignored, e.g. the Express.js routing.
I was quite excited yesterday because when hoovering one of the bars, I saw the node id of my function node:
This kind of information is like gold for us . Because based on that node id you can relate memory to a specific node instance.
BTW the node id is set in the function node with this code snippet:
function createVMOpt(node, kind) {
var opt = {
filename: 'Function node'+kind+':'+node.id+(node.name?' ['+node.name+']':''), // filename for stack traces
displayErrors: true
};
return opt;
}
However my enthousiasm was over very quickly, because the node id's of all other nodes is not available in the profile file . For example for the object detection node:
Based on the file name you could determine automatically the amount of memory per node. But if you have N instances of that node in your flow, then you cannot determine which of those is consuming most. So it might be a help for troubleshooting, but it cannot be used to automatically show for each node how much memory it is using.
Summarized I see a number of issues:
Don't know if it is possible to pass the node id somehow to the profiler, by adding some code to the Node-RED core. Don't have time to investigate this now further.
Don't know if we can solve all limitations we had listed in the past. If somebody wants to create a test node which implements those use cases and test it manually with the chrome profiler, that could be a huge help for the community.
The same node will popup in the flame chart multiple times. For example the function node "persons detected" is shown twice in my screenshot above. Those need to be aggregated. Or perhaps they need to be kept separate, to drill down. Don't know. Needs to be investigated.
BTW I quickly implented an experimental node to start/stop a sampling memory profile:
I won't add it to Github, because I copied the code from the ClinicJs guys to parse the heap profile file. If somebody ever wants to continue with this, the code needs to be rewritten. Or at least ask the ClinicJs guys how you can reuse their code without violating their copyright!
This experimental node (which contains a hardcoded path /home/pi/.node-red for the heap profile file) does not send anything yet on its output, but it prints to the console all memory being used by functions with "node:" in their name. So it showed me this for the function node:
I'm no expert at all here. But in developing uibuilder, I wanted to know whether changes I made were going to impact memory utilisation.
So I added a very simplistic V8 heap dump. Once in settings.js to get a baseline then at various points in the node's code, I output the same again. Obviously that was a very rough guide but it gave an indication anyway. You could obviously also use this in your flows.
In case it is of use, here is the code:
/** Optionally display the Node.js/v8 engine heap size on startup */
const v8 = require('v8')
console.info(`V8 Total Heap Size: ${(v8.getHeapStatistics().total_available_size / 1024 / 1024).toFixed(2)} MB`)
let mem = process.memoryUsage()
const formatMem = (m) => ( m/1048576 ).toFixed(2)
console.info(`Initial Memory Use (MB): RSS=${formatMem(mem.rss)}. Heap: Used=${formatMem(mem.heapUsed)}, Tot=${formatMem(mem.heapTotal)}. Ext C++=${formatMem(mem.external)}`)
I managed to get OpenTelemetry integrated using the links below and I've got something of an output in Jaeger, however, the examples are only monitoring HTTP requests, so I need to find an instrumentation that can take CPU and memory measurements. I'll update this post once I have more information.
I think Bart's excellent write-up will be the way to go if I cannot find the instrumentation.
It looks like Prometheus is the way to go for getting metrics out using OpenTelemetry, so thanks for the recommendation for this node! It's saved me from manually adding the performance metrics. I don't suppose you know of a way to export the Prometheus data to something like a CSV, spreadsheet, or JSON file?
actually should be pretty easy (famous last words :-). Just create a flow that periodically does a get on the exporter endpoint and save the data to a csv... your flow can filter for only the metrics you are interested in.
# HELP process_cpu_user_seconds_total Total user CPU time spent in seconds.
# TYPE process_cpu_user_seconds_total counter
process_cpu_user_seconds_total 49107.20739600051
# HELP process_cpu_system_seconds_total Total system CPU time spent in seconds.
# TYPE process_cpu_system_seconds_total counter
process_cpu_system_seconds_total 8032.47916700002
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 57139.68656300017
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1666026803
[](http://voice.google.com/calls?a=nc,%2B11666026803)
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 259461120
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 638853120
# HELP process_heap_bytes Process heap size in bytes.
# TYPE process_heap_bytes gauge
process_heap_bytes 387731456
I tried using the Prometheus server with the web GUI to poll the NodeRed /metrics endpoint every couple of milliseconds. Not an efficient approach admittedly, but it looks like it'll give me enough resolution for what I need. When my testing is complete and Prometheus has all of the data I think I'll be able to create a flow that replicates a Python script I've seen (Export data from Prometheus to CSV | by Aneesh Puttur | Medium) to retrieve the data in a more digestible JSON form rather than the original TSDB.
I've taken a look at Grafana and I am struggling with it, to be honest. I can plot some data, but I am unsure how I could track the "execution windows" of the blocks I'm interested in and plot stats for those. I made a post on the Grafana forum to this effect:
I'm using Messaging Hooks : Node-RED to log the start and end times of each node execution I'm interested in and the idea is that I can group CPU and memory metrics based on each start time and the next nearest end time. Such that from that I can pull out stats from each grouping or from each type of node I am monitoring. Does that make sense? Do you know of any example dashboards or methods I can look at to create these execution windows? Thanks
. Because based on that node id you can relate memory to a specific node instance.
. For example for the object detection node:
