I saw tonight a discussion about CPU usage being 100% at irregular moments, which makes it very hard to figure out the root cause. Because:
Since you don't know when it happens you cannot simply start a profiling, because you cannot let the profiling run for e.g. 3 days.
Since we don't have continious profiling, like professional tools (e.g. Dynatrace) offer.
So I thought: suppose we could start/stop cpu profiling from within a custom node. Then you can start/stop cpu profiling :
Manually by injecting control messages.
Automatically e.g. when the measured cpu (e.g. via my node-red-contrib-cpu node). This way you could start the profiling automatically as soon as cpu goes to 100%, doesn't matter when that happens ...
This node allows you to generate a .cpuprofile file, which can be visualized via third party tools as a flame graph (e.g. via Chrome Developer tools as explained in my wiki).
Since I have not spend much time on this node, most probably I might have forgotten some things. So it would be nice if some users could test it, before I publish it on npm. You can install it directly from my Github repository (from within your .node-red) folder:
Yes a single node is enough, since it connects to the Google V8 Javascript engine (that runs your NodeJs server). So it will profile ALL the Javascript code that is being executed in your NodeJs instance.
On my readme page you can find an example flow, which shows how to start/stop the profiling via an Inject node manually. However I assume you don't know when cpu will be getting high, so you cannot start the profiling manually? In that case you could e.g. use my node-red-contrib-cpu node to measure the cpu continiously. And you could feed the cpu usage value into the node-red-contrib-value-for node, to determine that the cpu is e.g. between 95% and 100% for N minutes. If that is the case you can (once!!) start the profiling for N seconds (as you can configure on my node's config screen). The profile should be long enough to get good statistical information.
When you import my example flow, you will see that the last node in the flow is a FileWrite node. In my case the profile json is stored in my temp folder:
Of course you can choose whatever folder you like...
Yes indeed it is not very easy to start with interpreting flame graphs. I have added a few tips on my wiki, but I am not an expert in this.
If you find anything useful, please share it so I can add it to the wiki...
Or if you have any doubts, just share it here also!
Based on the feedback from @IAmKonni, I have completely rewritten my wiki tutorial. It now explains a step-by-step about how to find the root cause of high CPU usage in a Node-RED flow.
When a profile is complete, it will be send on a first output. But when a profile is interrupted prematurely, it will be send on a second output.
The output message now contains both the startTimestamp and endTimestamp.
These changes allowed me to write a new tutorial on the wiki about "Create a profile for high CPU usage at unpredictable times". Summarized: when the CPU usage is above 90% for 5 seconds, then it tries over and over again to record a CPU profile of 10 seconds long. That way users should be capable of profiling peaks that they cannot profile manually.
I am having ultra high load permanently and woanted to check it out with your node.
Nodered runs under Docker, which itself is under Ubuntu VM (Proxmox).
I found out, that the high load came from some MQTT node, that could not reach the Broker.
Still have no clue, why that would produce such shift load though.
Hi @jjm9,
I have no idea about that.
Had recently another issue (totally not related to yours) with the v8-profiler-next library that this node uses. They told me that only LTS versions of NodeJs were supported. Not sure which version of NodeJs you use?
Bart
System runs in a docker container under Ubuntu VM in Proxmox.
So I am not actually willing to mess with bringing NodJS to another version than supplied by the actual docker image.
Michael
Hey @jjm9,
I see another recent issue that is a bit similar to yours, but not exactly 100% the same:
Error loading shared library ld-linux-x86-64.so.2: No such file or directory
That issue should be fixed in v8-profiler-next version 1.9.0 which has been published on npm two weeks ago.
Could you please:
Check which version of that library has been installed in your containers? I would expect that the 1.9.0 version is installed automatically (as a dependency of my node). If you still have an older version, I will need to update the package.json file of my node (so you could test afterwards whether your issue is also fixed in that version).
If you have already the 1.9.0 version, then that fix clearly doesn't solve your problem. Could you then register an issue in the Github repo of v8-profiler-next? The author is very constructive, so hopefully he can assist you further with this. Once he has fixed it, other node-red-contrib-v8-cpu-profiler users can gain from it...
Hey @BartButenaers I'd like to check the version, but I am not exactly sure how to do that.
In the v8-profiler-next folder, I couldn't find out anything about it.
@jjm9,
On my Windows portable I check that like this:
This way you see which version of that library is being installed and used by my node.
Or if you have Node-RED running in a Docker container, you can attach an interactive bash terminal to that container (where you can execute your commands):
