[ANNOUNCE] node-red-contrib-process-resources: beta

Hi folks,

Our friend @kevinGodell has developed the "node-red-contrib-ffmpeg-spawn" node, that I want to start using soon to distribute the high intensive video surveillance work across the cores of my processor.(s)

However when dealing with video, the slightest thing can entirely ruin your performance. So when I use Kevin's node to create child processes, I would like to determine easily:

  1. How many child processes are currently active.
  2. How many CPU usage each of those child processes has.
  3. How many memory usage each of those child processes has.

To assist me with that, the node-red-contrib-process-resources node was born :champagne: :clinking_glasses: :partying_face:

By using the CPU usage simulation flow on my readme page, I can play with the;

  • CPU usage in the main process
  • number of worker threads inside the main process
  • number of child processes

Which results in something like this:

To test this node, you can install it directly from my Github repository (from within your .node-red folder):

npm install bartbutenaers/node-red-contrib-process-resources

As always, all "constructive" feedback is very welcome!

Bart

P.S. This will be my last performance related experiment. Now finally back to video surveillance...

5 Likes

It works. At the very least, it accurately showed 6 processes running under my node-red instance.
(1 main, 3 ffmpeg, 2 top).

And the other pi has a bit more processes, but also accurate.

2 Likes

A bit = 61 in the USA? :wink:
Thanks for testing!!!!

1 Like

It's not as accurate as the metric system, but seems to work ok. It's mostly ffmpeg consuming ip camera stuff.

I wonder what would happen if i unreferenced one of the ffmpeg processes without killing it. Would it still be tracked as a child process? I have had some stubborn ffmpeg process that refused to exit gracefully and have been found to be hanging around, (previous to me adding a kill timer that runs 2 seconds after the process was allowed to exit gracefully and runs a higher level kill command). Will have to test that.

Hey Kevin,
can you please explain a bit more in detail how you have setup that metric system?

That is something I also have been wondering yesterday. I assume it isn't monitored anymore, because this node only monitors Node-RED related package. If you would have some time to test it, then I will put your feedback on my readme page...

no comment. :blush:

I will setup some code to test that this weekend.

Currently, when ffmpeg-spawn node spawns or closes the process, it outputs a message as:

{
  "topic":"ffmpeg_spawn/ef95966b9bd7463b/status",
  "payload": {
    "status": "spawn",
    "pid":29950
  },
  "_msgid":"f54b72ebf51785ef"
}

It would be interesting if somehow extra special attention could be given to a pid that has been passed into your node. This would allow me to get rid of all of those running top commands that monitor cpu and memory of each process(currently in a non-cross-platform way).

edit:
forget that last request. I could just save the output of your node to another node and then later access the pid specific info saved there. no need to over complicate your node.

1 Like

Like our Scrum coach at works always says: "Good enough for now, good enough to go..." :wink:
Version 1.0.0 is available:

image

BTW perhaps this might not be very clear from my description above: my node-red-contrib-cpu node can be used to monitor the CPU of all the cores of a processor, while the node-red-contrib-process-resources can be used to monitor the cpu/memory of all Node-RED related processes on the OS.

Final testing shows that if I un-reference a spawned process by nulling or deleting any vars pointing to it, it is still tracked as a running process by your node. I can see that being valuable, especially in a situation where you may have a specific number of ffmpeg processes running and see that your node is reporting a different number. That would indicate something did not exit properly and may be wasting resources.

Hi Kevin,

Thanks for testing !!! Will need to put that on the readme page. Very useful information.

I also will try to find some time to add some extra features:

  1. The pidtree library - which I am using - will also report sub-children results. I think my node will only show the first level depth children, so you can overlook heavy consumer processes. Will need to add that, so my output message contains a tree of process resources.
  2. Add a checkbox to flatten the tree into a long array, which might be usefull to simplify showing all subchildren into a chart.
  3. Currently you can only see the Node-RED related processes. I will add a dropdown with following options:
    • Node-RED processes (of 'this' Node-RED instance, because you can have multiple Node-RED instances running on a host).
    • All processes on the system
    • Single process (pid)

When that would be ready, I think we have covered most of the requirements I think...

1 Like

Tested it and seems to work... but I miss a nice example to viel the output file cause I am lazy. :slight_smile:

Can you explain what you mean with "viel"? Even Google translate doesn't offer me any help with that word...

Then I strongly advise you to buy a commerical system with a good support contract :wink:

1 Like

"view" ... I mean view :slight_smile:
You added a nice snapshot where the output file is viewed in a nice chart(y) way. why not putting it to the examples ? :blush:

EDIT: I am a poor german... we have to feed half of Africa because out government wants it. I can not afford that expensive things you are talking about.

If you click on the link in my post above, you will see the readme page of the node. There all the flows are documented. Those flows can also be imported easily via the Node-RED "Import" menu:

image

With such a collection of example flows, I hope you feel like a rich man for at least a few minutes :wink:

1 Like

Good bless Netherlands ! I never noticed that button. Now I can die rich and in peace. Thx !

1 Like

Are you referring to me? I am from Belgium... Will forgive this you once :wink:

2 Likes

Belgium... that s what I wanted to say. :slight_smile:

2 Likes