Analysis and optimization solutions for the high CPU usage of Node red?

General
1

If the length of the transmitted msg is large, it will significantly increase the CPU usage of the system.

What optimization solutions are available when the payload length cannot be reduced?

I just noticed this phenomenon, but I'm not sure why the length of the msg leads to an increase in CPU usage. What is the reason?

2

what do you mean by "transited"? To another node? to an external thing (via MQTT? HTTP? TCP/UDP? PLC?)

Internally, messages are passed by reference when passed out to a single wire and should be near instantaneous. If you have multiple wires coming out of a node, then it is cloned for the 2nd, 3rd, nth wire - this was a design decision when V1.0.0 was developed.

Another reason for slowdown increasing with larger payload might be the node receiving data does some heavy processing - for example, the CSV node in earlier versions of Node-RED was notoriously slow. Visibly slow when processing large CSV data.


Can you give us an idea of what "large" means? rough size of the message data?

An example flow would be useful too.

4

Please wait a moment, I will provide a detailed description

5

For example, msg is passed across multiple adjacent function nodes,
It is not sent out through communication nodes like mqtt

6

So you have multiple wires coming out of a node going to multiple functions.

What are you doing in these functions - are you looping data?

Also, please answer other questions

7

I'm very sorry, it's difficult for me to capture an example of a certain section of the flowchart because it's too large.

I have calculated that in my process, the length of the MSG is 66.0 KB. My process carries such a large MSG, which flows through multiple function nodes, switch nodes, filter nodes, etc.

Finally, it is split through the split node and sent out using mqtt.

8

Can you provide a small sample flow, preferably with a small sample of data (you can provide dummy data via an inject node or template node)

9

Okay, I'll give it a try

10

I tried it, but as you can see, my flowchart is very complex and there are three similar flowcharts running simultaneously. Even with the use of pseudo data, it is still difficult to extract an example flowchart,

But I can describe the purpose of my program, which is to collect data from a remote device through the modbus protocol, classify the data (such as physical quantities, alarm quantities, control variables, etc.), and then send it to the cloud platform in batches using mqtt;

In the flowchart, I used quite a few loops because I had to traverse each collected data and classify it into different arrays for storage.

At the same time, there is indeed a situation where multiple wires are led out from one node.

12

Unfortunately, that picture is unclear and I cannot read your language (sorry).

Are you doing manual data conversions and data splitting of your modbus data?

Are you doing multiple small modbus reads?

You might benefit from reading this article: Modernize your legacy industrial data. Part 2. • FlowFuse

13

You have a LOT of nodes that have two wires, one of which is a debug. Try removing a load of them and see if that helps. Normally, such detailed debug nodes are useful when developing a specific section of your flow but should then be removed to avoid the processing that Steve has mentioned where multiple wires coming out of the same output port force Node-RED to make a deep copy of the msg data and this is an expensive process.

14

Do not assume that the problem is due to the large message size, it may well be due to what you are doing with the processing of the message in the nodes where you process it.

How many messages per second is your system handling?

15

No, I read over a hundred bytes at once and slowly parse them in subsequent nodes.
Thank you very much for your suggestion, it has inspired me a lot

16

Your suggestion is really enlightening to me. I am trying to remove unnecessary debugging nodes

17

I trigger an inject every 7 seconds, and each trigger results in 12 rounds of data collection, parsing, and reporting.

The process by which I became skeptical about the size of the msg is as follows:

My original msg was even larger, possibly more than twice the size of 66KB. At this time, my CPU usage was almost fixed at 110% (my system has a 4-core processor), and all node were running very slowly, taking almost 40 seconds to report data.

Later, I accidentally delete a large portion of unused fields during the transfer process and found that the CPU's duration of maintaining 110% was shortened, with most of the time between 50% and 80%. At the same time, the entire flowchart can be completed within 5 seconds.

18

During the process of running the flowchart, there were also countless debug nodes and mqtt sending nodes (used for debugging)

19

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.