NR flows update very slow when >> 500 flows

General
1

On a test Node-RED instance we face very slow update-time when we want to update a single flow (we tried /flows and /flow/:id APIs). The deployment of a single flow takes up to 20-30 seconds.

We currently have roughly 700 flows deployed (just a few nodes each). Is this expected? Can this somehow be improved?

2

Have you tried selecting modified nodes or flows from deply dropdown.

Screenshot_20240503-135301
Screenshot_20240503-1353011920×1080 160 KB

Then when you press deploy only modified nodes or flows are deployed.
Or have i miss read your post and you want to use the admin API?

3

Yes, we hide all flows but one (which one is provide during startup of Node-RED UI) and deploy this single flow only (in order to achieve this we slightly customized NR).

4

Hidding flow tabs does not affect what is posted.

Can you answer explicitly if you are using the "modified nodes" option as shown in @E1cid screen shot (the deploy button "modified nodes" option)

5

We allow only "Modified Flow". When I check the API call I see:

method: POST
url: /flows
headers: Node-Red-Api-Version: v2; Node-Red-Deployment-Single-Flow: [UUID]; Node-Red-Deployment-Type: flows

6

And what if you use "Modified Nodes" mode?

image
image641×564 44.2 KB

Is that 700 tabs with a few nodes on each?

Are you using subflows? And if yes:

  • when you make your changes, are you making changes to subflow internals or regular flows?
  • how many subflows do you have
  • how many instances of subflows do you have?
  • typically, how many nodes inside your subflows?

Has the slow down only just started happening (i.e. did you recently add flows or install a contrib node)?

7

And what if you use "Modified Nodes" mode?

This option we removed (I would have to test it... can I do it via API)?

Is that 700 tabs with a few nodes on each?

Yes (would be 700 tabs but we hide all but one during a session).

are you making changes to subflow internals or regular flows

Regular Flows. We don't use subflows.

Has the slow down only just started happening (i.e. did you recently add flows or install a contrib node)?

The more flows the slower. We just noticed it recently when we created (cloned) flows for testing purposes.

8

Again, that makes zero difference.

Modified flows reinitialises "all connected nodes" (those that connect via a wire).

The deploy oprtation ALWAYS sends FULL FLOW JSON, its the bit under the hood that changes:

  • Full Deploy destroys and recreates everything (connections are terminated, instances are destroyed, then all recreated)
  • "Modified Nodes" destroys recreates everything connected to a modified node (by a regular wire, not a dotted link wire)
  • "Modified Nodes" destroys recreates only nodes that were changed (and that must be a property change. x/y position doesnt count)

Yes, change "Deployment-Type" header as per the docs

Why do you have 700 flows with only a small amount of nodes on each tab? That is a LOT of overhead for little reason as far as I can see. Perhaps it is time to scale horizontally?

9

Thanks Steve!

Again, that makes zero difference.

I understand, I just wanted to explain why we don't care about the 700 (which might seem a lot).

Modified flows reinitialises "all connected nodes" (those that connect via a wire).

As the flow is always a single flow this part would be fine I guess... however, we also noticed that always the full json is posted.

Why do you have 700 flows with only a small amount of nodes on each tab?

Because a user is only interested in one or a few flows out of the 700... we only want to show one flow at a time...

That is a LOT of overhead for little reason as far as I can see. Perhaps it is time to scale horizontally?

How would you solve it with less overhead? I don't get this... Regarding scaling horizontally... you mean to say we should use more than one Node-RED instance? Actually I believe this is coming up soon, yes. However, as you might imagine this than rather complex (auto scale-out and scale-in would be expected...).

10

You have not mentioned what hardware you are throwing at this - which could make a difference

I also recall someone else recently had a massive number of flows and was having problems with deployments - ended up being some orphaned config nodes that were point to incorrect/obsolete external servers if i remember correctly

Have you checked your config nodes to make sure they are all valid ?

Craig

11

I had a same "sluggish" behavior of Node-Red during development tasks .. I had a lot of flows (less 200) and a lot of global and local vairables.

I'm running Docker Node-Red and have laptop for development using Chrome. The Node-Red on docker is barely using any resources (~10-20 on average, 50% peak)

I just upgraded my 8GB laptop to 16gb memory and now everything is smooth again :wink:

12

We run on AWS ECS Fargate: 1vCPU und 30GB RAM.

Config Nodes are all valid. Thanks for the hint though.

13

If I check CPU and RAM consumption during design time it's very low.

14

WHat about I/O CPU and RAM during Deploy - i.e. where is the bottleneck as far as AWS is concerned ?

Managing 700 flows is a bit of a nightmare - i would definitely look at scaling horizontally with more instances and divide the flows logically amongst them

With 700 flows there could be any manner of crap hanging around in one of the flows and you would have no idea without stepping through each of them one at a time, remove the flow and try a deploy, rinse and repeat

I assume the slowdown has just happened recently or has it been a gradual thing ?

Craig