Merge / Diff on the Node-RED flow is very slow

Core Development
1

Hello,

We are working on various ChatBot project using Node-Red to design conversation.
But "team work" on v0.19 is really painfull when making diff on the flow.
In a nutshell it takes about 2 minutes to make the diff

  • The file has 32000 lines
  • For about 2595 nodes (with 679 nodes function)
  • Everything is split in various tabs or subflow (30-40)
  • There is almost 60-80 nodes / tabs it's readable for Node-RED it's not an "over-usage"

Any ideas to speed up the GIT Diff ?
Thanks !

1 Like
2

Hi, saw you asked this on stack overflow as well.

You have 2595 nodes that need diffing. That is quite a lot and far more than any of my testing uses. I'm not surprised the diff is slow.

I'm not sure what sort of suggestions you're looking for. If you want to know 'as a user, how can you speed up the diff' then the answer is simply to use fewer nodes.

The more meaningful conversation will be around how we can improve the efficiency of the diff code to handle larger flows. If that's something you're interested in contributing to, then you'd be welcome.

I'm largely offline until next week, so any meaningful conversation will have to wait until then.

3

Thanks Nick (sorry for the ping) !

Well I thought it was too much nodes but in fact the UI respond very well and the view is very clean. We only have 60 nodes per tabs, not so much tabs.

On the bot perspective, other "big" bots have the same amount of nodes (I think we could optimize but not so much)

On the GIT perspective, I will do more test but I assume the GIT won't take 2 minutes. My guess is more on the UI displaying the diff.

  • May be the UI diff can be performed only on a given tab ?
  • Or may be the UI can only display conflict, a lighter way ?
  • (If it comes from the git diff) may be we could split the flow by tab ? (a little bit kludge)
4

Some brief comments as I'm not really online this week.

It does not use git diff.
It does have to examine every tab to see if nodes have been moved between tabs.
There's a lot of looping in the code ... Im sure there will be plenty of optimisations that could be made. No one other than me has worked on that code... and I've not spent any time optimizing it.

It just needs some focused work to understand where the time is being spent.

5

In the past, I too have toyed with finding some optimizations for this code... I've also been interested in learning how to use the test-helper node/functions. So maybe this is a good time to kill both birds -- would anybody care to put together a test case with big enough sets of test data so we could measure statistically significant increases in performance?

I would be willing to work on profiling the code, as long as I don't have to build the test data.

6

Thought you had dashboard tabs to hide first :wink:

7

That is true... just trying to queue up the next piece of work I can start and then fail to finish ;*(

But thx for the prod -- I'll get back on the hidden tabs tonight...

8

Hello,

I tested with a flow of 200 nodes and there was a noticeable delay before the diff would appear.
The diff is actually generated fast, the DOM generation on the other hand takes some time.

I noticed the DOM was created for all the properties of all the nodes in the flow upon diff opening.
What do you think of only generating the first level on panel opening (flow / configuration nodes / sub-flows) and generating the sub-levels for toggled nodes only ?

Thus, the DOM would be generated only for nodes that the user explicitly wishes to see...

1 Like
9

Hello again,

So I just tested moving the DOM generation for the diff of the node properties to the row click handler.

I have a flow with 3300 nodes and it used to take 5 minutes to display the diff. Now it only takes 3 seconds.
Unfortunately, I cannot share this flow with you as it is proprietary.

Should I submit a pull request and an issue as well ? I couldn't tell from your Github guidelines as it is a kind of feature change request rather than a bug.

Thank's !

1 Like
10

Thanks @TothiViseo

A pull request on the dev branch would be welcome. You'll find the files have moved around alot between master and dev, but the content of the files you will have edited for this will not have changed much so it shouldnt be too much work to retarget the dev branch.

11

Thank's!
For those who might be interested, the pull request is here : https://github.com/node-red/node-red/pull/1989

1 Like