How to track dependencies between nodes

[tve]

One of the things that keeps nagging me in the implementation of FlexDash is the tracking of dependencies between nodes. The std dashboard has exactly the same issue and I'm running into this elsewhere too. I fear there is no good answer right now, but I'm wondering whether there's any energy to try and find a clean solution.

In FlexDash, widgets depend on the container (e.g. panel, grid, tab) in which they're contained. The dependency itself is handled by Node-RED in that a node can depend on a config node. However, FlexDash also needs to track the order of dependents because that's the order in which they get placed on the screen. This tracking of dependency meta-data is not supported by Node-RED.

The std dashboard has the exact same issue. It encodes the order in the dependent as an index. I tried to understand the code that maintains those indexes and failed. FlexDash instead encodes the order as an array of node IDs in the container. It's not pretty code either.

The issues that I believe both dashboards run into have to do with deleting nodes (disabling nodes further muddies the water). In addition, there are issues around which nodes get affected by a change and around flow editor vs. runtime.

Focusing on the flow editor, some of the primitives I was missing when I implemented the FlexDash code:

• given a NodeID how to determine its state: active, disabled, deleted-but-undoable, deleted/non-existent
• getting a callback when a node transitions between the above states, or alternatively when the flow gets externalized (save/export)
• possibly getting a callback when all flows/nodes have been loaded so one can determine the state of all nodes to which one holds a reference

Let me try an example to illustrate the challenges:

• suppose you have nodes A, B, C, D in that order in container X and the user deletes node C
• the node ordering should now be changed to A, B, D
• the first question is when and how does the relevant code get triggered?
• the second question is what are the side-effects?
• in FlexDash the deletion changes node X where the array of dependents lives and thus dirties X
• in ui_base the index of D would have to be changed and thus dirties D
• both cause undesired effects on seemingly unrelated nodes that suddenly appear changed in the editor and need to be deployed

In terms of editor/runtime split, the above issue causes double fun: if C cannot be cleanly removed in the editor before deploy then when the runtime gets the flow the ordering is inconsistent/denormalized and the runtime code has to implement its own work-around. In the case of FlexDash where the config can also be edited live in the dashboard the issue propagates there too. All this can be worked around but it creates opportunities for fragile and inconsistent code. It would be much better if one could guarantee in the editor that what the runtime sees is normalized.

As I'm writing this, I'm seeing some solutions where I didn't pick the optimal option when I implemented the bulk of node-red-flexdash a couple of months back. If I remove/disable all live editing in FlexDash and if I had the following two primitives I could clean up a lot of code:

• RED.node.getState(node_id) -> active | disabled | deleted (dunno how undo factors in here)
• a way to register a callback for a node that gets invoked before the node gets externalized so it can modify the config, i.e. callback(config) -> config_to_output; this can normalize the config for the run-time (I believe I've seen code that does exactly this for internal purposes)
• a way to get a callback when any node is deleted (again, dunno how undo factors into this)

What these primitives would enable is:

• know when the ordering of dependents in the current config can be safely changed to remove deleted nodes
• normalize the config for the run-time and remove code from the run-time to deal with seemingly missing nodes

NB: it would also be great to have a clear description of how undo works 'cause it does factor in here and I'm pretty sure the FlexDash nodes are not doing the right thing currently.

[TotallyInformation]

@tve

given a NodeID how to determine its state: active, disabled, deleted-but-undoable, deleted/non-existent

You can monitor node changes using RED.events.on with the 3 events: nodes:add, nodes:change, nodes:remove. Here is an example from uibuilder:

    RED.events.on('nodes:add', function(node) {
        if ( node.type === 'uibuilder') {
            // Keep a list of uib nodes in the editor
            // may be different to the deployed list
            editorInstances[node.id] = node.url
            // -- IF uibuilderInstances <> editorInstances THEN there are undeployed instances. --
        }
    })

I needed to monitor these because I needed to prevent people from using duplicate URL's but originally could only check deployed nodes and I needed to account for where people added multiple uibuilder nodes before deploying.

[BartButenaers]

@tve

it would also be great to have a clear description of how undo works

Unfortunately I have never had a look at that.
In case nobody jumps in to explain it, you could have a look at this pull-request:

Add 'Redo (un-undo)' function

Don't know the Discourse nickname of k-toumara, so cannot mention him.
He is one of the members of the Node-RED team at Hitachi.

[Paul-Reed]

@ktoumura hasn't been around much lately, but he may respond.

[tve]

@TotallyInformation

You can monitor node changes using RED.events.on with the 3 events: nodes:add, nodes:change, nodes:remove. Here is an example from uibuilder:

Good point. What are the semantics of nodes:remove?

Specifically, if I get that callback am I guaranteed that I will never see its node ID again (except for random collision)? If not, then how does it help?

Also, if someone imports a flow or opens a flow and it contains a Node ID and RED.nodes.node(id) returns null, what can I conclude about it?

[TotallyInformation]

@tve

Good point. What are the semantics of nodes:remove?

    RED.events.on('nodes:remove', function(node) {
        if ( node.type === 'uibuilder') {
            mylog('>> nodes:remove >>', node)
            delete editorInstances[node.id]
        }
    })

@tve

Specifically, if I get that callback am I guaranteed that I will never see its node ID again (except for random collision)? If not, then how does it help?

I think the id's are UUID's? So they should be pretty unique, at least within a specific flow file. I don't know for sure though.

@tve

Also, if someone imports a flow or opens a flow and it contains a Node ID and RED.nodes.node(id) returns null, what can I conclude about it?

When you import, if there is a clash, I think that node-red sorts out the ID's. In which situations would Node IDs change ? Import or any other as well? - General - Node-RED Forum (nodered.org)

Sounds like some testing is in order!

RED.events : Node-RED (nodered.org)

[tve]

Mhh, we're not speaking the same language...

If I get a callback for nodes:remove can the user hit undo and the node reappears? or is it truly gone?

And WRT import/open I wasn't asking about clashes. As I mentioned, I keep an array of dependent nodes in FlexDash container nodes. NR doesn't do anything about them. But my code needs to decide whether they should stay or should go, so I need to query something. (The std dashboard doesn't have this specific difficulty, but it has other very related difficulties.)

[TotallyInformation]

@tve

If I get a callback for nodes:remove can the user hit undo and the node reappears? or is it truly gone?

It isn't gone completely until the user completes a re-deploy. After that, you can't do an undo.

@tve

And WRT import/open I wasn't asking about clashes.

It is surely a different aspect of the same issue. Whether something is allowed. We both have to keep our own list of dependencies. URL's in my case, FlexDash container nodes in yours.

Ultimately, there are two potential sets of Node-RED nodes. Those that have been deployed and those that are in the Editor not yet deployed. Of course, there is an overlap between them.

I keep track of all uibuilder nodes whether deployed or not. I also track whether a node is deployed (since deleting one of those also requires me to offer to delete the physical folder that goes with it). I use onEditDelete to firstly check deployed instances (via an API call) and then if a node is deployed, offer to delete the folder (another API call).

[BartButenaers]

@tve

am I guaranteed that I will never see its node ID again (except for random collision)

Another good question. Don't have the answer unfortunately, but perhaps you could debug quickly the RED.nodes.id() function, which is responsible in the flow editor frontend for generating id's for new nodes (see here an example in my xterm node). Perhaps you can see in the code if it is random enough for your purpose.
My lunch break is over now...

[Colin]

@TotallyInformation

It isn't gone completely until the user completes a re-deploy. After that, you can't do an undo.

I don't think that is correct. Try adding an Inject node to the flow, Deploy changed nodes, and Undo. It comes back again.

[knolleary]

@tve

I get a callback for nodes:remove can the user hit undo and the node reappears? or is it truly gone?

If the user hits undo you will get a nodes:add event for node as it has been readded. The id will be the same.

If you rely solely on the nodes:* events to track the comings and goings of nodes then you should stay fully in sync with the editor state. That handles all scenarios of add/deleting nodes, undo/redo, import etc.

[tve]

I know that that's the answer for simple nodes. As I tried to explain above, a dashboard (whether std dashboard or FlexDash) isn't a simple node. A quick search in ui_base.html brings up 17 uses of RED.history, so clearly more than just watching nodes:remove/nodes:add is necessary to deal with undo.

Also, would you mind commenting on:

@tve

a way to register a callback for a node that gets invoked before the node gets externalized so it can modify the config, i.e. callback(config) -> config_to_output; this can normalize the config for the run-time (I believe I've seen code that does exactly this for internal purposes)

I know my initial post was long. I tried to be detailed. Perhaps it is too long?

[knolleary]

@tve

know my initial post was long. I tried to be detailed. Perhaps it is too long?

I'm currently away from my laptop for a few days so my replies are what I can type out on my phone, working only from memory of how things work... So forgive me if I cannot give a full and complete answer to all your points until later in the week. Here I was just addressing the question you'd asked in the latter post.

@tve

given a NodeID how to determine its state: active, disabled, deleted-but-undoable, deleted/non-existent

We don't have the concept of these different deleted states - it either exists or it doesn't. For whether it is disabled or not, you get the node and check its disabled flag.

@tve

getting a callback when a node transitions between the above states, or alternatively when the flow gets externalized (save/export)

As discussed, the node:* events should have you covered here for general edit actions.

There is no hook into the export functionality. What sort of work would you want to do at this point? Would you want to do it on the exported json or the node object itself?

@tve

possibly getting a callback when all flows/nodes have been loaded so one can determine the state of all nodes to which one holds a reference

We use the nodes:add event for this where we need to do this sort of work as it tends to be the same logic for initial import as it is whenever anything new is added. A technique we use is to debounce the requests (set a timer to handle the event in 200ms, and reset the timer if the event occurs again). That means we only do the actual work once after the initial load happens.

As for undo handling... that probably needs a bit more of a write up than I can sensibly do in my current situation. I'll endeavour to write something later this week when I'm home. It is an area I'm going to be looking at in fine detail soon as we look at how to add support for concurrent realtime editing of flows...

[tve]

@knolleary

I'm currently away from my laptop for a few days so my replies are what I can type out on my phone

Understood, I can wait 'til you're back at a laptop.

@knolleary

@tve

given a NodeID how to determine its state: active, disabled, deleted-but-undoable, deleted/non-existent

We don't have the concept of these different deleted states - it either exists or it doesn't. For whether it is disabled or not, you get the node and check its disabled flag.

OK. The pain point I have is that if I need to hold to some data to make undo work cleanly (position-in-container in my case) then when can I safely delete that data?

@knolleary

There is no hook into the export functionality. What sort of work would you want to do at this point? Would you want to do it on the exported json or the node object itself?

Maybe what I'm trying to do doesn't work. From your above statement we have the following situation:

• The flow editor has:
  • active nodes
  • disabled nodes
  • deleted nodes (for undo)
• The run-time has:
  • active nodes
  • disabled nodes

In the editor RED.nodes.node provides access to active and disabled nodes, deleted nodes cannot be accessed.
In the run-time, RED.nodes.getNode provides access to active nodes, IIRC disabled nodes "do not exist".

The upshot is that an array of "widgets in this container" may contain disabled nodes, so the run-time has to deal with the fact that some nodes seemingly don't exist.

@knolleary

We use the nodes:add event for this where we need to do this sort of work as it tends to be the same logic for initial import as it is whenever anything new is added. A technique we use is to debounce the requests (set a timer to handle the event in 200ms, and reset the timer if the event occurs again). That means we only do the actual work once after the initial load happens.

Why not offer an official callback for this?

I look forward to the undo write-up! Thanks for tapping out the responses on a phone!

[knolleary]

@tve

The pain point I have is that if I need to hold to some data to make undo work cleanly (position-in-container in my case) then when can I safely delete that data?

I assume this information only needs to exist in the editor in the period where a user might hit undo and restore something they deleted in the current edit session?

Undo history is held in memory in the browser. So there's no possibility user loading the editor a week later and restoring a previously deleted node.

Given that, I would just cache the information in the browser memory and just let it get cleared when the tab is closed or the editor reloaded.

@tve

Why not offer an official callback for this?

As I mentioned, I found in all of the cases that could use such an event I would just end up duplicating the logic needed for the *:add events - as it was logic that needed to run whenever anything was added, whether the initial load or a subsequent add/import.

I'm not saying such an event can't have a role to play, just that I've simply not seen it as being indispensable. If there is a strong use case for having it, that is distinct from the existing events, then we can certainly consider it.

[tve]

@knolleary

Given that, I would just cache the information in the browser memory and just let it get cleared when the tab is closed or the editor reloaded.

True. This assumes there is a callback on undo to restore that info.

@knolleary

As I mentioned, I found in all of the cases that could use such an event I would just end up duplicating the logic needed for the *:add events

Wait! You just wrote one message back:

@knolleary

We use the nodes:add event for this where we need to do this sort of work as it tends to be the same logic for initial import as it is whenever anything new is added. A technique we use is to debounce the requests (set a timer to handle the event in 200ms, and reset the timer if the event occurs again). That means we only do the actual work once after the initial load happens.

You pretty much say it: the purpose of a "loading complete" callback is not to process additions, but to do garbage collection! Once everything is added, anything that points to non-existent stuff is garbage and can be removed.
Waiting 200ms is just asking for trouble, especially with async loading...