Custom cloning functionality for types not supported by `lodash.cloneDeep`

Problem

When using types relying on native bindings, such as tf.tensor from TensorFlowLite, the standard clone functionality provided by Node-RED is not capable of deep cloning, meaning that only the first output of each node can be used reliably.

Workaround

One possible solution is that a Node library that uses such types and requires the message to be cloned also provides a custom cloner node taking one message as input, does a custom deep cloning on 2 or more outputs. The drawback of such approach is that all Nodes with multiple outputs cannot be used to process such messages, and they functionality must be duplicated, likely using function nodes.

Proposal

My proposal is to provide the possibility to define custom cloners for given Javascript types, (identified by their prototype) which are not supported by lodash.cloneDeep. I propose to use lodash.cloneDeepWith to perform the cloning.

Sample code:

The var m = cloneDeep(msg) in RED.utils.cloneMessage is replaced by

var m = cloneDeepWith(msg, function(v) { 
    const cloner = customClonerMap.get(Object.getPrototypeOf(v));
    return cloner ? cloner(v) : undefined;
})

Where customClonerMap is a Map containing the cloner functions associated to each prototype.
cloneDeepWith recursively clones the object using the function. In case the return value of the cloner function is undefined the standard lodash.clone is performed.

How to register a cloner

1. Nodes register their own cloner

Nodes that want to register a custom cloner do so by calling RED.utils.registerCloner(type, cloner) which register the cloner function for the type.prototype (where type is a Function).

Multiple nodes might register a cloner for the same type: in this case the latter cloner registered with a call to registerCloner would be one used. Since the goal of the cloner function is to correctly clone the specific type, it should not matter which is the actual cloner function that will be used since it should correctly clone the given type. This is also needed in order to allow for multiple nodes in a same contribution to register the same cloner as any of the nodes can be added to the flow.

My main concern with this approach is that there is no effective way to prevent or warn the user when a malicious node register a malicious cloner function. Message passing and cloning is at the core of the trust user have in Node-RED and I do not like the eventuality that such trust might be compromised.

2. Users add cloner via a specific configuration node (preferred)

One alternative solution would be specify a new type of node, similar to config nodes, that is global to the Node-RED instance and whose task is only to register the custom cloner for a given type. In this case, it would be responsibility of the user to add that specific node, and Node-RED could enforce that only one such node per type can be used.

Normal nodes should be able to check that the required cloner nodes have been added to the flow.

If a cloner configuration node required by a node in the flow is missing, the flow cannot be deployed.

Eager to read what the Node-RED community thinks!

@massi-ang,
I think I would prefer option 1, because:

  1. I like nodes where I don't have to install anything manually.
  2. Option 2 seems a lot of work for the core developers.
  3. A config node is also not installed separately. I think it would be confusing to me if I have to install a node that is not available in the palette.

Or do you mean a new type of Node-RED plugin perhaps, instead of a new type of node?

Hi Bart,

I do not know about plugins enough to say if that could be the right path. Can you point me to some docs that explain what Plugins are and how they can be created?

Sorry can't find it anymore. Don't know either if plugins are only for UI purposes...

Hi @massi-ang

thanks for putting together this clear proposal.

We're always wary of any code that ends up in the core messaging path - especially the cloning path as that can have such a large impact on performance.

Some thoughts on the implementation:

  1. it should not allow a custom clone function to be set on any built-in/native type. (eg Object.prototype, String.prototype etc)

  2. cloneMessage should only use cloneDeepWith if a custom cloner has been registered. For most users that will remove any overhead of using cloneDeepWith

I think this makes it too complicated to use. It is a level of detail users should not need to care about. I would go with option 1.

Plugins are used to add custom code to both the runtime and editor. But given the custom cloners exist hand-in-hand with custom nodes that want to produce/consume a particular object type, then I don't think this is a plugin feature at this stage.

1 Like