If Node X 's output is used as the input of Node Y, the output message topic of Node X must match the input message topic of Node Y, the easiest solution I found is to insert a Change node between them for explicit topic conversion.
But this means every pair of tightly-coupled nodes like the above require passing data via an extra Change node. This cost quickly adds up and could result in a cluttered network.
I doubt that this would be the Node-RED best practice. What am I missing?
Can you provide an example of what you mean?
If you need to set properties on a message, then the Change node is a perfectly reasonable node to use.
For example,
{topic: "my_name", payload: "john smith"}
{topic: "his_name", payload: ...}
Then you mean a Change is inevitable for Y to use "my_name" as "his_name", right?
Is there any other way to reduce the use of this extra Change node, so that we could hook up X and Y directly via a wire?
If you need to modify a message's properties, you need to have a node to do that work. I'm not sure how else you think it would work.
OK. That's sad.
I vaguely remember somewhere on this site, some post mentions that other than Change node, you could have some kind of context trick to bridge the gap I mentioned. But I can't find it anywhere now. Wonder if I got the wrong memory.
I can't think what that would be. If the node expects a message to have msg.topic set to a particular value, then you need to make sure it's set to that value.
Yes.
I often wish if a node could allow for multiple input ports then the topic matching might not be needed anymore. You just wire any single output port to any single input port and hopefully no topic ambiguity needs further clarification.
Is there any specific reason that this architecture may never fit, which I fail to see?
if a node had (say) 3 inputs you would still still need to match one of them - and now every node has the added complexity of having to define each of those - and even if they did there is no guarantee that one of them would have been set to expect "his_name" (to use your example).
Of course if either the sender or receiving node is a function node you can do the check/conversion as part of that function node.
(There are lots of other reasons to do with timings of inputs and input state management and error handling that just make it totally non-trivial that it is not on any radar)
What types of nodes are you using that require the topic matching?
Most of my nodes are custom nodes. It seems I always need to match the topics whenever one node (X) outputs a message to another (Y). Especially when Node Y depends on multiple input variables that are aggregated into one input port.
I was more from the other FBP-like environment like Unreal Engine
In those systems, each node can have an arbitrary number of input ports, where they are essentially bound with strongly-typed variables.
OK, that is not the most common paradigm in node red. In that case I don't think you have much option other than to use change nodes, or allow the specification of the output topic in the messages sent by your custom nodes.
Why?
What is it about your flow that is making you tightly couple nodes to a specific topic string? Normally, if X connects to Y, you would expect both to have the same topic unless the purpose of Y was to change the topic. The most usual reason for checking the topic in a flow is to filter messages or redirect them to a different flow based on their topic.
So if you could explain why you feel the need to have all of your Y nodes only work with a specific topic, we might be able to help further.
A picture of an example flow might also help.
You may want to speak with the developer of those nodes. As others have said, this is a very unusual requirement. I am curious about its purpose or value. I have published several contributed nodes that require inputs with specific values of msg.topic, but they allow the user to define that value when the node is deployed.
"allow the specification of the output topic in the messages sent by your custom nodes."
I supposed this would be a better middle ground than using Change nodes everywhere.
Thanks!
Maybe my screenshot of Unreal Engine above could explain a little?
Maybe I'm still not super clear about my needs? Let me try again.
When a node expects multiple inputs, the most common architecture I've been exposed to uses multiple input ports on the message receiver end. In those cases, there is NO ambiguity among the message topics, e.g.,
The GetFile node in the screenshot expects 5 inputs, each with a specific name and variable type. This allows for a one-to-one mapping between any output port and input port, so that the ports easily accepts or rejects data inputs that don't fit.
With Node-RED, the message receiver node can have only one input port. Therefore, if I try to feed the same 5 pieces of info into that single port of my GetFile node made in Node-RED, I'll most probably use a Merge node to aggregate 1-5 other nodes' output messages so that a packed message is sent through that single connection to my Node-RED GetFile node's single input port. Then GetFile node must distinguish between those 5 streams of arguments internally, and match them with its own 5 internal variables used by its own main logic.
Just look at the 4 nodes in the screenshot above, the 3 red ones and the blue one outputs info through their own ports with specific names.
So back to Node-RED, with a mirrored case, here I assume a name-matching must happen at the junction of those 4 nodes and GetFile node. Now, because other nodes that output the info that GetFile needs will name their output message in an arbitrary manner (there may be other nodes out there that are perfectly fine to connect to GetFile), GetFile will be unable to perfectly predict how other nodes may name their output messages' topic, and with the Merge node in between and without Change to explicitly translate the topics, GetFile will fail to use the 5 pieces of info.
I think @Colin may have got the idea, but just in case I haven't made my point to everyone in the thread.
I'm trying my very best to speak out loud what the screenshot above says.
There are other cases out there, such as this !PureData math
This is actually fine with Node-RED because the input arguments are symmetrical. Their "topics" are the same: numbers.
However, in reality, the majority of the cases I deal with demand that I pass various data types bearing specific meanings (file paths, shopping lists, text, numbers, ...) around to various nodes that uses 2+ inputs.
With the Unreal-Engine-like architecture, it's fairly easy to set up internal port-by-port data validation to accept/reject incoming data, which will work for all cases without the need for a name matching.
The realistic problem I'm facing is to control the complexity of an overall workflow, the exponentially growing needs for Change nodes to act as topic adaptors is a source of complexity and also a usability annoyance even with Colin's idea about using a specific config field per node for user to specify at every node-to-node junction.
I think that the "context trick" you are referring to is to keep all the different input variables in context, then when you want to trigger the node you retrieve them all from context and put them in a single msg.
The other option you have, if the variables are synchronous (their updates happen at more or less the same time), is to use a join node to combine all the inputs in a single message, then feed it into the input.
@OriolFM Thanks for the suggestion. I'll look into context, but still not sure how it could magically do the topic match-making. Please see my comments just above you.
The merging idea is actually used in our pipeline now, but only AFTER each of the Merge node's input has been processed by a Change node beforehand, so that the names are set straight, but by using Merge/Join nodes alone it does not seem to solve the topic matching problem, unfortunately.
As other people mentioned before in the thread, node-red nodes have only one input. The workflow often involves combining several variables into a single object before injecting that object with a message into the node.
I understand that other frameworks have different workflows, but that is how it works in node-red. If your custom nodes are not within node-red specifications, it's not a surprise that they don't play well with the rest.
If those nodes can work as the ones in node-red are supposed to, and you need several input variables, they should be contained in the same message. In order to do that, either you use Join nodes to combine several incoming messages or you use context, as I pointed out earlier.
Some node-red nodes also use different topics to trigger different functions of the same node (the timer/delay nodes come to mind, also the gate nodes).
In that case where you need specific topics to trigger different functionalities, yes, either you change the topic with a change node, or you use a function node to change the topic according to certain parameters as defined by you. The latter is more versatile, and you probably can handle many more cases with it, not depending only on the original topic, but also on the msg parameters. You can also set up flags for later use in the flow according to your needs.
If you have a specific structure of nodes that you have to repeat several times in your flows, remember you can set them up as a subflow. That will help you declutter the other flow tabs.
The reason Node-RED doesn't work this way is due to the asynchronous nature of Node.js. You can have multiple inputs but a node can never know when or what order they will arrive.
So if you have a custom node that relies on multiple input messages, the node itself has to manage the inputs. In doing so, the author of the node has to make some decisions about the order of things. Should it wait for 1 each of the input msg types? If so, how long should it wait until it treats them as a new message? Or does it care and simply use the last input of a particular type. As I think you will see, this logic can get tremendously complex. But it is totally flexible.
That processing can, of course and as others have mentioned, be done upstream of your custom node such that the node expects all inputs in a single message. This is a LOT simpler for the node to process as it doesn't have to worry about timings and such like. However, this is just pushing the problem upstream. You still have to make all of those decisions somewhere if the inputs are coming from separate places or processes. You do, of course, have nodes such as the join node that can help with this but such a node inevitably only offers a subset of the possible choices.
Similarly, you can accept all of your inputs and create an amalgam in a flow/global variable and get your node to only look at that variable (or, in a more node-red like process, pass the variable into a single input msg). Again though, this is making a specific decision about timings and the order of things.
So which of these options best suits your needs depends on what the data is and how you need to process it over time. Node-RED allows for any need, it doesn't make assumptions.
Or you can now also use things such as the return link to enable a sub-flow. Or even use my event nodes if you want to be able to branch to different flows depending on the input.
Sorry, which ones?
Copyright OpenJS Foundation and Node-RED contributors. All rights reserved. The OpenJS Foundation has registered trademarks and uses trademarks. For a list of trademarks of the OpenJS Foundation, please see our Trademark Policy and Trademark List. Trademarks and logos not indicated on the list of OpenJS Foundation trademarks are trademarks™ or registered® trademarks of their respective holders. Use of them does not imply any affiliation with or endorsement by them.
The OpenJS Foundation | Terms of Use | Privacy Policy | OpenJS Foundation Bylaws | Trademark Policy | Trademark List | Cookie Policy
