[ANNOUNCE] node-red-contrib-sub-link

MQTT was my first option... but with time, and with the incresse of messages you start having tons of messages in and out subflows, overloading the mqtt server... plus inside the subflow a mqtt message is received by all the instances of that subflow... With this node, it's possible to do that if intended, plus selecting the exact instance of the subflow that will receive the message.

I'm working now on a option to change the topic and alias of the output node on the fly, this will allow to better select the subflow instance that will get the message.

1 Like

Wow, that is something I have never seen. What message rate were you using and on what hardware?

1 Like

I have just tried an experiment sending 1000 messages/sec to mqtt (mosquitto on localhost) which consumed about 85% of a core running node-red and 10% running mosquitto. I suspect node-red will max out long before mosquitto does.

1 Like

I'm not saying I have that problem, but it can happen... But the fact that all messages that go to the topic are received by all subflows, that happens. Plus, by my tests, the speed of transfer a message in this node is a lot faster that with mqtt. The speed is basically the same of the link node.

Don't get me wrong... I used the mqtt to send messages between subflows for a long time, and worked well, but with time I sarted to see some delays in the messages, and in fast acting situations I started to notice... For example a motion sensor triggering and 1 second to get a response from the lamp. It might not seem a lot, but I used mqtt for a long time.. and I sometimes I had over 100ms of response time between sending the message and arriving in the other end... With this node I get arround 10ms tops

Make the same test with this node :stuck_out_tongue:

I wasn't arguing that your node wouldn't be more efficient, just surprised, maybe even disbelieving, that node red could ever overwhelm mosquitto.

Also, I can't think of a single instance when I have wanted to send a message into a subflow, let alone do it so much that the speed becomes an issue. It seems to me to go completely against the idea of subflows. However, the great thing about node-red is that it can be applied to a vast variety of problems, and if you have problems that require that then that is fine.

The main use case that I use is to broadcast changes in "devices" (iot things) that will make trigger actions. For example, when a motion sensor triggers, I broadcast that to act in several areas... It's a solution to keep the code organized in the main flows.

Or, for example, monitoring DHCP to see when a device comes online, I see that and "broadcast" that, several areas can capture that event and act accordingly.

Could you do all this without this node or broadcasting? Probably yes... but I fell like I can organize my code better if I have it this way!

Both of those seem like ideal candidates for MQTT. The code that detects the motion publishes the fact that motion has happened. It doesn't know or care who makes use of that.
Any flows that need to take action when motion occurs subscribe to that topic (on the same node-red device or a different one if the system is distributed across several devices) and do what they need to do.

Yes, but you could need to manipulate something before! Imaging the same but if you update a variable, and need to broadcast that change... I don't know... I must confess that I really missed this function in nodered, linking to subnodes.... I used MQTT to send that messages... basically subscribed to a topic and send the messages...

For example... TCP communication in.. I need to broadcast something... I cant have a tcp in inside a subflow... that will make duplicated connection... What I do is to add it to the main flow and broadcast it... Before I did that with mqtt... now I use this...

This is a test... I basically downloaded a 1920 image from a link (to have a big message to extreme test), then I store that message in a flow var and send it once a second... That is broadcast to 30 exit nodes. This are the stats of my local nodeded with local mosquitto (both only using this). Mqtt messages are qos 0

2 graphs, the first the delta time before sending the message and after arriving. The second is stats from the garbage collector (memory)

30 outputs with mqtt

30 outputs with my node

The difference is huge

Certainly MQTT is not appropriate for such requirements. As I said, if your use cases are similar to that and you want to link into the middle of subflows (rather then setting topic values to indicate what the message is for and feeding the message in the normal input) then your node may well be a good solution.

On a slightly different point, have you used any undocumented API calls to implement this?

To call what? To develop the node? I only use a node to validate mqtt topics,

I meant have you used any undocumented methods in your custom node, so it might stop working following a node-red upgrade.

I don't think it's too out of the box! Basically uses the config node to manage all the work of casting the messages, this insures that multiple instances of the input and output nodes would not have big impacts in the performance and resources!

Does that answer the question:

I am not saying it is a disaster if you have, but probably you should document which you rely on if you have.

I have not had time to try out your new nodes, but you might have a look at this discussion. It touches on some of the issues raised by this approach. I experimented with something similar but much less powerful and decided that it was a bit too far out of the normal Node-RED pattern to be generally useful. One of the strengths of NR is that the editor lets you visualize the entire flow of a program. (Even the link nodes show their connections.) The capability to connect "anything with anything else" is exactly opposite to that philosophy. I'm sure it will work for you and possibly some others, but there is the real possibility of creating masses of "invisible spaghetti" that will be difficult to document now and even harder to troubleshoot or modify in a few months time.

1 Like

I agree with the abstraction of the flows! This has some good use cases! The ability to simulate broadcast for me is one! I must say that I have all my node red configuration based on this! Until I started developing this node I was using mqtt! But as demonstrated in the tests above, that makes it a lot slower!

At the moment I'm finishing some great improvements to the nodes, that I should release during Next week! Some feedbacks that I had and also some extras! One of the biggest is setting the topics and the alias (or labels, basically a way too limit the flows a step further). With this it will be possible to change the alias of the nodes, making it esyer to target specific node instance, even if inside subflows!

I've tried too look at the link nodes to understand how it makes the visual connection between the nodes, but still need more analyses! Probably I'll need some help from the developers too get that (and even if it's possible to do on nodes that are not core)

Hope to have more people using and suggesting improvements.

Thank you

As far as can remember from earlier discussions (also here), the mechanism used by core nodes (link, status, catch, and complete) is not available to contributed nodes. This is not likely to change, since the message-passing mechanism is both deeply embedded in the core code and likely to evolve to handle distributed implementations of NR and messaging between different instances of NR and different hosts.

Simulating broadcast is clearly a reach for NR. It is contrary to the concept of sending messages on wires, which is the foundation of the tool. A flow-based programming model may not be the right answer for applications where broadcast predominates and a pub/sub architecture might be preferable.

I think that was the core function of node red! But like everything, it's evolving! The evolution of the link node to jump between flows was the first steps! The main limitations on linking to subflows was the control of the messages, and selecting to what instance the message should go! This node helps with that!

I believe that this node, or other's similar with the same concept, have the place and use cases! To use it or not, it's up to the end user!

I can see the value in this plugin... will have a play later - many thanks.