Structuring Flows

General
1

I'm about to structure flows in a Server/Client structure. The server flow will sent messages to various client flows. I was contemplating two mechanisms for accomplishing this:
METHOD 1: Have the server send a message to a given client via a topic unique to that client.
METHOD 2: Have the server send a message to a generic topic that all clients subscribe to letting each client determine whether the message was for them by examining the message payload.
It appears Method 1 would result in high Broker load. While Method 2 would result in high client processing load.

Looking for opinions, which would be the preferred approach? Is there a better approach?

2

Not really enough to go on. Are the clients resource constrained? How many of them are there? Is the broker resource constrained? Which broker are you using? Have you tested your broker for throughput?

Will it be easier to maintain everything from a central service (method 1) or easier to maintain the clients (method 2)?

3

Unfortunately, I'm just starting out with Node-RED. I was hoping some user already had experience with a server/client model, where there could be 50 clients. Having individual clients process from a generic topic (Method 1) is probably the way to go as they will largely be on separate platforms. I would also think the generic topic approach would be easier on the Broker.

Perhaps I'm asking the wrong question. Is the work load on the broker easier if it has to send messages to all clients via a generic topic, or is keeping track of a unique topic for each client easier?

4

Honestly, it is very unlikely that you will stress out your broker.

5

Sending to all clients will be a greater workload for the broker.

If your scenario requires the ability to send a message to a particular client, then use well named topics to do it - this is what topics are for.

The work taken to do the pattern matching in the broker to identify the properly subscribed client will be much more efficient than sending every message to every client over the network.

6

I guess I was a little confused as to how MQTT functions. Being unfamiliar with the underlying architecture of MQTT, I was thinking that all clients that subscribed to a generic topic , the Broker would only need a single transmission to the generic topic and all the clients would then receive that message.

I gather from what your saying the Broker would need to send the same message individually to each client; which of course would be quite a workload for the broker. Having unique topics for each client would only require a single transmission to a given client.

My experience with a single buss architecture doesn't serve as a good analogy for MQTT Star Topology.

7

Right, it isn't a bus. Each client is connected to the broker through its own TCP connection. For a client to receive a message, the broker has to send it to that client over its connection. Much better to do the filtering in the broker.

1 Like
8

Thank you for helping me get over my Buss-bias, and remembering MQTT is a Star-Topology.

I'm already re-configuring my thought processes as to how I'm going to configure the Node-RED flows.

9

How often will you be sending messages to each of them?
What hardware will the broker be running on?
Is this on a local network or across the internet?

For an excellent tutorial on MQTT see MQTT Essentials - All Core Concepts Explained

10

Here's an example of sending MQTT messages between a master and a set of clients.
It's taken from my IoT Club. At the end of each session all the Raspberry Pi(es) need to be shutdown or if a student encounters an issue with a Pi during a session then that particular Pi needs to be shutdown.

The Master flow, shown below, can send a payload 'all' - if all the Pi(es) need to be shutdown, or a unique number (which equates to the 4th octant of the Pi's IP address) if a specific Pi needs to be shutdown.

Screen Shot 11-22-21 at 10.13 AM
Screen Shot 11-22-21 at 10.13 AM826Ă—365 45.4 KB 

[{"id":"f31578bd77f4736d","type":"tab","label":"Master shutdown","disabled":false,"info":"","env":[]},{"id":"1c0fa2a.31308dd","type":"mqtt out","z":"f31578bd77f4736d","name":"","topic":"local/shutdown","qos":"","retain":"","respTopic":"","contentType":"","userProps":"","correl":"","expiry":"","broker":"fe4d4074.20ff58","x":780,"y":100,"wires":[]},{"id":"f4d42d2.4272fd","type":"function","z":"f31578bd77f4736d","name":"pass thru","func":"return msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":600,"y":100,"wires":[["1c0fa2a.31308dd"]]},{"id":"703f41ce.c30bf","type":"comment","z":"f31578bd77f4736d","name":"Shutdown routine - shuts down all RPi on the network","info":"","x":280,"y":60,"wires":[]},{"id":"b8b2a0b0.f15358","type":"inject","z":"f31578bd77f4736d","name":"Start GLOBAL shutdown on the network","props":[{"p":"payload"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"all","payloadType":"str","x":560,"y":280,"wires":[["f4d42d2.4272fd"]]},{"id":"66650eee.75031","type":"inject","z":"f31578bd77f4736d","name":"Shutdown RPi-131 on the network","repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"131","payloadType":"str","x":540,"y":380,"wires":[["f4d42d2.4272fd"]]},{"id":"23de1a46.7d48ce","type":"inject","z":"f31578bd77f4736d","name":"Shutdown RPi-130 on the network","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"130","payloadType":"str","x":540,"y":340,"wires":[["f4d42d2.4272fd"]]},{"id":"5263a9c1.11749","type":"inject","z":"f31578bd77f4736d","name":"Shutdown RPi-132 on the network","repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"132","payloadType":"str","x":540,"y":420,"wires":[["f4d42d2.4272fd"]]},{"id":"fce06f4.ad0cd9","type":"ui_button","z":"f31578bd77f4736d","name":"","group":"146b3c10.143844","order":1,"width":"2","height":"1","passthru":false,"label":"Shutdown local network","tooltip":"","color":"","bgcolor":"","className":"","icon":"","payload":"all","payloadType":"str","topic":"","topicType":"str","x":190,"y":100,"wires":[["f4d42d2.4272fd"]]},{"id":"dd2d3a20.612c2","type":"ui_button","z":"f31578bd77f4736d","name":"","group":"146b3c10.143844","order":2,"width":"1","height":"1","passthru":false,"label":"130","tooltip":"","color":"","bgcolor":"","className":"","icon":"","payload":"130","payloadType":"str","topic":"","topicType":"str","x":130,"y":200,"wires":[["f4d42d2.4272fd"]]},{"id":"49862804.0a5478","type":"ui_button","z":"f31578bd77f4736d","name":"","group":"146b3c10.143844","order":3,"width":"1","height":"1","passthru":false,"label":"131","tooltip":"","color":"","bgcolor":"","icon":"","payload":"131","payloadType":"str","topic":"","x":130,"y":240,"wires":[["f4d42d2.4272fd"]]},{"id":"7e4ff06c.547068","type":"ui_button","z":"f31578bd77f4736d","name":"","group":"146b3c10.143844","order":4,"width":"1","height":"1","passthru":false,"label":"132","tooltip":"","color":"","bgcolor":"","icon":"","payload":"132","payloadType":"str","topic":"","x":130,"y":280,"wires":[["f4d42d2.4272fd"]]},{"id":"6945d990.0688b","type":"ui_button","z":"f31578bd77f4736d","name":"","group":"146b3c10.143844","order":6,"width":"1","height":"1","passthru":false,"label":"133","tooltip":"","color":"","bgcolor":"","icon":"","payload":"133","payloadType":"str","topic":"","x":130,"y":320,"wires":[["f4d42d2.4272fd"]]},{"id":"e18acf01.ef114","type":"ui_button","z":"f31578bd77f4736d","name":"","group":"146b3c10.143844","order":7,"width":"1","height":"1","passthru":false,"label":"134","tooltip":"","color":"","bgcolor":"","icon":"","payload":"134","payloadType":"str","topic":"","x":130,"y":360,"wires":[["f4d42d2.4272fd"]]},{"id":"bfc84208.1e7ca","type":"inject","z":"f31578bd77f4736d","name":"Shutdown RPi-133 on the network","repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"133","payloadType":"str","x":540,"y":460,"wires":[["f4d42d2.4272fd"]]},{"id":"3a39fbae.771564","type":"inject","z":"f31578bd77f4736d","name":"Shutdown RPi-134 on the network","repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"134","payloadType":"str","x":540,"y":500,"wires":[["f4d42d2.4272fd"]]},{"id":"3b98f6fe.746ad2","type":"ui_button","z":"f31578bd77f4736d","name":"","group":"146b3c10.143844","order":7,"width":"1","height":"1","passthru":false,"label":"135","tooltip":"","color":"","bgcolor":"","icon":"","payload":"135","payloadType":"str","topic":"","x":130,"y":400,"wires":[["f4d42d2.4272fd"]]},{"id":"6ab4ee81.aa911","type":"inject","z":"f31578bd77f4736d","name":"Shutdown RPi-135 on the network","repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"135","payloadType":"num","x":540,"y":540,"wires":[["f4d42d2.4272fd"]]},{"id":"505de00c.91df98","type":"inject","z":"f31578bd77f4736d","name":"Shutdown RPi-136 on the network","repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"136","payloadType":"num","x":540,"y":580,"wires":[["f4d42d2.4272fd"]]},{"id":"76f3ee7a.b9ef6","type":"inject","z":"f31578bd77f4736d","name":"Shutdown RPi-137 on the network","repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"137","payloadType":"num","x":540,"y":620,"wires":[["f4d42d2.4272fd"]]},{"id":"64097055.f8004","type":"inject","z":"f31578bd77f4736d","name":"Shutdown RPi-138 on the network","repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"138","payloadType":"num","x":540,"y":660,"wires":[["f4d42d2.4272fd"]]},{"id":"ea28151e.eebeb","type":"inject","z":"f31578bd77f4736d","name":"Shutdown RPi-139 on the network","repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"139","payloadType":"num","x":540,"y":700,"wires":[["f4d42d2.4272fd"]]},{"id":"cef4a338.6870a8","type":"ui_button","z":"f31578bd77f4736d","name":"","group":"146b3c10.143844","order":7,"width":"1","height":"1","passthru":false,"label":"136","tooltip":"","color":"","bgcolor":"","icon":"","payload":"136","payloadType":"str","topic":"","x":130,"y":440,"wires":[["f4d42d2.4272fd"]]},{"id":"3b764a8e.67a38e","type":"ui_button","z":"f31578bd77f4736d","name":"","group":"146b3c10.143844","order":7,"width":"1","height":"1","passthru":false,"label":"137","tooltip":"","color":"","bgcolor":"","icon":"","payload":"137","payloadType":"str","topic":"","x":130,"y":480,"wires":[["f4d42d2.4272fd"]]},{"id":"8ad8a842.b10e8","type":"ui_button","z":"f31578bd77f4736d","name":"","group":"146b3c10.143844","order":7,"width":"1","height":"1","passthru":false,"label":"138","tooltip":"","color":"","bgcolor":"","icon":"","payload":"138","payloadType":"str","topic":"","x":130,"y":520,"wires":[["f4d42d2.4272fd"]]},{"id":"180cacb5.3a0223","type":"ui_button","z":"f31578bd77f4736d","name":"","group":"146b3c10.143844","order":7,"width":"1","height":"1","passthru":false,"label":"139","tooltip":"","color":"","bgcolor":"","icon":"","payload":"139","payloadType":"str","topic":"","x":130,"y":560,"wires":[["f4d42d2.4272fd"]]},{"id":"959076a3ca03862b","type":"comment","z":"f31578bd77f4736d","name":"Shutdown specific RPi on the network","info":"","x":230,"y":160,"wires":[]},{"id":"fe4d4074.20ff58","type":"mqtt-broker","name":"RPi_156 server_Use_Me","broker":"192.168.1.156","port":"1883","clientid":"","autoConnect":true,"usetls":false,"compatmode":false,"protocolVersion":"4","keepalive":"60","cleansession":true,"birthTopic":"","birthQos":"0","birthPayload":"","birthMsg":{},"closeTopic":"","closeQos":"0","closePayload":"","closeMsg":{},"willTopic":"","willQos":"0","willPayload":"","willMsg":{},"sessionExpiry":"","credentials":{}},{"id":"146b3c10.143844","type":"ui_group","name":"Shutdown the various servers","tab":"44aa8728.0c17d8","order":1,"disp":true,"width":"5","collapse":false},{"id":"44aa8728.0c17d8","type":"ui_tab","name":"Server control centre","icon":"dashboard","order":3,"disabled":false,"hidden":false}]

Here's a an example of the flow in each client. The client can be shutdown in three ways. If it receives a MQTT message with the payload "all" or a number that matches 4th octant of IP address or manually.

Screen Shot 11-22-21 at 10.43 AM
Screen Shot 11-22-21 at 10.43 AM874Ă—672 47.8 KB 

[{"id":"307594940d1dffad","type":"tab","label":"Typical client","disabled":false,"info":"","env":[]},{"id":"56030af0.53d984","type":"exec","z":"307594940d1dffad","command":"sudo shutdown -h now","addpay":false,"append":"","useSpawn":"","timer":"","winHide":false,"oldrc":false,"name":"Shutdown the RPi","x":830,"y":620,"wires":[[],[],[]]},{"id":"3af9d959.3222e6","type":"inject","z":"307594940d1dffad","name":"Shutdown RPi-140","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"1","payloadType":"num","x":210,"y":520,"wires":[["5facdc3d.663004","48a2678e.976698"]]},{"id":"5b0b2f9c.7769c","type":"inject","z":"307594940d1dffad","name":"Confirm shutdown","repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"1","payloadType":"num","x":210,"y":620,"wires":[["c497a9db.c1cfe"]]},{"id":"ea924756.5d9c18","type":"function","z":"307594940d1dffad","name":"flow.set shutdown ","func":"if (msg.payload == 1)\n    {\n        flow.set(\"shutdown\", \"yes\");\n        node.status({text:\"Shutdown window = yes\"});\n    }\nelse if (msg.payload === 0)\n    {\n        flow.set(\"shutdown\", \"no\");\n        node.status({text:\"Shutdown window = no\"});\n    }","outputs":0,"noerr":0,"x":670,"y":520,"wires":[]},{"id":"5facdc3d.663004","type":"trigger","z":"307594940d1dffad","name":"5 sec wide hi-pulse","op1":"1","op2":"0","op1type":"num","op2type":"num","duration":"5","extend":false,"units":"s","reset":"","bytopic":"all","outputs":1,"x":430,"y":520,"wires":[["ea924756.5d9c18"]]},{"id":"47d42746.162328","type":"function","z":"307594940d1dffad","name":"","func":"var shutdown = flow.get(\"shutdown\") || \"no\";\n\nif ( (msg.payload == 1) && (shutdown == \"yes\") )\n    {\n        node.status({text:\"Shutting down RPi\"});\n        return msg;\n    }\nelse\n    {\n    node.status({text:\"Nothing doing\"});\n        return null;\n    }","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":640,"y":620,"wires":[["17d22b32.dae6dd","56030af0.53d984"]]},{"id":"48a2678e.976698","type":"function","z":"307594940d1dffad","name":"Inverter","func":"if (msg.payload == 1)\n{\n    msg.payload = 0;\n}\nreturn msg;","outputs":1,"noerr":0,"x":400,"y":560,"wires":[["47d42746.162328"]]},{"id":"dc212d2e.177488","type":"comment","z":"307594940d1dffad","name":"Routine to MANUALLY shutdown this Raspberry Pi","info":"","x":290,"y":460,"wires":[]},{"id":"8966756e.282d8","type":"comment","z":"307594940d1dffad","name":"Press the Confirm button within 5 seconds if you do really want to shutdown","info":"","x":370,"y":660,"wires":[]},{"id":"17d22b32.dae6dd","type":"debug","z":"307594940d1dffad","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","x":810,"y":680,"wires":[]},{"id":"c497a9db.c1cfe","type":"trigger","z":"307594940d1dffad","name":"2 sec wide hi-pulse","op1":"1","op2":"0","op1type":"num","op2type":"num","duration":"2","extend":false,"units":"s","reset":"","bytopic":"all","outputs":1,"x":430,"y":620,"wires":[["47d42746.162328"]]},{"id":"80c8ac1.b1ece5","type":"mqtt in","z":"307594940d1dffad","name":"","topic":"local/shutdown","qos":"2","datatype":"auto","broker":"fe4d4074.20ff58","nl":false,"rap":false,"inputs":0,"x":180,"y":260,"wires":[["4a49e4cc.8fafe4","5c1570b.9895b1"]]},{"id":"33bfc969.a4e39e","type":"mqtt out","z":"307594940d1dffad","name":"","topic":"local/response","qos":"","retain":"","respTopic":"","contentType":"","userProps":"","correl":"","expiry":"","broker":"fe4d4074.20ff58","x":640,"y":260,"wires":[]},{"id":"4a49e4cc.8fafe4","type":"function","z":"307594940d1dffad","name":"","func":"var ip_4 = flow.get(\"ip_4\") || 0;\n\nif ((msg.payload == \"all\") || (msg.payload == ip_4))\n{\n  msg.payload=\"RPi with IP=\"+ip_4+\" is shutting down\";\n  return msg;\n}\n","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":380,"y":260,"wires":[["33bfc969.a4e39e","ba383537.bfff2"]]},{"id":"ba383537.bfff2","type":"delay","z":"307594940d1dffad","name":"","pauseType":"delay","timeout":"5","timeoutUnits":"seconds","rate":"1","nbRateUnits":"1","rateUnits":"second","randomFirst":"1","randomLast":"5","randomUnits":"seconds","drop":false,"outputs":1,"x":620,"y":320,"wires":[["56030af0.53d984"]]},{"id":"5c1570b.9895b1","type":"debug","z":"307594940d1dffad","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","x":390,"y":320,"wires":[]},{"id":"4b3ca65a.8b362","type":"hostip","z":"307594940d1dffad","name":"Host IP","x":340,"y":120,"wires":[["597242e3.1c3cf4"]]},{"id":"c647d1b4.5d449","type":"inject","z":"307594940d1dffad","name":"","repeat":"","crontab":"","once":true,"onceDelay":0.1,"topic":"","payload":"1","payloadType":"num","x":170,"y":120,"wires":[["4b3ca65a.8b362"]]},{"id":"597242e3.1c3cf4","type":"function","z":"307594940d1dffad","name":"Get and store the 4th octant of the IP address","func":"var ip_address = msg.payload[0].address;\nvar parts = ip_address.split(\".\");\nvar ip_4 = parseInt(parts[3]);\nflow.set(\"ip_4\",ip_4);\n","outputs":1,"noerr":0,"initialize":"","finalize":"","x":640,"y":120,"wires":[[]]},{"id":"865a415f.38e148","type":"comment","z":"307594940d1dffad","name":"Routine to AUTOMATICALLY shutdown this Raspberry Pi","info":"","x":310,"y":200,"wires":[]},{"id":"39f09cdc.5c09e4","type":"comment","z":"307594940d1dffad","name":"Capture the IP address for THIS Raspberry Pi","info":"","x":270,"y":80,"wires":[]},{"id":"fe4d4074.20ff58","type":"mqtt-broker","name":"RPi_156 server_Use_Me","broker":"192.168.1.156","port":"1883","clientid":"","autoConnect":true,"usetls":false,"compatmode":false,"protocolVersion":"4","keepalive":"60","cleansession":true,"birthTopic":"","birthQos":"0","birthPayload":"","birthMsg":{},"closeTopic":"","closeQos":"0","closePayload":"","closeMsg":{},"willTopic":"","willQos":"0","willPayload":"","willMsg":{},"sessionExpiry":"","credentials":{}}]

Notes:

  1. In the above examples ALL the RPi(es) subscribe to the 'master' (topic: local/shutdown).
  2. Each client returns a MQTT message to the master saying it is shutting down.
    (topic: local/response)
  3. If I was building the flow again - I'd use a drop-down menu for the selection and load the content from a text file as this would make the flow eaiser to maintain.

Hope this helps to get you started or provide you with some ideas.

1 Like
11

Colin,

I'm still just playing around with test Flows to organize a structure that works best for my application; everything is beginning to gel. Currently I just have a Broker running under Node-RED, but will be switching to Mosquitto. I've already had success demonstrating clients communicating, running under Node-RED, MQTTBox on two PC's and umqtt running on an ESP-01. I've only just started familiarizing myself with Node-RED, MQTT Java-Script & microPython over the past month.

Thanks for the info, X

12

dynamicdave,

Thanks for the informative Flow structures; they certainly demonstrate what can be done with Node-RED. I've only just started familiarizing myself with Node-RED, MQTT Java-Script & microPython over the past month. I've already had success demonstrating clients communicating, running under Node-RED, MQTTBox on two PC's and umqtt running on an ESP-01.
The FOG is beginning to clear :grinning:

13

You haven't said approximately how often messages will be sent to each client. For example it might be once an hour, once a second, or 1000 times a second.

14

Haven't gotten that far in my design process; only been familiarizing myself with the tools for about a mouth.

15

How can you possibly make even preliminary design decisions it if you don't have any idea at all how much data is moved about? There is a huge difference between a system that has to handle 1 message a minute and one that has to handle 50,000 messages a second!

16

Colin,

I've tried to make it clear that I've never used microPython (or Python), Java-Script, MQTTBox, Node-RED, ESP8266, or the Thonny IDE. And yet within less then a month I've gotten communication between two PC's and an ESP-01.

But to answer your criticism; the Flow structure will be event driven; and given the anticipated frequency of events will not come anywhere close to taxing the network, or the Broker.

17

In that case, going back to your original question, where you were worried about overloading the broker, don't worry about it. Design it in the simplest and most understandable way. Worry about optimisation after it is up and running and you can see where the bottlenecks are.

18

I'm sorry I my original question was ambiguous; I was just trying to understand how messages were processed via the Broker. Several posts helped clarify Node-RED for me.

Your suggestion is precisely the approach I intend to follow.

19

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.