Structuring larger projects

[cerietke]

In short I am looking for guidelines/tips on how to keep larger / more complex projects manageable in Node-RED. What follows are details on my situation and where I run into these issues.

I'm using Node-RED primarily for home automation and I've been stuck on some things. I learned to program in Java and PHP. When thinking about code I tend to think in a 'divide and conquer' kind of approach where you go more and more detailed and don't want higher levels to have to worry about the details of an implementation. Practically speaking, I have 3 types of lights at home, but at the highest level, I just want to send the same commands to turn them on, off or toggle and then at a lower level that would get converted to what a specific light needs (and in the case of a toggle that's not supported by the light itself I might keep track of the light's state there).

I can sort of accomplish this by having a lights flow and then different flows for the different light types. But as I implement more and more I start needing a lot of flows and it feels like I am losing the general overview. Currently I have the actual logic for what to do when in one flow, so that I may more easily re-use bits of code as for example I may have several different ways to turn on a light (in the case of my bedroom a MyStrom button on the wall and an M5Stack Core on my nightstand). As I find myself doing more and more for my home automation especially the logic flow starts getting very complex with many lines crossing as I wouldn't want to have duplicate code. I realize I could probably create and re-use subflows, I am already doing that in places, but I fear that ultimately even that becomes very complex.

Most examples I have been able to find on how to do things in Node-RED have been small flows to demonstrate a node or principle, what I wonder is if there are some guidelines somewhere on how to structure larger projects?

Additionally I find myself wanting to keep track of a lot of states. I understand that I could use flow or global variables, or perhaps store them in MQTT. How do you generally approach this for larger projects? This is the reason I've been looking at adding Home Assistant into the mix, because as I understand it, it's pretty good at keeping track of states, but if I am honest every time I check it out the interface confuses me and I don't think my project structure would get any better with it in the mix. On the other hand it feels like if I was using MQTT for this I would sort of make my own limited HA. I do like the idea of everything working through MQTT as I could have a lot of small bits of code triggering on changes and then writing back their values, which might then be monitored by another bit of code (making it easy to re-use as you just need to set a value in MQTT). Downside is an outside dependency and communication speed, it feels like there should be a Node-RED internal way to do something similar.

I've been looking at using openHASP to create little dashboards to show on M5Stack Core2 devices I have on my walls. I know I will need to keep track of quite a few things and inserting this into my current setup is probably going to increase the complexity quite a lot, so hence my questions at this time as it may be wise to re-structure everything so that it becomes easier to add this and other functionalities in the future (controlling where music is playing through Foobar2000/HEOS is also on my wish list for example, another fairly complex addition if I want it controllable through these dashboards).

Thanks!

[TotallyInformation]

Hi, some great questions there.

I'm sure that you will get multiple answers. But here are my thoughts.

Firstly, complex home automation is made a lot easier by introducing MQTT. Give it a go with some tests if you aren't using it yet and you will soon wonder how you lived without it! MQTT will help you with persistence since you can use "retained" topics. A retained topic will send updates to any MQTT-in node that is listening but will also send the current value when that node first connects (e.g. after a restart of Node-RED). So you don't have to manage that inside your flows.

To simplify complex flows, there are several things you can do but the same principle of DRY applies. Keep the flows in small, manageable, readable chunks. Here's what I tend to do (there are no absolute rules though):

• Use tabs wisely to partition the flows logically. For example, one for lighting, one for heating, one for low-level sensor interactions, one for startup, one for bots, one for warnings, notifications, ....
• Never cross the streams! (sorry, channelling Ghost Busters there for a second!) - I mean try not to cross wires too much. Link nodes are a god-send here. I use them a lot. I have a particular aversion to output wires going backwards (to the left).
• Use sub-flows when you need to be able to have a different instance of the same code.
• Use groups. They can have their own environment variables (like sub-flows) and help differentiate chunks of flow.
• Standardise your MQTT topics and sensor/controller data. This makes complex control easier because you always know what data you've got and can easily manipulate topic strings using splits, loops and joins where you need to.
• Disaggregate sensor data from logical controls and logical controls from physical ones. That way, you standardised sensor data can be used to manipulate logical controls and even where you need to change sensor or control hardware over time, you won't have to change the main logic. This will also help reduce the number of logic flows because you can have a standard logic flow that is capable of dealing with many inputs/outputs as they are standardised regardless of the physical end of things.

Hope that makes some kind of sense?

[cerietke]

Thanks for your elaborate response!

I am in fact already using MQTT, though not extensively. I have a couple of power strips with Tasmota on it and the M5Stack devices use it. I also have various computers on my network run their own copy of Node-RED and report their online state on MQTT (if needed they also listen for instructions from my main Node-RED instance if they have some job to do).

What I wasn't sure of was to what extend inserting MQTT in the flows would slow down processing. At the moment Node-RED and Mosquitto are both in Docker on the same device (a Firewalla Blue Plus), so I suspect there's not a big delay, but, a direct connection within Node-RED would probably still be faster? I noted that in a recent Node-RED update it was made possible to connect and disconnect from topics dynamically, which makes using MQTT even more flexible.

I wasn't familiar with DRY as an acronym, maybe I should add that I am not a developer by profession (I am a Product Owner, so still in IT), but yes, that makes sense and something I would absolutely want to do knowing that it may sometimes be months before I will work on something again and huge flows would take a lot of effort just to get my head around again before making improvements.

Agreed on partitioning the flows, but that's kinda where I run into challenges. In traditional programming I can essentially create as many levels in a hierarchy as I need, but in Node-RED it's essentially limited to flows, subflows and perhaps groups (thanks for mentioning those, I should check that out). As such I don't really have as many hierarchical levels as I would like. I guess if I could group (sub)flows in some way (and group the groups) then I would be able to structure everything much better (at least, better fitting with my way of thinking).

Crossing the streams should be much easier to avoid using MQTT. Everything can essentially either write to MQTT, read from MQTT or both and it would be a lot easier to write small chunks that do a single thing and that can just be re-used by reading from or writing to the right MQTT topics. I even imagine one might be able to define the logic in MQTT, making an extremely flexible system using interpreters of that logic, but I don't want to go that far.

I will have to look into groups, thanks.

On the whole you seem to be thinking along the same lines as I was when it comes to using MQTT, however I would wonder about delays as a result of it and whether there would be (Node-RED internal) alternatives?

It's making sense, but the last bullet was a bit harder for me: I can guess what you're getting at, but particularly that first sentence there was a bit more challenging when it comes to figuring out exactly what you mean (maybe because I am not a native English speaker?)

[TotallyInformation]

@cerietke

What I wasn't sure of was to what extend inserting MQTT in the flows would slow down processing. At the moment Node-RED and Mosquitto are both in Docker on the same device (a Firewalla Blue Plus), so I suspect there's not a big delay, but, a direct connection within Node-RED would probably still be faster?

Honestly, I wouldn't worry about the few ms difference as you will never notice that difference. Even on my old Pi2 which is where I started running Node-RED with Mosquitto and other services, I struggled to ever make the thing break into a sweat.

@cerietke

it was made possible to connect and disconnect from topics dynamically, which makes using MQTT even more flexible.

True. Though again I think you are over-thinking things. You can be subscribed to dozens of topics even on a really limited computer and never notice the load even with thousands of messages going through Mosquitto per second.

@cerietke

however I would wonder about delays as a result of it

Seriously, you will not notice in general use. Again, using my old Pi2 as an example, there was never more than a second of delay when processing from a switch to a relay and most of that was due to the type of wireless I was using and not down to Node-RED/MQTT processing.

@cerietke

whether there would be (Node-RED internal) alternatives?

There are always other ways of doing things but why bother? Over optimisation is something also to be avoided. Better to spend your time on more productive matters.

@cerietke

the last bullet

Yes, I did kind of gloss over that a little. There is a lot to think about in that bullet.

As a simple example of what I meant.

You could have a light sensor somewhere that you wanted to use to turn on/off some lights. You could have an input from that sensor and wire it directly to each relay output and that absolutely works. But what if you then wanted to add more relays, soon you have a rats-nest of wires.

Instead, if you output the sensor to a standardised MQTT topic, lets say SENSORS/light/hallway (if thats where the sensor was located). Now you can move the relay output nodes to a different tab and you can listen to that topic to control the relays.

But what if, later on, you wanted to add some different different controls to the lighting. Lets say a couple of different types of switches. You also output those to standardised topics. But now you need a second/third, etc set of wires into the relay nodes. Another rats-nest in the making but perhaps manageable. Until you start getting a bit smarter at the processing needed to decide on whether the lights should be on or off. Or maybe some of the relays get replaced with different ones and you discover that you now need to rip out and replace a bunch of wires.

By going a step further in the disaggregation, you create virtual outputs as topics. Lets say LIGHTS/LivingRoom, LIGHTS/Landing, etc. Now when you replace the relay hardware, the flows you need to change are controlled from those virtual outputs and you don't need to see or worry about the intermediate processing because it is on a different tab, you only need to worry about the link from the virtual to the physical output.

Hmm, I hope that is clearer though I'm not sure it is! It is much harder to describe than to do.

Tab1
  Startup processing
Tab2
  Physical inputs -> (Process to standardise input data) -> Output to std topics
Tab3
  Input from std sensor data -> (process to virt ctrls) -> Output to virt ctrls
Tab4
  Input from virt ctrls -> output to physical controls

Here are some of my tabs on my live home automation instance of Node-RED as an example:

And here, an example from the Devices tab where I'm processing inputs from my older 433MHz devices. The physical inputs are coming via an RFXtrx433e dongle.

For fun, I track all of the unknown inputs - there are hundreds of them because of the sensitivity of the receiver! Whereas I have a global variable that lists all of the known device ID's so that I can standardise the inputs and add things like lastSeen timestamps. I also differentiate between sensor inputs and control inputs (some inputs might be both of course such as the light sensor mentioned before or a PIR).

And some examples of virtual control outputs. Lighting in this case.

And finally an example of a virtual to a physical control output, again, this one from my old LightwaveRF 433MHz devices that are nearly all gone now, just the 1 left, replaced with Zigbee & WiFi.

The switch to Zigbee is another good example of why you want to disaggregate. Because the Zigbee devices and sensors are all used via Zigbee2MQTT. Processing is done in Node-RED but none of the physical inputs/outputs, I only need MQTT for that in Node-RED now. Having everything disaggregated made that an easy switch.

Hopefully the examples also illustrate that disaggregation lets you have small groups of nodes that are a lot easier to comprehend.

You will also note 1 further simplification of flows. That I use function nodes a lot. That's something that you can skip but it can often be better to collapse a tangle of nodes into a function node. Of course, that depends on your willingness to do some JavaScript coding. I try to keep a balance between the two, making sure that my function nodes contain small, manageable chunks of code that are easily understood when I come back to them a year or two later. Personally I often find those chunks of code easier to understand than a tangle of nodes. But you can do that kind of optimisation over a longer time period. Bear in mind that this is probably my 4th major rewrite of my home automation code.

Anyway, that's just my take on things. Final advice - don't over optimise - but do have fun!

[Steve-Mcl]

@cerietke

Agreed on partitioning the flows, but that's kinda where I run into challenges. In traditional programming I can essentially create as many levels in a hierarchy as I need, but in Node-RED it's essentially limited to flows, subflows and perhaps groups (thanks for mentioning those, I should check that out). As such I don't really have as many hierarchical levels as I would like

You should checkout the link call. I find it more manageable than subflow & they can be easily linked and nested.

Here is a basic example: Feasibility of implementing functional operators without javascript - #8 by Steve-Mcl

[cerietke]

@TotallyInformation

Honestly, I wouldn't worry about the few ms difference as you will never notice that difference. Even on my old Pi2 which is where I started running Node-RED with Mosquitto and other services, I struggled to ever make the thing break into a sweat.

It wouldn't be performance I am worried about as much as delays (how long 1 message takes if it keeps leaving NR to go to MQTT to be picked up again in NR vs a direct link within NR). Perhaps though I am worried about something that wouldn't in practice be noticable.

@TotallyInformation

True. Though again I think you are over-thinking things. You can be subscribed to dozens of topics even on a really limited computer and never notice the load even with thousands of messages going through Mosquitto per second.

I was thinking more about flexibility, as MQTT retains a value I would just connect (again) to fetch the current value and have no need to store that value within NR. With all the values already in MQTT anyway writing some dashboard for my M5Stack Core2's could become easier.

Perhaps I end up not using it, but I like having the option.

@TotallyInformation

Hmm, I hope that is clearer though I'm not sure it is! It is much harder to describe than to do.

I think I got what you were getting at initially. It's exactly what I am thinking about: how to split up the code in ways that keep it more maintainable by giving everything little bits to do rather than large complex stuff. I probably need to sit down and design the structure I would want, something similar to this:

@TotallyInformation

Lets say LIGHTS/LivingRoom, LIGHTS/Landing, etc.

Possibly with the more detailed information I would want for the dashboards as well.

@TotallyInformation

Because the Zigbee devices and sensors are all used via Zigbee2MQTT

(I've been using Phoscon, but this is something I should possibly look at too)

@TotallyInformation

Bear in mind that this is probably my 4th major rewrite of my home automation code.

This is looking to become my 3rd. If possible I would prefer to avoid a 4th ;-). I'm not against using JS, I sometimes use it at work within the system we use to track our bugs and stories, but I am better with PHP. That said I already have a few functions for where something wasn't easily done in nodes.

@TotallyInformation

Anyway, that's just my take on things. Final advice - don't over optimise - but do have fun!

Agreed, but sometimes a more flexible approach may actually take less code, so I like to spend a lot of time thinking over just diving in and finding I could've done better. I like the MQTT approach you suggest though, I could see it work well for me. Additionally I could just keep what I have now and try it for something new like the music control I mentioned.

@Steve-Mcl

You should checkout the link call. I find it more manageable than subflow & they can be easily linked and nested.

I was already using In and Out, but I wasn't aware of the extension made to them through the link-call. That looks very useful!

[TotallyInformation]

@cerietke

Possibly with the more detailed information I would want for the dashboards as well.

I do have a lot more information, I just didn't want to confuse things further

@cerietke

better with PHP

:shudders-dramatically: Haven't used that since I chose to move away from it and over to Node.js - probably over a decade ago now. The main idea being to have one language for both front- and back-ends.

[cerietke]

@TotallyInformation

:shudders-dramatically: Haven't used that since I chose to move away from it and over to Node.js - probably over a decade ago now. The main idea being to have one language for both front- and back-ends.

Well, I was last officially a developer in '07...

I am not going to defend PHP, I am ill equiped as I last used 5.something and it's up to 8 now I believe, but, doesn't that make Node.js likely to end up in a Jack-of-all-Trades-Master-of-None situation? Or, if nothing else, less clean separation of different layers? In any case I guess that's more of a discussion for PM.

Appreciate your responses, thanks.

[TotallyInformation]

@cerietke

doesn't that make Node.js likely to end up in a Jack-of-all-Trades-Master-of-None situation?

Well, if I'm being honest, I don't very much like the direction JavaScript has been taking recently. Some very clever people are adding massive complexity to it. Much of which is very ill thought through. And the speed of change is such that many browsers and applications can't keep up with the rate of change. A poor show all round.

However, it does work - and mostly quite well - in both node.js on the back end and in the browser and you can build desktop apps as well and even mobile ones. So regardless of its failings (and they are surely many), there is nothing else that comes close.

As for mastery, well it has come a long way and actually there isn't much you can't do, at least as a scripting language. And performance is even such that you can build really complex apps with decent performance.

Perfect, certainly not, but useful certainly yes.

@cerietke

less clean separation of different layers

Personally, I don't see using the same language at the different layers as a problem for separation at all. Web apps already have a very clear delineation between client and server. Also, microservices is an approach that has always been heavily part of Node.js so back-end layers are also well defined for multi-layer systems.

[cerietke]

Just wanted to report back to say I did start to put more on MQTT. I created the music playback control solution that way and like the way of working. I do need to overhaul quite a lot before everything is done that way and have some minor bugs right now as a result of leaving parts of my older code in place, but didn't have time to replace it all.

I would prefer if there was a way to fetch a value from MQTT on demand, even if I fetched it before, but I reckon I should accept it is just not a database.

Thanks for your thoughts.

[TotallyInformation]

@cerietke

I would prefer if there was a way to fetch a value from MQTT on demand, even if I fetched it before, but I reckon I should accept it is just not a database.

I know what you mean. I've long wanted a context store in Node-RED that outputs events on-change since you have to have the variable in Node-RED as well as MQTT if you want to be able to reference it at times other than when it has been updated. That way, you wouldn't need MQTT for simple requirements. Just not had the time to sit down and work out how to do it effectively.

[dceejay]

@cerietke

if I fetched it before

If you fetched it before, then you already know what it is, as you would get an event if and when it changes.

[ScheepersJohan]

Do you mean something like this? Store the last mqtt message in a context flow variable, so when you have made changes you can just inject the last message from mqtt into the flow by recalling it from a inject node.