Need some opinion on smart home creation

Hi there,

I would like to set up some sensors around my home and get the necessary data on my android phone. In other words, make a hobby-type smart home.

I have a few basic sensors and PI3 as well as some Elegoo. I will use a node-red dashboard so data can be seen on the phone.

I am wondering about the node-red.. are there tutorials with flows available from other members? That's usually the fastest way how I could learn and then doing some adjustments.

Perhaps you could suggest something?

Regards

a lot of people use "home assistant". I don't.

I missed the memo when it was sent.

But basically you need to write some code which gathers all the sensor inputs and then start to work out how you want to show it.

That is a whole other thing to getting the data.

Have you looked here? https://flows.nodered.org/

And the docs also have examples here ... Documentation : Node-RED

Part of the issue is that home automation setups in Node-RED tend to get complex and rather specific to the originator over time. So they are hard to share.

If you want a simple setup with less admin, use Home Assistant or something similar. If you want fun, control, adaptability and flexibility and to learn how things work - then you are in the right place here :smiley:

If you decide to join the fun, there are a few things to think about ...

How will you manage your sensor and controller devices and link them to locations around your house. The easy way to start is not to worry about it and jump straight in. Add some sensors and build a dashboard. And you may find that is all you ever need to do. But at some point you will probably reach the point where you no longer want to rejig flows every time you want to change a sensor or controller. At that point you will find yourself unpicking your spaghetti and disaggregating source sensors from target controllers and setting up small data structures for doing things like mapping your device ID's to real locations around the house.

I really would encourage you to set up a couple of sensors and maybe a couple of light switches so that you can have an initial play with things. Then start to think about how you want to manage the devices, what metadata you want to manage in relation to the devices. That is also the time to start thinking about historic data - do you want to track sensors and switches over time? If so, you will probably want to start learning about InfluxDB and Grafana.

Best advice I can give is to keep inputs and outputs on separate flows (disagreggated).

Capture inputs and normalise the data as far as possible so that inputs use common property names and value types and you add timestamps where the sensor doesn't provide them.

Feed those through to MQTT using a standard topic naming convention such as DEVICES/<deviceid> and maybe a copy to InfluxDB where you want the historic data. You could also accumulate that data into a global variable as well for later use though I'd probably hold off on that until you actually need it. That is one distinct set of flows. You now have the data available for whatever processing you want to do.

Next set up flows to output data to controllers (switches, etc). But don't connect them directly to any input flows even if you intend to use a sensor (e.g. light or heat) to control the output later. Drive them from the MQTT topics or CRON+/Inject nodes instead. That will let you more easily add other processing later when you need to.

You should now be able to see how easy it will be to add more sensors and controllers with minimal effort - sometimes no effort at all.

You might at this stage, also want to build a generic flow to output to other alerting tools such as Telegram, email, SMS or whatever is convenient. That way you can pass whatever data you like using standard msgs, let the flow reformat the data to the required output format. Reusable.

Then you can add more complex control flows - such as building some alerting/alarm features by listening for certain sensor events and outputting to both your alerting tools and to a controller (e.g. lights or a sounder).

Then you can start to build other flows that get data from other sources such as weather or news. Now you can weave that into your output flows as needed or simply send to your dashboard for information.

Node-RED is so flexible and has so many options for inputs, outputs and processing that there really isn't one way of doing things. You have the joy of building something that both works for you specifically and gives you the enjoyment of the design and build not just the using.

3 Likes

This kind of project is exactly what I'm working on, and I think @TotallyInformation has summed up most of the things you'll need to think about. Particularly his comments on keeping flows simple, separating input and output parts of the system, and using a standardised naming convention for messages, both over MQTT and internally.

My approach is using a simple software framework running on ESP8266 and Raspberry Pi Zero devices, and adding just the "module-specific" functionality. The same could be adapted to Arduino-based modules as well. I've kept each module to a single function, or a few closely related ones, such as room environment (temperature, humidity and light level), motion sensing or lighting control, which simplifies the design of each. Mixing multiple functions on a module could lead to complex and difficult to maintain software.

I've sort of "classified" the functionality into major areas including heating, lighting, security etc., although there's a bit of overlap in some areas. Again, you might want to have a think about how the different parts of your system are going to interact as well.

All communication is via MQTT with JSON payloads, which allows for future changes to message data without needing a lot of code or flow rewriting.

I've still got a lot to do even with the basic system, and have plans for a whole range of modules to add to it eventually, but the approach of keeping things small and simple makes the (never ending) project manageable.

That is also what I'm moving to. I've used Tasmota and EasyESP in the past but I'm slowly replacing those with my own code that does just what I want it to. Once set up, it never needs updating because it works. My own sensor framework uses my favourite sensors with temperature, humidity, light and sound level being fairly standard. The code evolves slowly over time but really I don't ever need to update it on a sensor platform unless one of the parts dies (never have yet). It really isn't that many libraries and bits of code to have all of those sensors on a single ESP8266 and I can use compiler switches to turn on/off sensor code for each device as I compile. There aren't so many that I have to worry about managing different versions and device types.

Things like PIR, and open door sensors I tend to buy as a package, up to now, they've generally been HomeEasy EU based as I have a RFXtrx433e so they are fairly cheap and they tend to look nicer than home-grown packages (I don't have a 3d-printer). The environment sensor platforms are ESP8266 based - so far uncased since they get tucked away. I've collected various cases though and I'll mount them one day. :slight_smile: They are also all USB powered since I can place them in reach of a power outlet.

Really, the Tasmota/ESPEasy/etc type framework isn't worth it for me, the updates that are needed because of their complexity I don't really want. Also, I find that their complexity leads them to crash far more often than my own code which tends to be simpler and more efficient.

Mains control plugs and switches I purchase because I'm not confident enough to do my own for mains power and by the time I'd put in thermal fuses and other protection and then put them in a reasonable looking case, it wouldn't really save any money. I do need to learn how to replace relays though as I've had quite a few go over the years - I've kept all the devices though so I can repair them eventually :smiley:

I've done that in the past but I'm really simplifying my MQTT now because I find I don't actually need that much. Really all I use it for is comms to/from WiFi devices and comms between my 2 Pi's (older code) and my home automation laptop (which is replacing the Pi's). So much of the logic is now inside Node-RED flows and with the use of link nodes and retained variables, my need for MQTT is now much less.

One day, I'll write a variable handler for Node-RED that includes event handling and then the need for MQTT will be even less since that is the one thing that is currently hard to do in Node-RED (link nodes are good but not really substitute for a variable handler that notifies flows when something changes and a variable listener node). If someone doesn't beat me to it that is.

1 Like

@TotallyInformation - we seem to be heading in the same direction for a lot of design choices. I've looked at Tasmota and ESPEasy as well, but like you I wanted something cleaner and simpler. They're good frameworks for getting started, but way too complex with all the variants and options.

Rather than compiler switches, I just have a pair of .cpp/.h files for each different device, and include the framework as libraries (well, currently compiling them in from source, but will move to libs once I'm 90% happy with them).

I've gone with the JSON over MQTT to keep communications simple and standard. It's easy to convert and process message fields by name, and if I need to add a field it's simple to do in both the device or flow.

It's always evolving though, and half the fun is in the development process :wink:

I'm not much of a C++ coder :blush: and I only use the Arduino IDE, everytime I try to use something else like PlatformIO, I just find it too much bother to set up and I give up and go back to the Arduino IDE again.

Yes, same here. It is only NR internal flows where I've removed most (maybe 80%) of the MQTT as link nodes and retained variables are generally more than enough.

Absolutely, that's why NR is so much fun. :smiley:

1 Like

I suppose I've been coding in various languages and using different development environments for <redacted*> years now, so I've learned to adapt to just about anything :slight_smile:

Currently I'm using Visual Studio with the Visual Micro plugin, although I've played with VS Code, Eclipse and the Arduino IDE as well. They all do pretty much the same things, just some have more bells and whistles, so go with what you're happy with. (One of these days I should write up a comparison of IDEs.)

[ * > 50... ]

1 Like

Hi,

So I managed to play around with node-red. I managed to set up different sensors and see them on the node-red dashboard.

I would like to move to the next step and make each sensor independent, meaning the use of MQTT and sensors to monitor different parts of the home.

For example, I would like to monitor different rooms (movement, temperature, humidity, plant moisture level, etc.) so to achieve that I will use MQTT and Pi3 as the main data collector via the node-red platform.

Is this even possible? MQTT would give me the option to save on cables and options to use in different places at home.

Later I want to connect data with Power Bi, to visualize the data (history).

What are your thoughts, would my concept event work with MQTT, Node-Red, and Pi3 ?
Perhaps there are better ways of what I am trying to achieve?

This is possibly the most common use of Node-RED (or was until the uptick in professional use over the last 18m or so).

I have multiple ESP8266 sensor platforms around the house measuring temperature, humidiy, sometimes light, sound and pressure too. A couple of other devices feed back power consumption. These all communicate direct over MQTT to a broker and Node-RED consumes the data also from the broker.

I also have other devices that have 433MHz radios and so can't talk MQTT directly (they are pretty dumb - cheap - devices), these talk direct to a transceiver device on my home server where Node-RED is running. I generally translate these into forms similar to the ESP device outputs and send those to MQTT as well though I don't really need to since NR is doing all of the actual processing and recording to a database, showing dashboards, etc.

PowerBI is an interesting (and certainly overkill) way to produce a dashboard for your home :slight_smile: Worth noting that Excel has the same processing engine as PowerBI in the form of PowerQuery. So you can use that instead if you don't need the web sharing of PowerBI. I assume you have a corporate license for PowerBI that you are using since the paid versions are quite expensive. I do have a PowerBI Pro license but I wouldn't bother to use if for home monitoring myself. Many of us do, however, use the free Grafana tool which is a more open dashboard builder. Not as powerful in some ways but certainly very good to use, especially when working with timeseries data from InfluxDB.

1 Like

Hi Sorry for a late reply/interjection.

I currently do not use MQTT, not because I have an objection but because I have not found a need for it yet.

Could you please expand in a little more detail what it is, that is difficult in Node-RED w.r.t. MQTT ?

P.S. I too find grafana and influxDB very efficient. InfluxDB + RED on Pi with dashboard display and control and grafana on my desktop.

For unfamiliar users of influxDB, one thing to note in my experience is that if you in inadvertently save a field that later you don't like, it is hard to remove that field from the series data without trashing other data with it.

I think you will find this expanded in another recent thread. But in a nutshell, MQTT allows you to decouple your hardware devices from the Node-RED server. This:

  • Makes it easier to add/remove/move hardware IoT devices without having to change code in Node-RED
  • Makes it easy to run up a test or development instance of Node-RED and still get live data from your hardware
  • Have more robust communications thanks to the QoS capabilities of MQTT
  • Have potentially higher throughput and more reliable messaging since the MQTT protocol is tuned for this task.
  • Run Node-RED flows only on data changes rather than having to poll for changes from your hardware. No code is needed for this on Node-RED, only the mqtt-in node.
  • Use the same methods for talking to/from other Node-RED instances.

One other potential is that you can use other tools if you want to in order to get data from your IoT devices direct into InfluxDB via MQTT. Telegraf is a great tool for this and is from the same vendor as InfluxDB itself. Telegraf is a great way to capture system monitoring and performance data into InfluxDB but also has the ability to listen for MQTT topics as well.

Perhaps consider MQTT as the email server for IoT. It acts as a post office for exchanging information between systems quickly and reliably.

Thank you for the explanation. Is MQTT too heavy on resources for you to look forward to replacing it with your own event q ?

Reading one of @TotallyInformation's comments earlier up this thread, I am also feeling that the deeper I get into NR, the more bespoke my setup gets.

I've halted all development in NR and uibuilder as I'm starting to think about how to develop a single framework for my entire system:

  • automated lighting
  • automated heating
  • automated music
  • automated control of blinds, curtains, skylight/windows
  • centralised management of events data
  • scheduler for minor ad hoc tasks
  • mobile & email alerts / tts
  • other automations e.g. good night/good morning flows triggered from wall switches
  • all the UI required to view, manage and configure the above
  • graphical visualisations of data from the system

I don't want to delve into and develop one part of my setup only to find later that I am going to break it up again to fit the next part that I deploy.

When I think about it in its entirety it's a little daunting, yet also very exciting.

I'm wondering if anyone has embarked on this before and whether there are projects I've missed on how people have done it.

I seem to fall too often into being a victim of building something that I later find has already been built before... that doesn't necessarily stop me from charging ahead anyway though :wink:

@jordan
I'd echo the comments from above, start small, prototype different modules, different systems (if you have the time) and then make your mind up as to whether you REALLY want to go down the more bespoke route or whether actually, all you want is to set it up and use it. If you go down the bespoke route it has to be a passion / hobby as it's almost a bottomless pit.

MQTT is very light on resources. Why would one want to replace it?

Why? It will be hugely more work that using node-red.

I suppose if you are a web developer then you have that option.

I am not a web dev. (electronics and lots of real-time C) so I have found NR and Dashboard to be almost perfect in allowing me to do almost all of what you describe. This is not taking sides over uibuilder/dashboard, more that I think they match different skill sets. So my preference is to expand if I can on the capabilities within dashboard because I feel more secure in that environment.

Hey if you can do it then write your own platform. I would love to see dynamic menu with cytoscape or d3 for that matter.

Good luck.

Thanks that is why I asked.

Oops, I think I've mislead some people.

  • I'm definitely going to build the whole thing on NR/ubuilder/VueJS.

  • I am definitely not going to stop using MQTT - I meant that I want a "home automation" event handler that does basic stuff, like update global variables with latest values for other flows to use, trigger necessary flows centrally, maintain naming consistency throughout all flows by managing a device naming mapping table etc etc

I've just stopped developing on NR until I wrap my head around all my requirements and think of a solid design... attached is a spreadsheet I am using to log ideas. I just created it this morning, so it's still a work in progress.

I doesn't cover all the UI's I want to build, but does cover core system functionality.

Darn, can't upload XLS files, I guess macros as bad... here are some images... arrgghh, images are too small... give me a minute to figure out how to share the file.

Ok, here goes.. most of the file is there. I hope it doesn't serve to confuse further...

1 Like