Hi,
Appoligies in advance because these questions may seem a bit simplistic.
Backround:
I have built a few projects with ESP8266 and arduino that send data to a webserver.
I use simple HTML PUT and GET to php pages on the webserver, which then gets transfered to a mysql database.
I'm far from an expert on any of this ( I have no formal programming or network training) .. but these systems work and seem easy to create.
It has been suggested to me that I should consider using MQTT and NODE-RED.
Could someone explain the the differences and benifits between my system and the proposed system.
( I've seen large amounts of turorials and on Mqtt and node-red... but very little that explains how these are better than my system)
Is there better security with MQTT /Node-RED... if yes could you explain why and how ?
I would much appreciate your insight to this .. and please bear in mind I'm not an expert on any of the topic above so am just trying to educate my self.
You get a lot more capability with Node-RED. You are able to run logic on a Pi or better and integrate with cloud services, save historic data in databases, run more powerful dashboards and much more. You have many more integration options.
Node-RED also gives you more options for security, including https, firewalls and the like.
MQTT lets you disaggregate your IoT devices and compute capability in Node-RED. It makes it much easier to move things around and expand your IoT network over time. It also makes processing easier in Node-RED since you can subscribe to a topic and only process when changes happen.
I agree with the comments made by @TotallyInformation especially if you are thinking of experimenting with IoT or converting your house to a SMART home. Using Node-RED running on a RPi makes a great central processing hub and using ESP8266 (e.g. Wemos D1 Mini) as slaves is a perfect combination as you can capture information, send it to Node-RED (probably by MQTT), process it, display it, store it - the choice is yours.
I also understand there is a great forum to help you along this journey.
Thanks to Totallyinformation and dynamicdave for you prompt replies..
... am i correct in saying that all the benifits you mentioned in Node-RED , are also available using HTML / PHP /java etc ...... it's just that Node-RED has so much of these ( modules) ready to-go , where as in php/ jave etc... all these features these would have to be written from scratch ?
Effectively Node-RED is a pre-made package of modules , with a new and different IDE ?
as regards MQTT, I still havnt got a good understanding of it what it is and what it compare to ,
As regards my project ( ESP8266 sends data to Raspberry pi webserver via HTMP/PHP and this then stores to Raspberry pi (same) sql database ) ... if I switch my project to MQTT, what aspect of my project would change and what benifit would it give me over the current setup ?
I really recommend to have a read of the hivemq mqtt basics series which will give you a very good idea of what mqtt can do for you:
Some points which make mqtt so useful for iot are:
a asymmetric publish and subscribe model, so you can have more than one subscriber act on for example the publishing of a new value by a sensor or similar
its a very lightweight and robust protocol that is easily integrated into a plethora of services/platforms
the ability to work with retained messages which for example enables a subscriber which was offline to still receive the most recent value when it connects again
things like last will messages/topics when a publisher goes offline
due to the flexibility of the topic based publish subscribe model which can include things such as wildcards and multiple levels its very easy to create a ordered system even for complex ontologies
Thats just some of the features of mqtt that let to such a widespread adoption in iot/industrial/maker solutions.
So its just very suited for this task.
When it comes to nodered, its much more than just another ide with a lot of modules. As nodered is a combination of both a graphical way to build processing logic and wire together different tasks but also gives you all the flexibility of writing actual javascript code when you need it it makes a lot of tasks very quick to prototype. At the same time at least in my experience it is also a very stable solution that can run month on end without any interruption. The contrib nodes are in many cases much more than a simple module but actually often build a simple interface and abstract away much of the hassle that working with the modules or functionality they wrap provides and even in many cases add extra features. And it has a great community behind it that is one of the most positive and helpful ones that i ve been part of.
At the end of the day its very easy to try on a Raspberry Pi or similar so you could just give it a spin to check it out. It’s definitely worth it.
Part of your question is like asking... "What can I do with a Swiss Army Penknife?"
Node-RED has a number of core nodes to perform most of the basic things people want to do. This is supplemented by a huge and ever growing set of additional nodes, developed by enthusiastic people, to support all sorts of situations. There is also a 'function node' that allows you to drop into JavaScript and write small pieces of code that are specific to your project/application.
I use Node-RED in two main roles: tutoring my students (11 to 15 year olds) in IoT techniques and gradually converting my house into a SMART home. In both instances Node-RED has been a brilliant platform as it allows my students and myself to experiment, lash things together, make changes / improvements - whilst having a lot of fun. I have always believed there is a strong link between learning and enjoying yourself.
So to answer one of your other questions about switching your project to MQTT and Node-RED... yes you would have to make changes and there is a learning curve to climb (like any new subject). The thing to remember is... you don't have to do it all at once. Maybe start by capturing data from one ESP8266 using MQTT and looking at the data in Node-RED (Debug node and/or the dashboard). Once you have mastered the basics you could move on and write data to a database and then start drawing graphs - beware... you will probably get addicted to Node-RED like a number of people on this forum!!!
and of course you don't have to go all-in to start - Node-RED can also be an http endpoint so you could just start by doing that part so the sensors can stay http - then investigate mqtt as a next step - etc.
Ok, thanks to all for your inputs ...
I think I have an undestanding on Node-RED now ... and agree is is much more than an IDE with pre-made modules ....and I can see it's benifits now .... I just wanted to be sure my understanding of it is correct .
As regards MQTT, I'm not fully sure i understand this yet , and admit I have some more reading to do ... thanks JGKK for your reading reference ..
On the MQTT subject , as 'dceejay' suggests , is it correct to say , MQTT could replace my Http in my project?
if yes, could you give more detail on the benifits of this ?
.. I have a suspicion ... but dont know enough about it ... but because http uses port 80, then this is a risky ( from security point of view) ... and MQTT uses something else ... that is more secure .. would that be correct ? ,
Is that that the main reason to go to MQTT ?
If yes, then could someone explain why usig http - port 80 is such a security risk .... and then, what does MQTT do differently that makes it more secure ?
No. MQTT is no more or less secure than HTTP. They are both protocols for communicating with servers. They both have authentication mechanisms built in to ensure only authorised clients connect, they both rely on TLS if you want to encrypt the connection so the data can't be read off the wire.
MQTT provides a different way of exchanging messages between a client and a server. More correctly, MQTT provides a way to exchange messages between different clients all connected to the same server (or 'broker' as it is called in the MQTT world).
MQTT uses a publish/subscribe model of messaging. Clients "publish" their data to a "topic" on the broker. Other clients "subscribe" to that topic - so when one client publishes their data to it, the subscribed clients will receive the message.
With HTTP, your client will send an HTTP Request to the server containing the data, and it is up to your server that receives that request as to what happens next.
One of the advantages of MQTT in this scenario is you can build your application with your clients publishing to a topic, and another client subscribed to receive the messages and put them in the database. If in the future you think of something else to do with those messages, you can add a second subscriber that will also get the messages - without having to touch the first subscriber.
MQTT clients establish a single TCP connection to the broker and keep it open - exchanging messages over it. With HTTP, you have the overhead of establishing a new TCP connection every time you want to send a message.
There are lots of other useful things about MQTT - definitely worth reading through the HiveMQ link above to learn more.
But to reiterate what Dave said, you could start be reproducing the PHP code you have to create the HTTP end points in Node-RED. Once you get that working, it's a relatively quick task to swap the HTTP input nodes for MQTT input nodes.
Ok - great - thanks very much for all the information above .. it does certainly make things a bit clearer..
I'm going to do as suggested and Start with Node-RED and reproducing the PHP code I have to create the HTTP end points in Node-RED...
once I have the knack of it .. I'll then look at MQTT
Maybe also worth mentioning about MQTT I have understood the protocol itself has much less overhead compared with HTTP, means it is faster in transferring data. It also handles re-connections in case you would lose such. You can also set the QoS (quality of service) for connections/subscriptions. This is not available at all in HTTP
Slightly more than that. It isn't just that there are lots of ready-made modules for Node-RED. NR itself fosters rapid development and minimal coding. This makes it a very fast prototyping tool. Whether from the core logic using flows or rapid HTML development using Dashboard or uibuilder, creating custom processing and UI's with Node-RED is vastly faster than other platforms - I know, I've done most of them over the years - but without the walls that something like HA throws around you.
Your devices will talk via MQTT, this will overall simplify the interconnectivity since you will have a single interface for every IoT device (using an existing library to assist, all you need to do is configure a different device ID that you can pass to MQTT) and a single (pre-built) interface between Node-RED and MQTT that triggers flows when a new piece of information comes in from an IoT device. Nicely applying the DRY principle.
Using MQTT seems like an unnecessary overhead when looked at from the outside. Once you start using it, you will quickly see the value. At least, that seems to be the way for most people and it certainly was for me.
It will replace the HTML part where the device needs to interface to the Node-RED server, nothing more.
You might still want a web interface on a device - maybe a simple one, up to you, for people to link to directly, not really needed but can occasionally be useful if something goes wrong with other parts of your setup. Otherwise, any user-facing interfaces can now move to Node-RED where you write the interface once and plug in the appropriate device ID with the data coming from and going to MQTT.
As Nick comments, Port 80 for HTTP and an MQTT default port are both equally insecure. For a home system, you should be avoiding such ports being accessible direct from the Internet otherwise you can guarantee someone or something doing nasty things to you.
Device <-> MQTT <-> Node-RED ... Node-RED (web user pages) -> Reverse Proxy -> Firewall -> Internet
is rather more secure.
Because it is unencrypted. That's to say that all information that is exchanged is visible to anyone who can intercept it. In a home automation environment that might include when you are home or away. Or might let an attacker turn on/off your lights or control your heating (turn it up to max while you are away?).
In truth, those kinds of attacks are fairly rare. What is a lot more common is someone using a security flaw in your server to gain access to the system itself and then use it either to lurk on your network sniffing other interesting traffic like your finances, shopping or whatever. Even more likely is that they hijack some of that compute power to conduct further attacks elsewhere on more lucrative targets - that is the most common type of hack. Such onward attacks are massively damaging and costly to other people, a story for another day.
Not sure about the absolute overheads but it is certainly tuned for exchanging relatively small amounts of data very rapidy. It also, of course, has features that help ensure that messages are successfully delivered including queueing methods and message acknowledgements.
