MQTT vs HTTP request

Hello to everybody,

I have got a Raspberry Pi 4 with Node-RED installed. I have set a lot of sensors and I am sending data to my server by means of a HTTP post request. I have on my server both Grafana and Influxdb.

I tried to configure MQTT communication, but I couldn’t get it to work. Can anyone recommend some tutorials to help me with this? Then, someone told me that mqtt communication could be better than http because in the mqtt case there is a bidirectional flux of information between client and server. Ok , well, but why is this important and in what circumstances?

Thanks

With HTTP requests, generally each request must make a handshake with the web server which may involve several interactions. And then, the client has to request everything from the server. There are some more complex methods that seek to lower these overheads but they can be more complex to set up and typically they all require TLS encryption (e.g. HTTPS) to be configured as well which comes with its own overheads and complexity.

The final issue with HTTP(S) API's is that they are many-to-one connections (again, there are ways to reduce this but at significant added complexity). Many-to-one means that many clients all connect to the single web server directly and, in the case of Node-RED's web server, each client connection must go through that handshake each time to make the connection before being able to send or request information.

MQTT, on the other hand, was specifically designed for lightweight, high-speed interactions. Indeed, it was born out of the birth of IoT, so specifically for the use-case you have mentioned.

MQTT is a "Message Queue" and works on a submit/subscribe basis. That means that your clients interact with a "broker", the MQTT server. You can submit messages but also (over the same connection) subscribe to "topics" and be notified when the topic gets a new message.

The initial handshake is typically very lightweight and fast, much faster than the typical HTTP(S) handshake. Indeed, connections will generally stay open once made which means that clients don't need to keep reconnecting as long as there is some kind of regular interaction (sometimes referred to as ping/pong).

Another advantage of MQTT is one you mentioned. MQTT has a limited set of response messages (much more so in v5). These are able to tell the connected device, for example, whether data they sent to the broker arrived successfully. There are message settings that also help you have guaranteed messages and even ones that can be retained offline until reconnection. Also, the broker can have "retained" messages.

Another advantage is that using MQTT decouples your client devices from your data processing server. The broker is a message hub - think email for machines but a heck of a lot more efficient. Brokers can also talk to each other which can easily create interesting messaging meshes.

The potential disadvantages of MQTT are:

  • Another server to set up - however, the most popular broker is Mosquitto which is crazy small and efficient. You can easily run it on even an early generation Pi along with Node-RED and other tools. Even with that configuration, it will likely handle many thousands of messages per second.
  • Typically, the data you exchange with a broker is likely to be a lot smaller than you might with a web server. This is generally not an issue. Not only are IoT messages often tiny, you can easily split more complex data down into individual topics and messages.

Check out here to learn the fundamentals of MQTT, it is far better explained than I can:

Using a broker gives you a high-performance, very highly robust message hub that won't be broken or impacted by some rogue code in your compute server (Node-RED). Even if Node-RED crashes, the broker will continue working.

I should also point out something of a compromise mechanism called Web Sockets (ws). Ws initially connects over HTTP(S) and then "upgrades" to WS(S) which is a persistent connection rather like MQTT. Node-RED, the Dashboards, UIBUILDER and similar nodes use ws to ensure efficient, near real-time communications. However, you are still totally reliant on Node-RED's web server and have no fallback if that server crashes or has performance issues.

2 Likes

Could you explain the part of your message that says.. "couldn't get it to work".
What error message did you get?
Can you tell us what didn't work?

I assume you have access to an MQTT broker (either locally or remotely)?
If not, you'll need to install a broker such as Mosquitto on your RPi-4, or gain access to a remote broker.

New users of Mosquitto are commonly tripped up by the fact that by default it will only accept connections from localhost. Not difficult to fix but surprising considering it's a communications protocol!

Tutorials such as https://randomnerdtutorials.com/how-to-install-mosquitto-broker-on-raspberry-pi/ show how to set it up.

@TotallyInformation has given you a very comprehensive overview of HTTP v MQTT.
I suppose if you are comfortable with exchanging data over HTTP it's not a problem, but for me the amount of HTTP crud surrounding your nugget of actual data is mind boggling.
Of course MQTT must have similar kinds of metadata but you don't have to create it yourself. You don't even see it.

1 Like

It is actually a good security position. It means you don't accidentally open up the whole broker. But yes, if you are running services across multiple devices, you will need to know a tiny bit about networking.

Unless you want to of course. :smiley: MQTT v5 has some much more complex capabilities. Thankfully, you don't have to worry about them unless you need them.

I've worked with some different use cases of MQTT and they vary wildly. It's more similar to websockets in that there is a live connection that can stream data back and forth. Both come in encrypted HTTP(S)/MQTT(S) variants. MQTT may also be used in more traditional API style with request/response. The documentation may more often be lacking than what's typical for HTTP. I don't worry so much about performance or technical overhead, most if not all of that is in the background and rarely something you have think about. And shouldn't be an issue unless you're getting millions of requests. However HTTP do provide super useful status codes, while MQTT is awfully quiet about reasons why when failing.

That is OK for low-volume HTTP interfaces but not so much for high-volume. Especially if not using Node-RED without a reverse proxy to take load off the node.js process. Node-RED on a constrained device with lots of connections will quickly get swamped if trying to rely purely on HTTP(S) and while also trying to do compute and possibly I/O tasks at the same time.

A reverse proxy can certainly help by offloading cached resources and potentially dealing with the TLS and other handshakes. But something like Mosquitto doesn't even break into a sweat when handling 10's of thousands of messages per second.

So while "rarely" is your current experience, it won't be everyone's.

And that is largely because it is a different type of transaction for the most part. The vast majority of MQTT transactions are fire-and-forget and may not matter if the message doesn't get through because another one of the same type will often follow seconds or a minute later (typical IoT sensor outputs).

The client and broker also use the QoS settings to try to meet expected message reliability and can queue up messages until they get a confirmation ping.

MQTT v5 does have the ability to deal with more complex responses.

So there is a lot you can do with MQTT beyond the basics. However, the entry point is really easy. But like anything else, it does help to understand some of the basics and some best practices. Thankfully, there are a LOT of people on the forum who are well versed in MQTT. :smiley: And HTTP for that matter.


Bottom line is, if you need a direct one-off interaction between a user and the server, HTTP(S) may well be a good choice. As it would be if you need to transfer a large chunk of data such as a file.

If you need to fire ongoing status updates and want the ability to create an event-driven process flow, MQTT is a great choice. Especially with large volumes of relatively small messages.

If you want a balance between the 2, then web sockets may be a good choice. Though for greater reliability, Socket.IO is generally a better choice. It can use web sockets but can also fall back to HTTP(S) polling. This is how UIBUILDER and both Dashboards communicate between the clients and Node-RED.

The good news is that Node-RED is very well served on all 3 fronts!

I connected a mqtt in node to a debug node and some sensors connected to the mqtt out. It worked but I wanted to send data from my sensors to my cloud server by means of mqtt, is it possible to do that? I want to send data to my server influx db. I have got on Node-RED influxdb out node and I usually send data to influxdb by means of http post request, this time I wanted to use mqtt but I did not manage to do that.. Maybe Have I to install mosquitto on the server? Thanks