Which MQTT broker (self hosted vs. Mosquitto or others)?
Which Linux? Command line or with Desktop GUI?
See Best Linux Version to Node-Red for a discussion on this. I am surprised that you did not see that post, presumably you did search the forum before asking the question.
Whether you want a gui or command line install is up to how you want to use it.
Mosquitto. Self hosted.
Many thanks
Which linux has been addressed many times. I won't come back to it.
If you need 24/7 operations, then having a redundant broker is (for me) key.
Mosquitto is an excellent choice, unfortunately, the free and open source version can't be clustered.
For clustered broker, EMQX as free and open source is an option.
RabbitMQ for MQTT only is very complex.
VerneMQ is a very good choice, but, you'd have to pay for a licence.
In front of those, you will need a (redundant) load balancer.
HAProxy is a good option.
My 2 cents...
Can you explain why? What is the benefit?
Well, your unique instance of broker may fail (the software, the host were it runs...), it is a SPOF.
To avoid that, having redundancy active/standby or else is key.
For MQTT, the clustering is a way to address 24/7 operations.
But if your node-red has failed then what is the point of having a standby mqtt broker? So that is also a single point of failure, as is your influxdb service.
1 Like
I would agree that if you really want guaranteed uptimes, you need to think about end-to-end resilience.
But as usual, this is totally dependent on the background requirements - which haven't been offered to us as yet.
So we really need to ask for more background info before recommending something.
In my app, I have an MQTT cluster (VerneMQ), Redis in cluster, and NodeRed runs in docker and I have two instances running in actve/passive mode.
I agree with Julian, all is very context dependant...
FlashMQ - better than Mosquitto
Depending on your needs.
But OP's subject is 'data logger' and that is not what MQTT does.
Ok, I have a story which needed a log to help me win the battle.
My IP would change daily anywhere from twice to eight times.
I rang my PROVIDER and they tried to dismiss my claims.
I sent them my logs. 4 years of data of my IP address and the UPLINK status.
Logged every 20 seconds for 4 years.
Yes, there were holes in there now and then. You can't really work around power losses.
Alas my solution is to log the data locally.
Though I do use MQTT to share information between my 3 machines.
I found a lot of problems in code I wrote too - though now I digress. Sorry.
I only have ONE MQTT broker and it worked for my needs.
Though I do agree that TWO may be better.
Though if the network goes down, how ever brokers you have: they won't work.
I'm afraid that needs some explanation if you want to recommend something other than a tool that has been battle tested by many over a very long period. They don't even include Mosquitto in their benchmark comparison. It also doesn't seem to be open source and only provides Debian Linux versions. It also appears to be just a couple of years old.
Mosquitto is open source, backed by the Eclipse Foundation, Has GitHub releases going back to 2014 and is open source. And has the benefit that a lot of people on the forum know it well.
I was thinking about this recently and thought suddenly of Kafka. Kafka has the ability to store messages that are sent by a producer. Consumers can rewind and reload old messages or listen to the current messages coming in.
The basic idea is that a producer just does fire&forget and Kafka does the storage of messages and sending out of messages to consumers. This allows the producer not to worry about message delivery or loss - Kafka ensures the consistence and preservation of data.
Data loss is avoided because Kafka keeps a store of the last X days/weeks/months. Any new consumer can begin at the beginning of that store. The Kafka protocol ensures that messages are indexed in order so that rewind is possible.
Question is does MQTT/Mosquitto offer this feature too?
I came across FlashMQ through Victron's OS (Venus), which replaced it some time ago. I've also had a few contacts with the developer, nice man, very helpful.
I have now switched all brokers at home from mosquitto to flashMQ
MQTT v3 has some basic controls around guaranteed delivery. MQTT v5 expands on that considerably and adds capabilities for message receipts as well. But it is an MQ service and so not expected to record history and you should attach suitable, MQTT enabled data recording if you need that. Kafka has significantly higher requirements and overheads than an MQTT Broker. There are other MQ systems that handle larger message queues and multi-step receipts and other features. But again, these tend to be enterprise scale tools with appropriate complexity and resource requirements.
The purpose of MQTT is to be lightweight and simple. Allowing brokers to run just about anywhere and still maintain high throughput rates.
1 Like
I confirm what Julian wrote. If you use QOS = 2, then it gives some confidence that messages sent will be received. If the client (subscriber) is offline for a while, then with QOS = 2, messages kept will be sent. However, in all broker implementation, there is a limit on the number of messages that are kept "in the fridge" when the client is away. This is why I wrote "some".
MQTT is a data exchange protocol. Not a database.
1 Like
