I'm sure others have discovered this, but I don't see a recent mention so thought I would share this tip.
I use mqtt for all the standard purposes, but also as the core of my flow and device monitoring. All important flows send their key data (input data, variables, decisions, etc.) to mqtt. Sometimes they send a big msg object containing tens of parameters. I then use mqtt explorer (http://mqtt-explorer.com) as the UI to view trends and details. It's a simple "single pane of glass" approach to keeping tabs on everything. I use grafana also, but for a different purpose. For debugging, I find mqtt ideal.
I also have non-Node Red automation and systems (AIS receivers, ADSB receivers, etc) that dump their logs into mqtt by piping stdout to mosquitto_pub. It's easier than setting up a syslogd server.
Indeed, the output of my Telegraf system sensors not only goes into InfluxDB for historic reporting but also direct to MQTT for use in Node-RED and anywhere else I might like.
Similarly, I also have a custom Node-RED logger output configured in settings.js so that I can get trace level debug output for uibuilder but more reasonable logging for everything else - that also goes to MQTT and I have a custom debug webpage (using UIBUILDER of course ) that shows everything and is generally nicer to manage than trying to do it in a terminal window.
This sounds interesting. Do you log to dedicated mqtt server, local to each instance? If using cloud server to log mqtt, then edge instances won't log anything if network goes down or some process goes down? I always logged to file on the device directly. It might not be as easy to look through a bunch of files than this mqtt example? At the same time I always get the logs unless power shuts down, process shuts down or disk is full. Which are so huge problems you don't really need logging at that point.
No, my setup isn't large enough to warrant that. Also, I do my best to ensure that everything important still works if the server or its service go offline for any reason. Otherwise the wife would get super annoyed and I really don't want that!!
You will certainly want a local MQTT server to be honest. Your Internet is far more likely to go down than anything else (as long as your local power is reasonably good or you have local backup power).
Personally, I would never log to a cloud MQTT broker, especially a free or low-cost one. The host will almost certainly have a level of access to the data and with logs it is really hard to make sure you don't log anything sensitive.
If you absolutely must have logging in the cloud, you need to make sure that the host infrastructure fully protects your data and this is much more likely when using a log aggregation service rather than an MQTT broker.
The other issue with logging to the cloud is the amount of data you might be sending. This is typically costly.
What I would do - and partly I do - would be to have alerting to the cloud instead of logging.
Ideally, you would have a service somewhere in the cloud that pinged your local server and alerted if it couldn't reach the server after a given period.
I simply do alerting locally using Node-RED to Telegram. That's on the basis that Node-RED itself is very stable and unlikely to fail. But I have had problems very occasionally with other services failing after an auto-update (I'm too lazy/busy to keep the server updated manually).
Local logging should be done to the servers system logging function so that you can use appropriate tools for viewing/searching/etc. On a Pi, that is syslog. MQTT Logging all tends to end up on a single or a few topics and so is totally transient. It can be used for alerting or for sending to a live web page, both of which I do. MQTT is not actually a logging tool as I'm sure you recognise, it is simply a convenience for ease of alerting via subscriptions and ease of viewing via output to a web page.
That makes sense. So a prerequisite is already having a local mqtt broker. Always interested in convenient ways of logging. At the moment I stick to log files, as I don't have that local mqtt broker.
Yes, its good to think about alternatives occasionally. I wouldn't use MQTT purely for logging, thats for sure, but for the other reasons, it can be very useful.
Also worth remembering that a local Mosquitto broker takes hardly any resources so you can easily run one on a Pi along side other things.
You could have a local broker installed, configured and running within five minutes. Definitely the way to go. If you need remote access to your mqtt broker, consider zerotier as a vpn.
If you do install a local broker and have the networking kit to enable mucking with a local DNS, consider having an alias of "mqtt" for the host that runs the broker. This way, if you choose to move the broker to a different host, you don't have to update all your IoT and other hosts that use the broker.
You are thinking of it as a STORE and it isn't. It is a MESSAGE QUEUE. You can instruct the broker to permanently remember the LAST message of a particular topic but that's it.
A Message Queue is like email for systems, its purpose is not to act as a RECORD of activities but rather to facilitate COMMUNICATIONS of activities between different systems. That's why an MQTT service is called a "broker".
Mostly, MQTT will be used to facilitate immediate, real-time communications using the publish/subscribe pattern. However, it does have, as I mentioned, the ability to retain the last message on any topic so that a new subscriber will get that last message.
MQTT is absolutely the WRONG tool for logging. Use a proper logging tool for that. Such a tool will manage security, retention, log rotation and trimming, etc. It can be used for retrospective analysis as well as seeing immediate activities.
Do use MQTT to enable something like Node-RED or other systems to subscribe to log updates so that it can do things like alerting without the need for a more comprehensive logging extension for such things as you would expect to see in an enterprise setup (where you would have log aggregation services that feed up into your IT Service Management and Cyber Security Management tools).
As TotallyInformation mentions, mqtt can store only the most recent message (depends on the QoS set by the publisher), but if you keep the mqtt explorer GUI open, it does store quite a few-- maybe the 100 most recent? For debugging and generally trying to find out why some piece of automation is running amok, I find this is sufficient, but it wouldn't replace traditional logfiles (and the need for their rotation/truncation). I can't praise the mqtt explorer tool enough.
) that shows everything and is generally nicer to manage than trying to do it in a terminal window.
