My background is from programming, and the whole node red IoT world is kinda new to me. But I do have education and experience in networking. And there are some aspects that's not entirely clear to me. Normally, when I work with requests (whatever they may be) to some remote resource (whatever that may be), I have a system, a protocol or framework that includes some sort of receipt and error handling. Because sending a request is never guaranteed anything, right? Be it HTTP, I get a status code, or some sql connector, it will have some error codes too. But in the world of node red, I work with a lot of protocols and connections that seemingly behaves entirely different. I've had various experiences where no error code is returned, not even error message or a response at all! For protocols like bacnet, modbus or mbus. If I get a receipt, errors can be logged and monitored. I can discover ways to improve stability, backup, retry or otherwise mitigate data loss. Without this, I'm blind and can't do anything. How does other handle this? Or simply ignore it?
I soon need to upgrade a connection to a remote/cloud server to store data, and the choice is between mqtt and http. For http, I know exactly what to expect. For mqtt, I'm not so sure. What would you recommend?
Due to the eclectic range of protocols you're using, its very dependant on (if) the contributed Nodes do a good Job of error handling (I would hope most do)
if so - you could employee the Catch Node which is often overlooked IMO - this can be powerful if implemented correctly.
Other approaches involve some sort of 'Fire Escape' what I mean but that, is if a message is not received (and in the way you need) to react accordingly.
My Node (Below) can help with that - quite a bit - but is more for.. well.. unique accesss to a resource + but it can back out if it fails.
However... @Colin & @dceejay also built a Native Node RED version - but I could never remember how to find it.
If you prefer the Native approach, I'm sure they will link to it.
I'm afraid that I don't use any of those so I can't really comment on them.
Many IoT approaches though, because of severely constrained resources, may decide not to implement an ack. This would be very common for things like serial comms. With things like HTTP, while you may usually get a response, it isn't guaranteed. And things like WS I don't think give a response at all other than at the time of opening the connection. With all of these types of protocols, you would typically implement a timeout. If you are in a "listening" mode, that is generally easy enough. In a "sending" mode, you will implement your own ack or decide to not worry about it. A protocol like WS which is designed for real-time comms would typically have a ping/pong background message between the client and the server to detect whether the link is still live. I think that MQTT may also do something similar.
One thing that node-red can help with is to implement a poor-man's timeout where the underlying hardware/protocol doesn't support it. For example, if you expect your modbus sensor to be sending a new reading every minute, you can have a flow that will fire a message after 70 seconds if it didn't receive anything to reset the timer. You might also choose to normalise sensor data from a variety of different platforms.
You need to specify what you want to do with the connection to the cloud. Will there be a local element with the cloud acting as the aggregator? Or will the cloud service be the direct host to your IoT devices?
MQTT has "quality of service" levels, where QOS 2 "guarantees delivery once and once only".
It does this by exchanging receipts between the broker and publisher or the broker and subscriber, but these messages are not made available to the client.
Note though there is no guarantee that a published message is delivered to the subscriber, ie your Node-red and the cloud server.
The broker does not maintain a stack of undelivered messages in the expectation that a client will eventually be along to receive them.
If the cloud server is offline when you publish a message, the broker will confirm the message was received and happily discard it because there is no client connected & subscribed to the topic.
So to guarantee end to end delivery you need to provide your own "message received" channel, probably also over MQTT.
For me HTTP is a hopelessly overstuffed and ugly "language". MQTT is lightweight, fast and elegant. But it is not a point-to-point protocol.
Thanks all! Not sure if user error, but to the best of my effort, large portions of related community contributed nodes have been of rather poor quality in this aspect. So many weird & bad behaviors, no shortage of examples to draw from. So yeah, have explored and utilized catch nodes, timeouts and adding all sorts of "annex" on top so to speak. To get a robust and industrial result. Also sometimes feedback really helps and improvements are made by the contributors.
By response, I don't necessarily need an actual response from the remote resource, but rather an event that can be handled. For example even if target can't be reached, http at least results in a timeout error?
For modbus, I'm not sure if the remote device or the node itself fails to give error event, but I made a retry mechanic and configurable timeout, typically I set maybe 8 retries with 2 sec delays between each attempt and a 5 sec timeout. It depends on how frequent data needs to polled and how many requests (register ranges) there are in each poll.
For my specific connection to the cloud, it's thingsboard and we used it for many different purposes. Mainly node red sends telemetry, but also mqtt back and forth which allows users to interact and control devices in some cases.
MQTT hiding ack messages on higher QoS settings make sense, but also limits error handling? With http, I can take a backup when data fails to be sent, set up automatic and/or manual retries later. A sort of edge backup. That way I can keep a stack of undelivered messages, which is why I'm leaning towards http vs mqtt for sending telemetry. No additional channel required, and all the complexity that brings.
But I can certainly agree http is bloated and ugly. I've lost count how many times I've confused params with headers haha. A lot of available details I never had the use for. At least it's possible to keep it simple in many cases.
For MQTT, @colinl/node-red-guaranteed-delivery includes an example that guarantees that all messages are delivered to the MQTT broker, but if you want to guarantee that the subscriber has received the message you will have to add that yourself. I believe the MQTT V5 extensions include some features that help with that.
Even with http though, you can get a response that indicates that the server has received it, but you mentioned that you could be saving it to a database. If you want to know that this has also succeeded then you will have to do that at a higher level too.
Hi and happy weekend! I never considered the case where http is received successfully but not saved to database. I would assume that situation would be covered by Thingsboard's internal logic? Perhaps they make sure to save the data before returning success. At least I've never noticed anything otherwise. The possible future database I might use and administrate myself would be an edge backup, running on the same pi as node red, where pi can dump backups of data that failed to be sent to thingsboard. If that also fails, all hope is lost hehe.
If you are writing to a db. Make sure you use a prepared statement. If you need consistent updates across multiple tables, make sure you use transactions. That way, you know that your db updates will remain consistent even if one fails part way through.
MQTT is designed to be lightweight. So it foregoes the bullet-proof acks that heavyweight message queue systems use. If you need absolute guarantee's, use an enterprise message queue system like IBM MQ.
