We have http-request, tcp-request, serial-request - any objections to an mqtt-request?

Exactly. Nailed it.

The MQTT broker remains a pure, stateless postman. It doesn't care about the relationship between the packets - it just routes the request to the backend and the response back to Node-RED.

Ok so the mqtt-request node only differs from the http-request and tcp-request nodes in that it will have state and maintain a lookup table with all the requests that have been sent. There could be a risk of memory exhaustion if too many unanswered requests are sent. Are you going to add a limit on the memory consumption? Or use something like a circular buffer to ensure that NR doesn't crash if a third party server goes down and doesn't respond?

I would hide the correlationData - i.e. the mqtt-request node should always manage that. After all, I as a user of the node don't want to be bothered with setting it if the node matches on the response for me anyway. For me it's a bit like the res/req msg attributes for http nodes - I won't want to be setting them, they're managed for me.

Yeah, point was raised by Dave yesterday:

It is the same mechanism other core nodes employ so I am happy it was flagged.

[edit]

To add to that, there is a hard requirement for a timeout value greater than zero (tho currently no upper limit). That timeout clears the items from the map as they expire (self healing). It can still be problematic faced with a flurry of requests obviously but that is now somewhat contained by the nodeMessageBufferMaxLength


That was actually how I designed it initially! :oncoming_fist:

However, while auto-generating a hidden UUID works perfectly for a closed ecosystem where Node-RED talks to (for example) another Node-RED instance, correlationData in the MQTT v5 spec can be a UTF-8 string or raw binary data. I have read as much as I can stand to determine if I should or should not show or hide this field. The main reason I decided to show it was thinking of the likes of low powered devices (like PLCs) where the correlationData might be utelised as a register ID, or a composite value with meaning, or an index number for sequential operations, etc. I also remember reading something about specific requirements for OpenTelemetry (I may have been hallucinating at that point).

In the end, I just bit the bullet and decided to show it but make it opinionated by default, with user override possible. By default, it will automatically handle everything via an auto-generated unique ID

Thanks for the tasking questions tho - it helps me determine if right choices were made.

It does also make me think that perhaps a collapsed "Advance Options" panel might be preferable (to keep the edit panel light)

Yer that might be a good comprise. I would assume that if the correlationData has meaning, then the user will probably end up creating a custom solution using the existing MQTT node anyway.

More and more, I've tended to less and less - i.e. less is more principle, hence I would remove the field and see if there are requests to actually have the field come in and what the use case could be.

But I'm not target audience! For me the correlationData is like a session id inside a cookie: I just want the session, I don't care about the session id or the how the association to session is made.

Thanks for the explanation and you've convinced me, it definitely sounds like a core feature.

Does the node queue requests and wait for the previous one to complete before sending the next one or can there be multiple transactions underway at once? If the latter then how will the requesting client match responses to requests?

Multiple transactions can be underway simultaneously. The node does not block unless the limit is hit.

It relies on the internal lookup map.

When a request is sent, the node stores the original msg reference in memory using the unique correlationData as the key (along with a node identifier). When a response arrives, the node looks up that incoming correlationData key, grabs the matching msg, and passes it out of the respective node.

Because every concurrent request gets its own unique correlation ID (auto-generated as a UUID by default), multiple in-flight requests can happily coexist without overlapping or blocking each other.

Note, this map is per-broker (if you have such as situation, they can never collide due to protocol wire separation)

[EDIT]
Forgot to add - collisions are detected up front (before publish). If a correlation value is ALREADY in-flight, that message is rejected. Yes, this is opinionated, but feels like the right fit.
In short, one could re-use the same correlation data over and over BUT they would need to ensure the last request was resolved BEFORE they pushed that same correlation message again. That is user choice.

Not sure if that is 100% clear - feel free to say "what are you talkin about willis!" :wink:

This might seem unlikely but similar to the hidden _msgid field that is often the same because messages are just recycled. Anything that uses a split might well have several messages with the same correlationData value if the user is setting them via the msg.

On the other hand, it's a good way to prevent duplicate requests from being sent, no one wants to turn on their lights twice when they're already on :wink:

Caveat Emptor I would say.

How does the client flow match a response to the the request? If the flow sends in multiple requests, the responses could presumably appear in a different order.

I maintain the Snap! MQTT Library and one user has a great need for request/response so its been added in :slight_smile:

It is implemented using MQTT3.1 - it'll be interesting to see if it is compatible with this implementation

At the other end (the other client) will receive the topic, the response topic, correlation data (and other properties). They route this message accordingly based on whatever their desire (topic, payload content, correlation data value, user properties), then post the result back to the topic specified in response topic AND, importantly, include the original correlation data value. This way it doesn't matter if messages are handled out of order.

At the mqtt request side (originator), it is completely seamless. You push your message in and the associated response is automatically taken from the map and emitted out of the node.

Hmm, just like a REAL message queue service! :wink:

So if I send in, say, three requests, then the responses (or timeout errors) will appear in the same order, even though that may mean that the node has to delay later responses until earlier ones have completed or timed out.

No, sorry if i am being unclear. It is not a stack. The map means responses (as they arrive - in order / out of order) are taken from the map and emitted out of the MQTT Request node as they land.

If your 1st message takes 6 seconds, 2nd message takes 4 and the 3rd message takes 2 seconds, they will be output in reverse order - immediately upon reception back at the requestor.

So if there is a timeout, is there any way of knowing which message has not got through, other than providing my own correlation data? Or queuing the messages externally and passing them to the request node one at a time.

The node will throw and you catch it and can get the topic (and other data) that you can identify the original message. But it does give me pause for thought. I will run some scenarios through my head.

In fact I realise that it isn't an issue for my particular use case, which is guaranteed delivery of MQTT messages to an end client, as I will use @colinl/node-red-guaranteed-delivery, which queues the messages and releases them one at a time, so the new node will only ever be handling one message at a time.

Presumably, if a message times out, but later a response is received from the end client, that will just be silently ignored. Similarly if two responses are seen with the same correlation data the second one will be ignored.

Correct and correct. Once "spent", it is not available to output a 2nd time.

One other detail, if a valid response is seen then the node will send that, and trigger any Complete node attached, and if there is a timeout then it will trigger a Catch node and not send a message?

That is the intent (catch for sure) but I will need to double check the done (complete node signaller) calls when i get back into it - thanks for all the clarifying questions guys.

Great thanks.

Will there be a way of testing it before it is released?