My recent journey into MQTT and messages. Two handy nodes to help you

Preface:
This was brought about because of a lot of help I got from other people in my quest to send complicated messages over MQTT.

Thanks to @knolleary, @JGKK for helping me .... resolve the problem and for the inspiration to sit down and nut out the problem and so lead me to this trick.

What is MQTT?

MQTT is a great thing I kind of took to early on and it is great for sending messages between devices.

Normally you are only sending simple messages which only have a payload.
The topic is used by/in MQTT to control who gets the message.

I want to expand MQTT's usability to make it more like how a link node works on a machine.
You can use a LINK OUT node to go to another tab where you have a LINK IN node and it connects the two points as though they were connected.
They are used to help tidy flows up and separate parts so they are easier to understand.
What you send to the LINK OUT is received and the LINK IN.
Simple.

So basically a typical message going into a MQTT OUT node would have a payload and topic. Though the topic can be set in the MQTT OUT node.

Terrific if you are only sending simple (or basic?) msg.payloads

But what happens when you start getting adventurous with what you are sending?

Well: then it all gets complicated.

If you want to send (say) a payload and a colour?
msg.payload and msg.colour

Then, you have to re-construct the msg so it is actually structured as:
msg.payload.payload and msg.payload.colour
This can be achieved by adding a function node and putting a line in it like:

msg1.payload.payload = msg.payload;
msg1.payload.colour = msg.colour;
return msg1;

Maybe not exactly correct, but that is the idea of what is needed to send the message.

(And if you want to then add something else, you have to open the node and include a new line for the new piece of the message you want to send.)

Then, at the other end, you will receive the message in the new format.
Which had me going nuts for days back in the early days.

Also:
Back then, the MQTT IN node also didn't have an option to make the payload a JSON string. So first off you had to put the message through a JSON node.

Then on the other machine, you have to remember the new structure of the message.
Rather than being:
msg.payload and msg.colour, it becomes:
msg.payload.payload for the payload and
msg.payload.colour for the colour.

Really annoying.

I get it that MQTT has its limits and I am not knocking them. It is just annoying if you have code on one machine and want to send a message (complicated) to another machine.

The structure of the messages will be different to what it was before it went through MQTT.
So, to send it: you have to re-construct the message as just shown, and
when you receive it: the structure is different.

Therefore you can't just export/import code from one machine to the other and it works.

Yes, not the end of the world but can be a pain for people who are new to this, and annoying as you have to keep remembering to modify how the message is de-constructed at the receiving end.
(As well as having to remember to add a new line at the sending end too!)

This is a great cheat which is only two nodes to be added.
One at the sending end and one at the receiving end.

With these two nodes included the flow at either side won't care/see what is happening.

This is an example flow to demonstrate the application of the two nodes.

All nodes with an * in their name has more to be read by opening them and looking in/on their comment page.

[{"id":"d8761a94.8ed48","type":"debug","z":"d103f472.19f3c8","name":"Received at other end","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":920,"y":3640,"wires":[]},{"id":"5161ffa8.9139b8","type":"mqtt out","z":"d103f472.19f3c8","name":"","topic":"temp","qos":"","retain":"","broker":"378c0403.8cda04","x":380,"y":3800,"wires":[]},{"id":"c5f87b0a.6c061","type":"mqtt in","z":"d103f472.19f3c8","name":"","topic":"temp","qos":"2","datatype":"json","broker":"378c0403.8cda04","x":530,"y":3800,"wires":[["50588e79.36139"]]},{"id":"50588e79.36139","type":"function","z":"d103f472.19f3c8","name":"*CHEAT NODE 2*","func":"var newMsg = msg.payload; \nreturn newMsg;","outputs":1,"noerr":0,"x":700,"y":3764,"wires":[["d8761a94.8ed48"]],"info":"This node takes the *modified* message and puts it back to what it was on the other machine before it was sent via MQTT.\nSimple.\n2 lines of code."},{"id":"5f7589e.a37ae78","type":"debug","z":"d103f472.19f3c8","name":"Message wanting to be sent","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":610,"y":3640,"wires":[]},{"id":"581f3926.17eb3","type":"function","z":"d103f472.19f3c8","name":"*CHEAT NODE 1*","func":"//node.warn(msg);\n\nvar newMsg = {payload:msg}\n\nreturn newMsg;\n\n","outputs":1,"noerr":0,"x":200,"y":3770,"wires":[["5161ffa8.9139b8","7c5b4d3d.dba8ac"]],"info":"This node has 2 lines and accepts dynamic message structures.\nSo you don't have to remmeber to add extra / new lines if/when you change the `message` structure.\n"},{"id":"8240bcf1.fb6dc8","type":"inject","z":"d103f472.19f3c8","name":"Test","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":210,"y":3590,"wires":[["9c01ed2f.ab1a68"]]},{"id":"9c01ed2f.ab1a68","type":"function","z":"d103f472.19f3c8","name":"Make a copmlicated message structure","func":"msg = {\n    payload:\"This is the payload\",\n    blah1:\"This is blah 1\",\n    part1:\"This is part 1\",\n    something_else:\"This is something else\"\n}\nreturn msg;","outputs":1,"noerr":0,"x":270,"y":3640,"wires":[["581f3926.17eb3","5f7589e.a37ae78"]]},{"id":"7c5b4d3d.dba8ac","type":"debug","z":"d103f472.19f3c8","name":"Sent into MQTT","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":410,"y":3740,"wires":[]},{"id":"561afa98.ebdb2c","type":"comment","z":"d103f472.19f3c8","name":"Read me *","info":"This top part is an example which represents your existing flow which is sendig a *complicated* message.\nYou need to send this message to another machine.  But without a lot of messing around you have to make a node (similar to the one below) but not as simple.\nEverything you want/need to send has to have a line put in to correctly *format/resctructure* it to pass through MQTT.\n","x":400,"y":3590,"wires":[]},{"id":"fd70d098.5b6d48","type":"comment","z":"d103f472.19f3c8","name":"These two `debug` nodes are what matter. *","info":"You see what you flow is sending and you see what is received on the other end of the `MQTT` pipe.\n","x":780,"y":3590,"wires":[]},{"id":"378c0403.8cda04","type":"mqtt-broker","z":"","name":"MQTT HOST","broker":"192.168.0.99","port":"1883","clientid":"","usetls":false,"compatmode":false,"keepalive":"60","cleansession":true,"birthTopic":"","birthQos":"0","birthRetain":"true","birthPayload":"","closeTopic":"","closeQos":"0","closePayload":"","willTopic":"","willQos":"0","willPayload":""}]

You can see that the message being sent in is the same as the one received.
Which is the idea.

I hope it is handy/useful for people who are sending messages via MQTT and are having trouble when needing to send more than a simple message.

All you need are the two CHEAT NODEs Shown in blue.
Yes, I probably should have called them Cheat OUT and Cheat IN.
Sorry. After thought.
But I hope they are found to be useful.

Well......

Thinking a bit bout this I think there is actual a rather obvious reason behind what you refer to as "complicated messages"

Think about yourself entering a train wagon, it's empty, you can stroll around as you like, finally go and sit wherever you want. But if there are more passengers, they all have to be seated. And in first class, you have to behave and sit on specific seat numbers

So, the same with payload. If you only wanna send one "thing", no problem, if you wanna send more "things", you get my thinking....you have to organize the "passengers" in some way.

JSON is such a good way of organizing "things" as passengers. A JSON Object is actually what you are creating with your Cheat OUT node. The JSON Object is then sent via MQTT as a string. On the receiving end you have to convert it back to a JSON Object. Now the passengers can step out of the wagon...

This can easily be achieved also with the following flow (using JSONata in the Change node)

But more importantly is the understanding of how you have to organize your data before sending it to be able to resolve at the receiving end

Best regards, Walter

[{"id":"917016c6.7508a8","type":"tab","label":"Testing","disabled":false,"info":""},{"id":"18876129.7b8ddf","type":"inject","z":"917016c6.7508a8","name":"","props":[{"p":"payload"},{"p":"color","v":"blue","vt":"str"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"test","payload":"just for fun","payloadType":"str","x":250,"y":280,"wires":[["44aeb32d.85cebc"]]},{"id":"461fcf03.f4bab","type":"mqtt out","z":"917016c6.7508a8","name":"","topic":"test","qos":"","retain":"","broker":"75eba16c.094f9","x":660,"y":280,"wires":[]},{"id":"3fbace99.126842","type":"mqtt in","z":"917016c6.7508a8","name":"","topic":"test","qos":"2","datatype":"auto","broker":"75eba16c.094f9","x":240,"y":350,"wires":[["3f4be658.54537a"]]},{"id":"9905f52a.ea93d8","type":"debug","z":"917016c6.7508a8","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":660,"y":350,"wires":[]},{"id":"44aeb32d.85cebc","type":"change","z":"917016c6.7508a8","name":"","rules":[{"t":"set","p":"payload","pt":"msg","to":"{'payload':payload, 'color':color} ","tot":"jsonata"}],"action":"","property":"","from":"","to":"","reg":false,"x":450,"y":280,"wires":[["461fcf03.f4bab"]]},{"id":"3f4be658.54537a","type":"json","z":"917016c6.7508a8","name":"","property":"payload","action":"","pretty":false,"x":450,"y":350,"wires":[["9905f52a.ea93d8"]]},{"id":"75eba16c.094f9","type":"mqtt-broker","z":"","name":"","broker":"127.0.0.1","port":"1883","clientid":"","usetls":false,"compatmode":true,"keepalive":"60","cleansession":true,"birthTopic":"online","birthQos":"0","birthPayload":"BULB-1/LWT","closeTopic":"","closeQos":"0","closePayload":"","willTopic":"offline","willQos":"0","willPayload":"BULB-1/LWT"}]
1 Like

OK, you actually put a lot of effort into that, Thanks for sharing!
But isn*t mqtt designed as a pub/sub mechanism, not as an end2end send/receive?
The way you use it, requires the sender to already know who the potential receiver is and what information is required on the other side.
This is OK for your own flows and distributed node-red installations you control yourself.
But as @krambriw already pointed out, there are other ways to achieve this.
Trying to link this back to the original intend of what mqtt is used for, i think as follows:

What about linking to other node-red instances, that are not controlled by you?
Maybe something like an implementation of the homie convention (I think there is even an implementation for node-red around) of node-red's internal message structure would make a great extension to what you have created.
This way one could offer to link flows with others around the world, using mqtt :wink:

Don't get me wrong, I like what you have done and will eventually use it, as mqtt is such an easy, lightweight tool.

You are correct @krambriw with what you say.

And I thought I had covered the reasoning well enough.

These nodes are not supposed to be a golden solution to any/all problems.

MQTT is designed (as was said by @hominidae) for publish and subscribe use.

But often I have found I have a flow working on a machine and it is doing a good job.

I want to use half that flow on another machine to do the exact same thing.

but the demarcation of what is the publisher and who/where is the subscriber in the flow is not the meaning.

I want to be able to take the working code (part of the flow) on one machine and put it on another.

This is complicated because the messages between these two parts is not done at a simple level. The message sent between these two parts is what I called complicated.

This then means I had to sit down and work out the structure of the message being sent and add lines in a function node to move them to a MQTT compatible format.

Then at the other end, deconstruct the message back to what it was when it was sent. Allowing me to easily copy/paste the code from the other machine.

For some this is trivial. It has taken me ..... 2 years of adhoc (how ever it is spelt) node coding as needed.
In the past week or so it has again risen and I have wasted hours (see one of my recent threads) trying to get something working which is easy and flexible.

This is it.

If it is not useful. Don't use it.

But to me, I have spent man days with this exact problem several times.

Yes, that is a pitty. One reason why careful design & analyze is is so important before you start coding

1 Like

As true as your statement is, you can never know what you don't know.

I am constantly seeing things I don't understand and have to deal with them.

I write code with an idea (or concept) of what I want to do. Not a definitive one.
This is partly because I have never done the latter.

And I find that if I spend time trying to definitively define every nut and bolt of a program..... It just never seems to get written.

I can do it for small things and the like, but a lot of what I do in/on Node-Red is fly by the seat of my pants because I am exploring the limits of what it can and can't do. Likewise with my abilities.

Not that it helps in your or my situation but maybe a reason why they learn programming in schools today. From the first grade upwards. Remember why the Raspberry Pi was invented. Purpose was initially for children in school, not for you and me

Alas I am from a time when mammoths walked the planet.

I learned BASIC by myself.

Touched Machine code, but that was a lost cause with C happening.

I then dabbled with AREXX back in the 90's for a wile. But they were very simple scripts.
Nothing like here/now.

I have tried to do courses to learn, and read books, but like so many things I see today:
They books are written with the pretence your already know the topic.
Courses: if you ask questions, you are told not to be disruptive and do the course.
To me that is an indication the teacher (poor use of the word) doesn't know and doesn't want to bother trying to help.

So I have to throw myself at things and bust a gut ........ trying to learn.
It is slow. Painful. Riddled with traps and very poor return.

But my curiosity seems to keep in interested.

Oh, yeah. I am also trying to get my head around RasPies (Ubuntu) as I know ..... 30% of the O/S and how it works.

Is it wrong I am doing this? I don't know. But it is of interest and I try.

I think when you know everything - or are no longer interested in learning - it is time to put you in a small wooden box and put you 6 feet under.

I think I have a long life ahead of me. :man_shrugging:

2 Likes