# My recent journey into MQTT and messages. Two handy nodes to help you

**URL:** <https://discourse.nodered.org/t/my-recent-journey-into-mqtt-and-messages-two-handy-nodes-to-help-you/31863>\
**Category:** General\
**Created:** [22 August 2020 02:55 UTC](https://discourse.nodered.org/t/my-recent-journey-into-mqtt-and-messages-two-handy-nodes-to-help-you/31863 "2020-08-22T02:55:36Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![Trying\_to\_learn](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/trying_to_learn/32/28400_2.png) [@Trying\_to\_learn](https://discourse.nodered.org/u/Trying_to_learn)\
**Post date:** [22 August 2020 02:55 UTC](https://discourse.nodered.org/t/my-recent-journey-into-mqtt-and-messages-two-handy-nodes-to-help-you/31863/1 "2020-08-22T02:55:36Z")

</div>

Preface:  
This was brought about because of a lot of help I got from other people in my quest to send complicated messages over `MQTT`.

Thanks to @knolleary, @JGKK for helping me .... resolve the problem and for the inspiration to sit down and nut out the problem and so lead me to this _trick_.

# What is `MQTT`?

MQTT is a great thing I kind of took to early on and it is great for sending messages between devices.

Normally you are only sending _simple_ messages which only have a `payload`.  
The `topic` is used by/in `MQTT` to control _who_ gets the message.

_I want to expand `MQTT`'s usability to make it more like how a `link` node works on a machine._  
_You can use a `LINK OUT` node to go to another tab where you have a `LINK IN` node and it connects the two points as though they were connected._  
_They are used to help tidy flows up and separate parts so they are easier to understand._  
_What you send to the `LINK OUT` is received and the `LINK IN`._  
_Simple._

So basically a typical message going into a `MQTT OUT` node would have a `payload` and `topic`. Though the `topic` can be set in the `MQTT OUT` node.

Terrific if you are only sending **simple** (or basic?) `msg.payload`s

### But what happens when you start getting adventurous with what you are sending?

Well: then it all gets complicated.

If you want to send (say) a `payload` and a `colour`?  
`msg.payload` and `msg.colour`

Then, you have to re-construct the `msg` so it is actually structured as:  
`msg.payload.payload` and `msg.payload.colour`  
This can be achieved by adding a `function` node and putting a line in it like:

```auto
msg1.payload.payload = msg.payload;
msg1.payload.colour = msg.colour;
return msg1;

```

Maybe not exactly correct, but that is the idea of what is needed to send the message.

(And if you want to then add something else, you have to open the node and include a new line for the new piece of the message you want to send.)

Then, at the other end, you will receive the message in the new format.  
Which had me going nuts for days back in the early days.

_Also:_  
_Back then, the `MQTT IN` node also didn't have an option to make the `payload` a `JSON` string. So first off you had to put the message through a `JSON` node._

Then on the other machine, you have to remember the new structure of the message.  
Rather than being:  
`msg.payload` and `msg.colour`, it becomes:  
`msg.payload.payload` for the payload and  
`msg.payload.colour` for the colour.

Really annoying.

I get it that `MQTT` has its limits and I am not knocking them. It is just annoying if you have code on one machine and want to send a message (_complicated_) to another machine.

The structure of the messages will be different to what it was before it went through `MQTT`.  
So, to send it: you have to re-construct the message as just shown, and  
when you receive it: the structure is different.

Therefore you can't just export/import code from one machine to the other and it works.

Yes, not _the end of the world_ but can be a pain for people who are new to this, and annoying as you have to keep remembering to modify how the message is de-constructed at the receiving end.  
(As well as having to remember to add a new line at the sending end too!)

This is a great _cheat_ which is only two `nodes` to be added.  
One at the sending end and one at the receiving end.

With these two nodes included the flow at either side won't care/see what is happening.

This is an example flow to demonstrate the application of the two `nodes`.

All nodes with an `*` in their name has more to be read by opening them and looking in/on their comment page.

```auto
[{"id":"d8761a94.8ed48","type":"debug","z":"d103f472.19f3c8","name":"Received at other end","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":920,"y":3640,"wires":[]},{"id":"5161ffa8.9139b8","type":"mqtt out","z":"d103f472.19f3c8","name":"","topic":"temp","qos":"","retain":"","broker":"378c0403.8cda04","x":380,"y":3800,"wires":[]},{"id":"c5f87b0a.6c061","type":"mqtt in","z":"d103f472.19f3c8","name":"","topic":"temp","qos":"2","datatype":"json","broker":"378c0403.8cda04","x":530,"y":3800,"wires":[["50588e79.36139"]]},{"id":"50588e79.36139","type":"function","z":"d103f472.19f3c8","name":"*CHEAT NODE 2*","func":"var newMsg = msg.payload; \nreturn newMsg;","outputs":1,"noerr":0,"x":700,"y":3764,"wires":[["d8761a94.8ed48"]],"info":"This node takes the *modified* message and puts it back to what it was on the other machine before it was sent via MQTT.\nSimple.\n2 lines of code."},{"id":"5f7589e.a37ae78","type":"debug","z":"d103f472.19f3c8","name":"Message wanting to be sent","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":610,"y":3640,"wires":[]},{"id":"581f3926.17eb3","type":"function","z":"d103f472.19f3c8","name":"*CHEAT NODE 1*","func":"//node.warn(msg);\n\nvar newMsg = {payload:msg}\n\nreturn newMsg;\n\n","outputs":1,"noerr":0,"x":200,"y":3770,"wires":[["5161ffa8.9139b8","7c5b4d3d.dba8ac"]],"info":"This node has 2 lines and accepts dynamic message structures.\nSo you don't have to remmeber to add extra / new lines if/when you change the `message` structure.\n"},{"id":"8240bcf1.fb6dc8","type":"inject","z":"d103f472.19f3c8","name":"Test","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":210,"y":3590,"wires":[["9c01ed2f.ab1a68"]]},{"id":"9c01ed2f.ab1a68","type":"function","z":"d103f472.19f3c8","name":"Make a copmlicated message structure","func":"msg = {\n payload:\"This is the payload\",\n blah1:\"This is blah 1\",\n part1:\"This is part 1\",\n something_else:\"This is something else\"\n}\nreturn msg;","outputs":1,"noerr":0,"x":270,"y":3640,"wires":[["581f3926.17eb3","5f7589e.a37ae78"]]},{"id":"7c5b4d3d.dba8ac","type":"debug","z":"d103f472.19f3c8","name":"Sent into MQTT","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":410,"y":3740,"wires":[]},{"id":"561afa98.ebdb2c","type":"comment","z":"d103f472.19f3c8","name":"Read me *","info":"This top part is an example which represents your existing flow which is sendig a *complicated* message.\nYou need to send this message to another machine. But without a lot of messing around you have to make a node (similar to the one below) but not as simple.\nEverything you want/need to send has to have a line put in to correctly *format/resctructure* it to pass through MQTT.\n","x":400,"y":3590,"wires":[]},{"id":"fd70d098.5b6d48","type":"comment","z":"d103f472.19f3c8","name":"These two `debug` nodes are what matter. *","info":"You see what you flow is sending and you see what is received on the other end of the `MQTT` pipe.\n","x":780,"y":3590,"wires":[]},{"id":"378c0403.8cda04","type":"mqtt-broker","z":"","name":"MQTT HOST","broker":"192.168.0.99","port":"1883","clientid":"","usetls":false,"compatmode":false,"keepalive":"60","cleansession":true,"birthTopic":"","birthQos":"0","birthRetain":"true","birthPayload":"","closeTopic":"","closeQos":"0","closePayload":"","willTopic":"","willQos":"0","willPayload":""}]

```

You can see that the message being _sent in_ is the same as the one _received_.  
Which is the idea.

 ![Screenshot from 2020-08-22 12-50-17](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/0/c/0c4ecb21a37d27b1649c925537574e5dbbb90d7e.png)

I hope it is handy/useful for people who are sending messages via MQTT and are having trouble when needing to send more than a simple message.

All you need are the two **CHEAT NODEs** Shown in blue.  
Yes, I probably should have called them **Cheat OUT** and **Cheat IN**.  
Sorry. After thought.  
But I hope they are found to be useful.

---

<div class="post-metadata">

**Author:** ![krambriw](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/krambriw/32/5429_2.png) [@krambriw](https://discourse.nodered.org/u/krambriw)\
**Post date:** [22 August 2020 05:42 UTC](https://discourse.nodered.org/t/my-recent-journey-into-mqtt-and-messages-two-handy-nodes-to-help-you/31863/2 "2020-08-22T05:42:28Z")

</div>

Well......

Thinking a bit bout this I think there is actual a rather obvious reason behind what you refer to as "complicated messages"

Think about yourself entering a train wagon, it's empty, you can stroll around as you like, finally go and sit wherever you want. But if there are more passengers, they all have to be seated. And in first class, you have to behave and sit on specific seat numbers

So, the same with payload. If you only wanna send one "thing", no problem, if you wanna send more "things", you get my thinking....you have to organize the "passengers" in some way.

JSON is such a good way of organizing "things" as passengers. A JSON Object is actually what you are creating with your Cheat OUT node. The JSON Object is then sent via MQTT as a string. On the receiving end you have to convert it back to a JSON Object. Now the passengers can step out of the wagon...

This can easily be achieved also with the following flow (using JSONata in the Change node)

But more importantly is the understanding of how you have to organize your data before sending it to be able to resolve at the receiving end

Best regards, Walter

 ![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/f/4/f4cd7360267f6cc2775aeff7a78c1d03a6880c79.png)

```auto
[{"id":"917016c6.7508a8","type":"tab","label":"Testing","disabled":false,"info":""},{"id":"18876129.7b8ddf","type":"inject","z":"917016c6.7508a8","name":"","props":[{"p":"payload"},{"p":"color","v":"blue","vt":"str"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"test","payload":"just for fun","payloadType":"str","x":250,"y":280,"wires":[["44aeb32d.85cebc"]]},{"id":"461fcf03.f4bab","type":"mqtt out","z":"917016c6.7508a8","name":"","topic":"test","qos":"","retain":"","broker":"75eba16c.094f9","x":660,"y":280,"wires":[]},{"id":"3fbace99.126842","type":"mqtt in","z":"917016c6.7508a8","name":"","topic":"test","qos":"2","datatype":"auto","broker":"75eba16c.094f9","x":240,"y":350,"wires":[["3f4be658.54537a"]]},{"id":"9905f52a.ea93d8","type":"debug","z":"917016c6.7508a8","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":660,"y":350,"wires":[]},{"id":"44aeb32d.85cebc","type":"change","z":"917016c6.7508a8","name":"","rules":[{"t":"set","p":"payload","pt":"msg","to":"{'payload':payload, 'color':color} ","tot":"jsonata"}],"action":"","property":"","from":"","to":"","reg":false,"x":450,"y":280,"wires":[["461fcf03.f4bab"]]},{"id":"3f4be658.54537a","type":"json","z":"917016c6.7508a8","name":"","property":"payload","action":"","pretty":false,"x":450,"y":350,"wires":[["9905f52a.ea93d8"]]},{"id":"75eba16c.094f9","type":"mqtt-broker","z":"","name":"","broker":"127.0.0.1","port":"1883","clientid":"","usetls":false,"compatmode":true,"keepalive":"60","cleansession":true,"birthTopic":"online","birthQos":"0","birthPayload":"BULB-1/LWT","closeTopic":"","closeQos":"0","closePayload":"","willTopic":"offline","willQos":"0","willPayload":"BULB-1/LWT"}]

```

---

<div class="post-metadata">

**Author:** ![hominidae](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/hominidae/32/4174_2.png) [@hominidae](https://discourse.nodered.org/u/hominidae)\
**Post date:** [22 August 2020 06:23 UTC](https://discourse.nodered.org/t/my-recent-journey-into-mqtt-and-messages-two-handy-nodes-to-help-you/31863/3 "2020-08-22T06:23:05Z")

</div>

OK, you actually put a lot of effort into that, Thanks for sharing!  
But isn\*t mqtt designed as a pub/sub mechanism, not as an end2end send/receive?  
The way you use it, requires the sender to already know who the potential receiver is and what information is required on the other side.  
This is OK for your own flows and distributed node-red installations you control yourself.  
But as @krambriw already pointed out, there are other ways to achieve this.  
Trying to link this back to the original intend of what mqtt is used for, i think as follows:

What about linking to other node-red instances, that are not controlled by you?  
Maybe something like an implementation of the homie convention (I think there is even an implementation for node-red around) of node-red's internal message structure would make a great extension to what you have created.  
This way one could offer to link flows with others around the world, using mqtt 😉

Don't get me wrong, I like what you have done and will eventually use it, as mqtt is such an easy, lightweight tool.

---

<div class="post-metadata">

**Author:** ![Trying\_to\_learn](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/trying_to_learn/32/28400_2.png) [@Trying\_to\_learn](https://discourse.nodered.org/u/Trying_to_learn)\
**Post date:** [22 August 2020 06:49 UTC](https://discourse.nodered.org/t/my-recent-journey-into-mqtt-and-messages-two-handy-nodes-to-help-you/31863/4 "2020-08-22T06:49:59Z")

</div>

You are correct @krambriw with what you say.

And I thought I had covered the reasoning well enough.

These nodes are not supposed to be a golden solution to any/all problems.

MQTT is designed (as was said by @hominidae) for `publish` and `subscribe` use.

But often I have found I have a flow working on a machine and it is doing a good job.

I want to use half that flow on another machine to do the exact same thing.

but the demarcation of what is the `publisher` and who/where is the `subscriber` in the flow is not the meaning.

I want to be able to take the working code (part of the flow) on one machine and put it on another.

This is complicated because the messages between these two parts is not done at a simple level. The message sent between these two parts is what I called _complicated_.

This then means I had to sit down and work out the structure of the message being sent and add lines in a `function` node to move them to a `MQTT` compatible format.

Then at the other end, deconstruct the message back to what it was when it was sent. Allowing me to easily copy/paste the code from the other machine.

For some this is trivial. It has taken me ..... 2 years of adhoc (how ever it is spelt) node coding as needed.  
In the past week or so it has again risen and I have wasted hours (see one of my recent threads) trying to get something working which is easy and flexible.

This is it.

If it is not useful. Don't use it.

But to me, I have spent man days with this exact problem several times.

---

<div class="post-metadata">

**Author:** ![krambriw](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/krambriw/32/5429_2.png) [@krambriw](https://discourse.nodered.org/u/krambriw)\
**Post date:** [22 August 2020 08:03 UTC](https://discourse.nodered.org/t/my-recent-journey-into-mqtt-and-messages-two-handy-nodes-to-help-you/31863/5 "2020-08-22T08:03:22Z")

</div>

> [@Trying\_to\_learn](#):
>
> But to me, I have spent man days with this exact problem several times.

Yes, that is a pitty. One reason why careful design & analyze is is so important before you start coding

---

<div class="post-metadata">

**Author:** ![Trying\_to\_learn](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/trying_to_learn/32/28400_2.png) [@Trying\_to\_learn](https://discourse.nodered.org/u/Trying_to_learn)\
**Post date:** [22 August 2020 08:57 UTC](https://discourse.nodered.org/t/my-recent-journey-into-mqtt-and-messages-two-handy-nodes-to-help-you/31863/6 "2020-08-22T08:57:34Z")

</div>

As true as your statement is, _you_ can never know what you don't know.

I am constantly seeing things I don't understand and have to deal with them.

I write code with an idea (or concept) of what I want to do. Not a definitive one.  
This is partly because I have never done the latter.

And I find that if I spend time trying to definitively define every _nut and bolt_ of a program..... It just never seems to get written.

I can do it for small things and the like, but a lot of what I do in/on Node-Red is _fly by the seat of my pants_ because I am exploring the limits of what it can and can't do. Likewise with my abilities.

---

<div class="post-metadata">

**Author:** ![krambriw](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/krambriw/32/5429_2.png) [@krambriw](https://discourse.nodered.org/u/krambriw)\
**Post date:** [22 August 2020 09:22 UTC](https://discourse.nodered.org/t/my-recent-journey-into-mqtt-and-messages-two-handy-nodes-to-help-you/31863/7 "2020-08-22T09:22:48Z")

</div>

> [@Trying\_to\_learn](#):
>
> I am constantly seeing things I don't understand

Not that it helps in your or my situation but maybe a reason why they learn programming in schools today. From the first grade upwards. Remember why the Raspberry Pi was invented. Purpose was initially for children in school, not for you and me

---

<div class="post-metadata">

**Author:** ![Trying\_to\_learn](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/trying_to_learn/32/28400_2.png) [@Trying\_to\_learn](https://discourse.nodered.org/u/Trying_to_learn)\
**Post date:** [22 August 2020 09:33 UTC](https://discourse.nodered.org/t/my-recent-journey-into-mqtt-and-messages-two-handy-nodes-to-help-you/31863/8 "2020-08-22T09:33:42Z")

</div>

Alas I am from a time when mammoths walked the planet.

I learned BASIC by myself.

Touched Machine code, but that was a lost cause with `C` happening.

I then dabbled with `AREXX` back in the 90's for a wile. But they were very simple scripts.  
Nothing like here/now.

I have tried to do courses to learn, and read books, but like so many things I see today:  
They books are written with the pretence your already know the topic.  
Courses: if you ask questions, you are told not to be disruptive and _do the course_.  
To me that is an indication the _teacher_ (poor use of the word) doesn't know and doesn't want to bother trying to help.

So I have to throw myself at things and _bust a gut_ ........ _trying to learn_.  
It is slow. Painful. Riddled with traps and very poor return.

But my curiosity seems to keep in interested.

Oh, yeah. I am also trying to get my head around RasPies (Ubuntu) as I know ..... 30% of the O/S and how it works.

Is it wrong I am doing this? I don't know. But it is of interest and I try.

I think when you know everything - or are no longer interested in learning - it is time to put you in a small wooden box and put you _6 feet under_.

I think I have a long life ahead of me. 🤷‍♂️

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/1X/d073cd938eafa2e558d7c2cd59003b3ef4963033.png) [@system](https://discourse.nodered.org/u/system)\
**Post date:** [21 October 2020 09:33 UTC](https://discourse.nodered.org/t/my-recent-journey-into-mqtt-and-messages-two-handy-nodes-to-help-you/31863/9 "2020-10-21T09:33:45Z")

</div>

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.