# MQTT - what used to work now doesn't. Or so it seems. Structure of messages

**URL:** <https://discourse.nodered.org/t/mqtt-what-used-to-work-now-doesnt-or-so-it-seems-structure-of-messages/23488>\
**Category:** General\
**Created:** [23 March 2020 08:34 UTC](https://discourse.nodered.org/t/mqtt-what-used-to-work-now-doesnt-or-so-it-seems-structure-of-messages/23488 "2020-03-23T08:34:19Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![Trying\_to\_learn](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/trying_to_learn/32/28400_2.png) [@Trying\_to\_learn](https://discourse.nodered.org/u/Trying_to_learn)\
**Post date:** [23 March 2020 08:34 UTC](https://discourse.nodered.org/t/mqtt-what-used-to-work-now-doesnt-or-so-it-seems-structure-of-messages/23488/1 "2020-03-23T08:34:19Z")

</div>

I have been through a painful journey with MQTT and sending messages.

If I want to send a message with multiple _parts_, they aren't sent over MQTT as is.

What I need to do is _move_ them all into `msg.payload`.

At the other end of the MQTT line, I was told to use a `JSON` node to deconstruct the message back to its original parts.

Example flow:

```auto
[{"id":"704f811f.d9e7f8","type":"inject","z":"b95543c8.1478f8","name":"","topic":"","payload":"TEST_Payload","payloadType":"str","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":110,"y":120,"wires":[["d736dbcb.6d2bc8"]]},{"id":"d736dbcb.6d2bc8","type":"change","z":"b95543c8.1478f8","name":"","rules":[{"t":"set","p":"device","pt":"msg","to":"TEST_Device","tot":"str"},{"t":"set","p":"topic","pt":"msg","to":"TEST_Topic","tot":"str"},{"t":"set","p":"state","pt":"msg","to":"Test_OnLine","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":310,"y":120,"wires":[["23f518d6.fe0408","2b27f5e2.9ffa8a"]]},{"id":"23f518d6.fe0408","type":"function","z":"b95543c8.1478f8","name":"WAP msg construct","func":"\n//msg.payload = {payload: msg.payload + \" CHANGED \",device:msg.device};\nmsg.payload = {payload: msg.payload,device:msg.device,state:msg.state};\n\n//node.warn(msg.payload);\n//node.warn(msg.device);\n//node.warn(msg.payload.wap);\nreturn msg;\n","outputs":1,"noerr":0,"x":530,"y":120,"wires":[["552bd947.ef0db","c34fb03e.fe2be8"]]},{"id":"552bd947.ef0db","type":"mqtt out","z":"b95543c8.1478f8","name":"","topic":"TEST","qos":"","retain":"false","broker":"8a2e80be.f7c928","x":730,"y":120,"wires":[]},{"id":"d7c33418.01aea","type":"mqtt in","z":"b95543c8.1478f8","name":"","topic":"TEST","qos":"2","datatype":"auto","broker":"8a2e80be.f7c928","x":160,"y":260,"wires":[["d06fbdf0.2c3148","1f9e8c62.b1dd74"]]},{"id":"c34fb03e.fe2be8","type":"debug","z":"b95543c8.1478f8","name":"1","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":730,"y":80,"wires":[]},{"id":"d06fbdf0.2c3148","type":"debug","z":"b95543c8.1478f8","name":"2","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":290,"y":210,"wires":[]},{"id":"1f9e8c62.b1dd74","type":"json","z":"b95543c8.1478f8","name":"","property":"payload","action":"","pretty":false,"x":290,"y":260,"wires":[["195e78d7.76fb77"]]},{"id":"195e78d7.76fb77","type":"debug","z":"b95543c8.1478f8","name":"3","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":460,"y":210,"wires":[]},{"id":"2b27f5e2.9ffa8a","type":"debug","z":"b95543c8.1478f8","name":"0","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":490,"y":80,"wires":[]},{"id":"8a2e80be.f7c928","type":"mqtt-broker","z":"","name":"MQTT host","broker":"192.168.0.99","port":"1883","clientid":"","usetls":false,"compatmode":true,"keepalive":"60","cleansession":true,"birthTopic":"ARDUINO_STATUS","birthQos":"2","birthPayload":"connected","willTopic":"ARDUINO_STATUS","willQos":"0","willPayload":"disconnected"}]

```

So:  
`0` is the actual message to go through.  
`1` shows the message going into MQTT after _constructing_ the payload..  
`2` shows what is received. All the data is there packed into the `msg.payload` part.  
`3` should show the original message. It doesn't.

I've been through this and really got hung up on the fact that is is only the `msg.payload` which is sent over MQTT. So it was explained that I needed to put them all into the `msg.payload`.

That is done by the `function` node and the opposite is done by the `JSON` node. Which is kind of weird. Why not use a `JSON` at both ends? (I'm not going to say. I don't know.)

But it workED! Note: Past tense.

I can't exactly remember where I do this in the bigger picture, but I documented it as how to do it and this is the way I wrote it.

My usual question:  
What have I done wrong?

---

<div class="post-metadata">

**Author:** ![Colin](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/colin/32/17040_2.png) [@Colin](https://discourse.nodered.org/u/Colin)\
**Post date:** [23 March 2020 09:22 UTC](https://discourse.nodered.org/t/mqtt-what-used-to-work-now-doesnt-or-so-it-seems-structure-of-messages/23488/2 "2020-03-23T09:22:45Z")

</div>

You don't need to convert it from a js object to JSON before the MQTT Out node as the node recognises that you are passing it an object and converts it for you.  
Similarly, if you change the Output dropdown in the MQTT In node to `Parsed JSON object` then it will automatically parse the JSON it receives and convert it to a js object for you, so you don't need the JSON node at all.

**[Edit]** Just re-read your post again, what do you mean by it used to work? It is working perfectly for me. You still have to extract the data from msg.payload of course.

---

<div class="post-metadata">

**Author:** ![Trying\_to\_learn](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/trying_to_learn/32/28400_2.png) [@Trying\_to\_learn](https://discourse.nodered.org/u/Trying_to_learn)\
**Post date:** [23 March 2020 09:28 UTC](https://discourse.nodered.org/t/mqtt-what-used-to-work-now-doesnt-or-so-it-seems-structure-of-messages/23488/3 "2020-03-23T09:28:13Z")

</div>

Thanks Colin,

The _story_ is I was wanting to send _complex_ messages with more than just `msg.payload`.  
As only `msg.payload` is sent, I have to _build_ the message as with the first `function` node.

Reading your reply, I'm stuck with what you wrote:

> [@Colin](#):
>
> You don't need to convert it from a js object to JSON before the MQTT Out node

Sorry? Before the `MQTT OUT` node?

So we don't trip over each other's posts, (and I see @zenofmud is replying too) I'll stop here.

---

<div class="post-metadata">

**Author:** ![Colin](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/colin/32/17040_2.png) [@Colin](https://discourse.nodered.org/u/Colin)\
**Post date:** [23 March 2020 09:31 UTC](https://discourse.nodered.org/t/mqtt-what-used-to-work-now-doesnt-or-so-it-seems-structure-of-messages/23488/4 "2020-03-23T09:31:06Z")

</div>

> [@Trying\_to\_learn](#):
>
> Sorry? Before the `MQTT OUT` node?

You asked

> [@Trying\_to\_learn](#):
>
> Why not use a `JSON` at both ends?

That is why you don't need one at the front end, the MQTT node does the conversion for you.

---

<div class="post-metadata">

**Author:** ![Trying\_to\_learn](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/trying_to_learn/32/28400_2.png) [@Trying\_to\_learn](https://discourse.nodered.org/u/Trying_to_learn)\
**Post date:** [23 March 2020 09:37 UTC](https://discourse.nodered.org/t/mqtt-what-used-to-work-now-doesnt-or-so-it-seems-structure-of-messages/23488/5 "2020-03-23T09:37:14Z")

</div>

Ok, But the first sentence in your reply threw me. I don't put a `JSON` node there. Yeah. I got that.

The `MQTT In` node.... I am sure I tried that (there are 3 options and I tried them all and it just didn't work at reconstructing the message.)

Oh. Ok. Just got that bit about the `MQTT In`, the setting and the `JSON` node.

Shall try now.

I need a new brain!

---

<div class="post-metadata">

**Author:** ![Colin](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/colin/32/17040_2.png) [@Colin](https://discourse.nodered.org/u/Colin)\
**Post date:** [23 March 2020 09:42 UTC](https://discourse.nodered.org/t/mqtt-what-used-to-work-now-doesnt-or-so-it-seems-structure-of-messages/23488/6 "2020-03-23T09:42:07Z")

</div>

In case you think that there is an automatic way of unpacking msg.payload back into it's original form, you will need another function node (or change node) for that. There is no way that it can know that you want to extract bits of the payload out to individual msg attributes.

---

<div class="post-metadata">

**Author:** ![Trying\_to\_learn](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/trying_to_learn/32/28400_2.png) [@Trying\_to\_learn](https://discourse.nodered.org/u/Trying_to_learn)\
**Post date:** [23 March 2020 09:46 UTC](https://discourse.nodered.org/t/mqtt-what-used-to-work-now-doesnt-or-so-it-seems-structure-of-messages/23488/7 "2020-03-23T09:46:30Z")

</div>

I think I failed (miserably) in my documentation.

This is what I had documented:

> // How to construct messages to be sent over MQTT  
> // msg.payload construction to be done like this.  
> msg.payload = {  
> payload: "TelePi USB stick memory low",  
> who: "TelePi"  
> }  
> // At remote end you need to put data through a `JSON` node.  
> // and all parts will be prefixed msg.payload.xxxxxx  
> // So: to set msg.payload, you will have to say:  
> // msg.payload = msg.payload.payload;

Of course that is correct in what it says, but it neglects to say I need to put it through another `function` node (or what ever) to get the original structure.

Though I guess it is implied. But that is not always good.

And it neglects to say the `MQTT In` node could be set to `Parsed JSON object` and it would then not need the `JSON` node.

Shall update the notes now - before I forget.

Again:  
Thanks.

---

<div class="post-metadata">

**Author:** ![zenofmud](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/zenofmud/32/316_2.png) [@zenofmud](https://discourse.nodered.org/u/zenofmud)\
**Post date:** [23 March 2020 09:56 UTC](https://discourse.nodered.org/t/mqtt-what-used-to-work-now-doesnt-or-so-it-seems-structure-of-messages/23488/8 "2020-03-23T09:56:48Z")

</div>

Andrew, think of things this way.

NR passes a `msg` between nodes. `msg` is a JavaScript object. The object has many parts. In the simplest example, an `inject` node =\> `debug` you get

```auto
{
  "_msgid" : "c8c8fe5e.880d5",
  "payload" : 1584955368627,
  "topic" : ""
}

```

You can change the contents of that JavaScript Object like you did in your 'change: 3 rules` node so your msg object looks like this:

```auto
{
  "_msgid" : "d44db281.1a05b",
  "device" : "TEST_Device",
  "payload" : "TEST_Payload",
  "state" : "Test_OnLine",
  "topic" : "TEST_Topic"
}

```

When using MQTT you are leaving NR and now must play by the rules of MQTT. When you publish an MQTT message you only deal with two things: a `topic` and a string that contains the data. As you know, the `topic` is used by the MQTT broker to determine which 'clients' are sent the message.

In NR the MQTT-out node converts `msg.payload` (the rest of the msg is ignored) into a string before sending it.  
Prior to a couple releases ago, the MQTT-in node returned the data as a string but the node was changed to give you some options. One of them is to transform the incoming data to a JavaScript object and put it in msg.payload.

So your confusion might be that - in the past - if you setup msg.payload as a JavaScript object, it would be changed to a string, MQTT would send the string to a client. And if the client was in NR it would receive a string. To get it back to a JavaScript object, you would send it thru a `json` node.

Now (depending of the option you pick in the `MQTT-in` node) you can get rid of the `json` object.

I hope this helps you out.

---

<div class="post-metadata">

**Author:** ![Trying\_to\_learn](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/trying_to_learn/32/28400_2.png) [@Trying\_to\_learn](https://discourse.nodered.org/u/Trying_to_learn)\
**Post date:** [23 March 2020 10:05 UTC](https://discourse.nodered.org/t/mqtt-what-used-to-work-now-doesnt-or-so-it-seems-structure-of-messages/23488/9 "2020-03-23T10:05:06Z")

</div>

Yeah, thanks Paul.

I think you saw my poor documentation on _Sending MQTT messages_.

I have since updated it. (Given I have no training in how to write good documentation.)  
I have expanded the scope beyond what it was before explaining better the _sending_ side and how to construct the message and on the _receiving_ side/end how to set the `MQTT in` node and that the `JSON` node is now, no longer needed, and to put the message into another `function` node that does the reverse of what was done at the sending end.

Way back when I was stuck with getting the incoming message with the `{ }` around it stripped and Nick (@knolleary explained that is/was the function of the `JSON` node) but my lack of being able to write good docs failed me there.  
And the updated functionality of the `MQTT in` node to `Parsed JSON object` is good.  
Just I didn't apply that update to my docs in that respect either.

But I think now I have the problem fixed.

(And to think I am still finding problems with flows from _days of yore_ with problems like _time stamps_ etc. I am needing to go through the flows and update them using my new `timestamp` node/subflow.)

Maybe it is good I am not really in any serious computer stuff.

---

<div class="post-metadata">

**Author:** ![TotallyInformation](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/totallyinformation/32/31_2.png) [@TotallyInformation](https://discourse.nodered.org/u/TotallyInformation)\
**Post date:** [23 March 2020 11:46 UTC](https://discourse.nodered.org/t/mqtt-what-used-to-work-now-doesnt-or-so-it-seems-structure-of-messages/23488/10 "2020-03-23T11:46:16Z")

</div>

Just looking at your example data. If it is all like that, you could also possibly rethink the MQTT topic structure. When you get a chance, have a look at the Homie convention. Even if you don't use that, it will give you some ideas about how data can be deconstructed into individual topics in MQTT.

[https://homieiot.github.io/](https://homieiot.github.io/)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/1X/d073cd938eafa2e558d7c2cd59003b3ef4963033.png) [@system](https://discourse.nodered.org/u/system)\
**Post date:** [22 May 2020 11:46 UTC](https://discourse.nodered.org/t/mqtt-what-used-to-work-now-doesnt-or-so-it-seems-structure-of-messages/23488/11 "2020-05-22T11:46:21Z")

</div>

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.