MQTT - what used to work now doesn't. Or so it seems. Structure of messages

General
1

I have been through a painful journey with MQTT and sending messages.

If I want to send a message with multiple parts, they aren't sent over MQTT as is.

What I need to do is move them all into msg.payload.

At the other end of the MQTT line, I was told to use a JSON node to deconstruct the message back to its original parts.

Example flow:

[{"id":"704f811f.d9e7f8","type":"inject","z":"b95543c8.1478f8","name":"","topic":"","payload":"TEST_Payload","payloadType":"str","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":110,"y":120,"wires":[["d736dbcb.6d2bc8"]]},{"id":"d736dbcb.6d2bc8","type":"change","z":"b95543c8.1478f8","name":"","rules":[{"t":"set","p":"device","pt":"msg","to":"TEST_Device","tot":"str"},{"t":"set","p":"topic","pt":"msg","to":"TEST_Topic","tot":"str"},{"t":"set","p":"state","pt":"msg","to":"Test_OnLine","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":310,"y":120,"wires":[["23f518d6.fe0408","2b27f5e2.9ffa8a"]]},{"id":"23f518d6.fe0408","type":"function","z":"b95543c8.1478f8","name":"WAP msg construct","func":"\n//msg.payload = {payload: msg.payload + \" CHANGED \",device:msg.device};\nmsg.payload = {payload: msg.payload,device:msg.device,state:msg.state};\n\n//node.warn(msg.payload);\n//node.warn(msg.device);\n//node.warn(msg.payload.wap);\nreturn msg;\n","outputs":1,"noerr":0,"x":530,"y":120,"wires":[["552bd947.ef0db","c34fb03e.fe2be8"]]},{"id":"552bd947.ef0db","type":"mqtt out","z":"b95543c8.1478f8","name":"","topic":"TEST","qos":"","retain":"false","broker":"8a2e80be.f7c928","x":730,"y":120,"wires":[]},{"id":"d7c33418.01aea","type":"mqtt in","z":"b95543c8.1478f8","name":"","topic":"TEST","qos":"2","datatype":"auto","broker":"8a2e80be.f7c928","x":160,"y":260,"wires":[["d06fbdf0.2c3148","1f9e8c62.b1dd74"]]},{"id":"c34fb03e.fe2be8","type":"debug","z":"b95543c8.1478f8","name":"1","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":730,"y":80,"wires":[]},{"id":"d06fbdf0.2c3148","type":"debug","z":"b95543c8.1478f8","name":"2","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":290,"y":210,"wires":[]},{"id":"1f9e8c62.b1dd74","type":"json","z":"b95543c8.1478f8","name":"","property":"payload","action":"","pretty":false,"x":290,"y":260,"wires":[["195e78d7.76fb77"]]},{"id":"195e78d7.76fb77","type":"debug","z":"b95543c8.1478f8","name":"3","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":460,"y":210,"wires":[]},{"id":"2b27f5e2.9ffa8a","type":"debug","z":"b95543c8.1478f8","name":"0","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":490,"y":80,"wires":[]},{"id":"8a2e80be.f7c928","type":"mqtt-broker","z":"","name":"MQTT host","broker":"192.168.0.99","port":"1883","clientid":"","usetls":false,"compatmode":true,"keepalive":"60","cleansession":true,"birthTopic":"ARDUINO_STATUS","birthQos":"2","birthPayload":"connected","willTopic":"ARDUINO_STATUS","willQos":"0","willPayload":"disconnected"}]

So:
0 is the actual message to go through.
1 shows the message going into MQTT after constructing the payload..
2 shows what is received. All the data is there packed into the msg.payload part.
3 should show the original message. It doesn't.

I've been through this and really got hung up on the fact that is is only the msg.payload which is sent over MQTT. So it was explained that I needed to put them all into the msg.payload.

That is done by the function node and the opposite is done by the JSON node. Which is kind of weird. Why not use a JSON at both ends? (I'm not going to say. I don't know.)

But it workED! Note: Past tense.

I can't exactly remember where I do this in the bigger picture, but I documented it as how to do it and this is the way I wrote it.

My usual question:
What have I done wrong?

2

You don't need to convert it from a js object to JSON before the MQTT Out node as the node recognises that you are passing it an object and converts it for you.
Similarly, if you change the Output dropdown in the MQTT In node to Parsed JSON object then it will automatically parse the JSON it receives and convert it to a js object for you, so you don't need the JSON node at all.

[Edit] Just re-read your post again, what do you mean by it used to work? It is working perfectly for me. You still have to extract the data from msg.payload of course.

3

Thanks Colin,

The story is I was wanting to send complex messages with more than just msg.payload.
As only msg.payload is sent, I have to build the message as with the first function node.

Reading your reply, I'm stuck with what you wrote:

Sorry? Before the MQTT OUT node?

So we don't trip over each other's posts, (and I see @zenofmud is replying too) I'll stop here.

4

You asked

That is why you don't need one at the front end, the MQTT node does the conversion for you.

5

Ok, But the first sentence in your reply threw me. I don't put a JSON node there. Yeah. I got that.

The MQTT In node.... I am sure I tried that (there are 3 options and I tried them all and it just didn't work at reconstructing the message.)

Oh. Ok. Just got that bit about the MQTT In, the setting and the JSON node.

Shall try now.

I need a new brain!

6

In case you think that there is an automatic way of unpacking msg.payload back into it's original form, you will need another function node (or change node) for that. There is no way that it can know that you want to extract bits of the payload out to individual msg attributes.

1 Like
7

I think I failed (miserably) in my documentation.

This is what I had documented:

// How to construct messages to be sent over MQTT
// msg.payload construction to be done like this.
msg.payload = {
payload: "TelePi USB stick memory low",
who: "TelePi"
}
// At remote end you need to put data through a JSON node.
// and all parts will be prefixed msg.payload.xxxxxx
// So: to set msg.payload, you will have to say:
// msg.payload = msg.payload.payload;

Of course that is correct in what it says, but it neglects to say I need to put it through another function node (or what ever) to get the original structure.

Though I guess it is implied. But that is not always good.

And it neglects to say the MQTT In node could be set to Parsed JSON object and it would then not need the JSON node.

Shall update the notes now - before I forget.

Again:
Thanks.

8

Andrew, think of things this way.

NR passes a msg between nodes. msg is a JavaScript object. The object has many parts. In the simplest example, an inject node => debug you get

{
  "_msgid" : "c8c8fe5e.880d5",
  "payload" : 1584955368627,
  "topic" : ""
}

You can change the contents of that JavaScript Object like you did in your 'change: 3 rules` node so your msg object looks like this:

{
  "_msgid" : "d44db281.1a05b",
  "device" : "TEST_Device",
  "payload" : "TEST_Payload",
  "state" : "Test_OnLine",
  "topic" : "TEST_Topic"
}

When using MQTT you are leaving NR and now must play by the rules of MQTT. When you publish an MQTT message you only deal with two things: a topic and a string that contains the data. As you know, the topic is used by the MQTT broker to determine which 'clients' are sent the message.

In NR the MQTT-out node converts msg.payload (the rest of the msg is ignored) into a string before sending it.
Prior to a couple releases ago, the MQTT-in node returned the data as a string but the node was changed to give you some options. One of them is to transform the incoming data to a JavaScript object and put it in msg.payload.

So your confusion might be that - in the past - if you setup msg.payload as a JavaScript object, it would be changed to a string, MQTT would send the string to a client. And if the client was in NR it would receive a string. To get it back to a JavaScript object, you would send it thru a json node.

Now (depending of the option you pick in the MQTT-in node) you can get rid of the json object.

I hope this helps you out.

9

Yeah, thanks Paul.

I think you saw my poor documentation on Sending MQTT messages.

I have since updated it. (Given I have no training in how to write good documentation.)
I have expanded the scope beyond what it was before explaining better the sending side and how to construct the message and on the receiving side/end how to set the MQTT in node and that the JSON node is now, no longer needed, and to put the message into another function node that does the reverse of what was done at the sending end.

Way back when I was stuck with getting the incoming message with the { } around it stripped and Nick (@knolleary explained that is/was the function of the JSON node) but my lack of being able to write good docs failed me there.
And the updated functionality of the MQTT in node to Parsed JSON object is good.
Just I didn't apply that update to my docs in that respect either.

But I think now I have the problem fixed.

(And to think I am still finding problems with flows from days of yore with problems like time stamps etc. I am needing to go through the flows and update them using my new timestamp node/subflow.)

Maybe it is good I am not really in any serious computer stuff.

10

Just looking at your example data. If it is all like that, you could also possibly rethink the MQTT topic structure. When you get a chance, have a look at the Homie convention. Even if you don't use that, it will give you some ideas about how data can be deconstructed into individual topics in MQTT.

https://homieiot.github.io/

11

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.