Handling MQTT messages with occasional JSON payloads

General
1

As a kind of follow on from the other (recent) post, I am still stuck but with a new problem.

I am sending commands to remote machines and monitoring their reply.

When a command is received I get this:
{"topic":"COMMAND_REPLY/BedPi","payload":"Command received","qos":2,"retain":false,"_msgid":"71d39c0b.57bfe4"}

All being good, I get this back:
{"topic":"COMMAND_REPLY/BedPi","payload":"{\"code\":0}","qos":2,"retain":false,"_msgid":"a395b354.35cdf"}

In bad situations I get this back (as example)
{"topic":"COMMAND_REPLY/BedPi","payload":"{\"code\":127,\"message\":\"Command failed: ityut\\n/bin/sh: 1: ityut: not found\\n\",\"result\":\"FAIL\"}","qos":2,"retain":false,"_msgid":"4ec683cd.3abe1c"}

(Not rocket science.)

But the problem now happens.

These come back on COMMAND_REPLY/(devicename)
All good there.
But I then (now?) want to farm this off to another process which will alert me with real hardware.

This is the flow I have - which doesn't work.

[{"id":"c4be6f7.a9f739","type":"mqtt in","z":"9c1e5490.5d70b","name":"Return Codes from remote commands","topic":"COMMAND_REPLY/#","qos":"2","broker":"8d56c01f.a5662","x":210,"y":820,"wires":[["8cc09199.3899c8","40e0b56c.e338fc"]]},{"id":"8cc09199.3899c8","type":"function","z":"9c1e5490.5d70b","name":"Command spliter","func":"var result_ = msg.payload;\nvar fcommand = msg.payload.message;\nnode.warn(\"Fcommand\");\nnode.warn(fcommand);\nif (result_.indexOf('code\":1') !== -1)\n{\n    //\n    node.status({fill:\"yellow\",shape:\"dot\",text:\"Oops\"});\n    msg.payload = \"Oops\";\n    //  more code here for the second output.\n    var msg2 = {payload: \"yellow\"};\n}\n\nif (result_.indexOf('code\":127') !== -1)\n{\n    node.status({fill:\"red\",shape:\"dot\",text:\"Fail\"});\n    msg = {payload:\"Command fail\", command:fcommand};\n    node.warn(msg);\n//    msg.payload = \"Command fail\";\n    //  more code here for the second output.\n    msg2 = {payload: \"red\"};\n}\nreturn [msg,msg2];\n","outputs":2,"noerr":0,"x":490,"y":820,"wires":[["53a1e649.584bd8","e3691ea6.85f748"],["fa9d0647.0bcf7"]]},{"id":"40e0b56c.e338fc","type":"debug","z":"9c1e5490.5d70b","name":"RAW message","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":420,"y":740,"wires":[]},{"id":"fa9d0647.0bcf7","type":"debug","z":"9c1e5490.5d70b","name":"Alarm colour","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","x":680,"y":870,"wires":[]},{"id":"53a1e649.584bd8","type":"debug","z":"9c1e5490.5d70b","name":"Alarm message","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":650,"y":740,"wires":[]},{"id":"e3691ea6.85f748","type":"switch","z":"9c1e5490.5d70b","name":"","property":"payload","propertyType":"msg","rules":[{"t":"eq","v":"Command successful.","vt":"str"},{"t":"eq","v":"Oops.","vt":"str"},{"t":"eq","v":"Command fail.","vt":"str"}],"checkall":"true","repair":false,"outputs":3,"x":680,"y":790,"wires":[[],[],["1d82abad.643b34","2ef7273c.d9f948"]]},{"id":"1d82abad.643b34","type":"debug","z":"9c1e5490.5d70b","name":"URGENT","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":890,"y":820,"wires":[]},{"id":"2ef7273c.d9f948","type":"mqtt out","z":"9c1e5490.5d70b","name":"","topic":"TEST","qos":"","retain":"","broker":"8a2e80be.f7c928","x":880,"y":780,"wires":[]},{"id":"8d56c01f.a5662","type":"mqtt-broker","z":"","name":"MQTT host","broker":"192.168.0.99","port":"1883","clientid":"","usetls":false,"compatmode":false,"keepalive":"60","cleansession":true,"birthTopic":"SOM","birthQos":"2","birthPayload":"TimePi Comms Up","closeTopic":"EOM","closePayload":"TimePi shutting down","willTopic":"EOM","willQos":"0","willPayload":"TimePi Comms Failure"},{"id":"8a2e80be.f7c928","type":"mqtt-broker","z":"","name":"MQTT host","broker":"192.168.0.99","port":"1883","clientid":"","usetls":false,"compatmode":true,"keepalive":"60","cleansession":true,"birthTopic":"ARDUINO_STATUS","birthQos":"2","birthPayload":"connected","willTopic":"ARDUINO_STATUS","willQos":"0","willPayload":"disconnected"}]

The problem is it works for the "command received" and "command good".
But when I get a "command error" and the extra data the function node isn't splitting / dealing with the payload (?).

I put a JSON node in to get that formatted (?) correctly and it works, but it spits the dummy when it gets the "command received" and "command ok" messages.
unexpected character at position 0 I think.
Or it is TypeError: result_.indexOf is not a function because of the msg.payload structure.
(That is from the function node.)

I've been going around in circles because I got it working with the error coming back, but forgot the check what happens with the "received" and "ok" messages.

To get them working, it then falls over with an error message.

I am wanting/trying to construct a message which can be sent with MQTT to another device to then decode the error and indicate the needed information.

That (in this example) is:
The device name - BedPi in this case.
The command - Command failed: ityut/bin/sh: 1: ityut: not found in this case.

Does that make sense?
How can I get around the problem?

I'm open to maybe handling the original messages being re-parsed rather than going through an intermediate flow.

If the flow I posted, the MQTT OUT node at the end is the input to the next stage where the real world I/O happens. (Indicators etc)
It is publishing to TEST for now as I am still working on that part too.

I'll stop typing/digging now. Hope someone can see what I am trying to do and can offer a solution.

2

once again your title give no clue as to what the problem may be about.

3 Likes
3

That sounds like some of your MQTT messages are formatted as valid JSON and some of your MQTT messages are just pieces of text like command error.

You either need to standardise on all of your message payloads being JSON so you can handle them consistently, or you need to add some code to your Function that tries to tell the difference between them and then handle them accordingly.

4

(Updated the title so people might get you help easier/document it for future searches)

1 Like
5

I think that is the way I need to go. But I don't know how to do that.

(Don't worry. Just worked it out - I hope/think/believe)

6

It is one solution yes. However, standardising the payloads of your MQTT communication might help you in the long term. You posted 3 sample messages to your first post. The first has a string payload. The second a JSON string payload with just 1 property code. The third a JSON string payload with properties code, message and result.

In fact, both message 2 and 3 are valid JSON payloads. The first message, the one with the string payload which I think you refer to as "command received" and "command ok" messages, can easily be turned into a JSON string payload as well. In fact, you can standardise it to look like your other JSON strings. The third message you posted had after all a message property, and seeing as this command is returning a message to you stating on what happened, why not set it as message too:

{
    "message": "Command received",
}

(in readable format)
Or as message when returned from the MQTT node:
{"topic": "COMMANND_REPLY/BedPi", "payload": "{\"message\": \"Command received\"}", "qos": 2, "retain": false, "_msgid": "foo.bar"}.
This way you can parse all of them with a JSON node, or easier, tell the MQTT in to parse the payload as JSON rather than automatic or string.

Standardising the payload will give you a better maintainable and cleaner solution in the end, resulting in cleaner solutions on the long run. If you had to go back to this in the future, and something was changing to one of those messages, you will have to dive into the function node and figure out what is happening/why is it happening and how it has to be changed. It will involve editing the javascript code, when if you standardise the output, it is simply a case of checking the MQTT messages and check the according nodes. With the built-in MQTT node supporting JSON parsing, and a JSON node existing, I would personally never think of checking a function node to see the JSON parsing happening.

7

One of my students created this simple flow to detect for a 'json' format and route the message accordingly.

ScreenShot074
ScreenShot074.png800Ă—209 15.5 KB 

[{"id":"7d7518c0.376dd8","type":"debug","z":"968153b0.1619b","name":"","active":true,"console":"false","complete":"false","x":810,"y":240,"wires":[]},{"id":"fb749c5d.3c971","type":"json","z":"968153b0.1619b","name":"","property":"payload","action":"","pretty":false,"x":630,"y":180,"wires":[["7d7518c0.376dd8"]]},{"id":"b523b238.d158f8","type":"switch","z":"968153b0.1619b","name":"JSON format?","property":"$substring(msg.payload, 0, 1)=\"{\"","propertyType":"jsonata","rules":[{"t":"true"},{"t":"false"}],"checkall":"true","repair":false,"outputs":2,"x":460,"y":200,"wires":[["fb749c5d.3c971"],["7d7518c0.376dd8"]]},{"id":"451582e7.bc8234","type":"comment","z":"968153b0.1619b","name":"Check for leading \"{\" char and send to appropriate output","info":"","x":590,"y":120,"wires":[]},{"id":"16b10509.ff807b","type":"mqtt in","z":"968153b0.1619b","name":"","topic":"","qos":"2","datatype":"auto","x":190,"y":200,"wires":[["b523b238.d158f8"]]},{"id":"9893a265.6ffac","type":"comment","z":"968153b0.1619b","name":"Subscribe to a MQTT topic","info":"","x":250,"y":160,"wires":[]}]
2 Likes
8

Nice and clean solution. Readable as flow itself with a single look, and the comments only make it more clear. Works perfect for wildcard subscriptions too. While still I will repeat that standardising your messaging protocol for a single kind of I/O works in your favour on the long term.

1 Like
9

(This is weird) I worked that out myself too.

@afelix
I fully agree.

Normally (if there is such a thing) I didn't need to worry about the JSON stuff.

(Back story)

The remote machines run (sometimes) commands as instructed.
As the commands are instructed, it is given that they should work. It isn't I am sending arbitrary commands hoping they will work.

Worst case scenario they fail. Other than a reply saying they didn't work, what can I do?
Yes, it could be an idea to have a handler for each machine that knows the last command and awaits the reply. Possible. But a lot of work - I think.

As it is, commands are sent to the machine/s and a reply is sent to inform me the result.
(Ok/Fail)

That looked ok at the start, but I needed more information. I decided to send back the error as well. Thus the JSON part/s.

That message goes back to the original flow and the message is rearranged (don't know exactly why, but indulge me) and then on-sent to a new flow which is of the thinking that then depending on the message's attributes, a visible indication is given.

These messages are the JSON formatted ones, so by this time in the scheme of things all messages are JSON formatted.

Alas this has propagated to other alarm/error messages too. So I am trying to work out the easyest way to get it done.

So, that's where things are at this time.

Thanks (both) for the replies and help/suggestions.

10

Again, such a construction won’t prevent you from putting the rest of your command structure to JSON format too. It would allow you to add more status messages when needed at any point of time in the future without having to worry about the structure it creates and how to deal with that, like this thread is about. It would be different if you were dealing with third party devices/software, but a payload as generic and debug-like as “Command received”, especially on a Quality of Service level 2 (where you more or less know it will be received), suggests that this comes from another part of your flow. Meaning you can set it up in a way that all payloads would be JSON strings, so when parsed become JavaScript objects ready to read out. Yes, adding a code and status message in JSON helps for debugging, but standardising the format will allow you to add debug information on failure for any kind of payload, current or future. You won’t have to put an additional set of nodes after your MQTT in node, whether switch or function, and parse to JSON if needed. You would have 1 standard format that would work everywhere in your flow.

But again, you don’t have to do it if you don’t want to. It just improves readability and maintainability of your flow, and prevents any future problems similar to this one. In general when doing I/O operations for example over MQTT, having one standardised (base) format for all messages helps.

11

Proper error handling is quite a task; capturing something that you don’t expect: it could be anything and it will be anything (Murphy’s law)

2 Likes
12

Yeah, this is a big "can of worms" I have opened.

Again: Thanks. I'm trying to get my head around how far down this rabbit hole I want to go.

13

Note that detecting if something is valid JSON isn't really as simple as checking if it starts with a "{". Doing this will mean "{foo" is accepted as JSON (which it clearly isn't), probably causing the next node, that's expecting JSON, to throw an error.

An alternative would be to put a try/catch around JSON.parse, eg

try {
  if (JSON.parse(msg.payload)) {
    console.log("it's JSON!");
    //do JSON things
  }
} 
catch(e) {
  console.log("it's NOT JSON!")
  //do string things
}

This will only allow through something that the JSON parser can actually parse.

14

That is true.

Luckily that shouldn't happen given I am sending limited commands to the machines and their replies aren't random. Rather replies like I showed earlier.

So the problem became a problem when I started sending back extra information with the errors.

But it is good you pointed this out.

Thanks.

1 Like
15

Hi Guy @guysqr,
I totally agree with your observations.
The situation one of my 12-year old students was faced with was... deciding it was JSON or a plain text string, and his simple solution worked just fine for him (given it was done in a 45-minute session in the IoT Club).

I'll show (and explain) your solution to him next week.
Thanks for the input.

1 Like
16

Glad to be of assistance! :smiley:

17

Try - catch can be done without coding also

image
image

5 Likes
18

Thanks @hotniPi.
But the other way with a switch node works a bit better in some ways.

19

That's a clever solution, but maybe too clever? It would be reasonable if getting JSON was the normal state but in this case getting a string is going to be common enough to be accepted as within the bounds of normal. I guess I only use catch nodes for catching genuinely unexpected errors.

1 Like
20

Well the catch node carry the original payload and you can "do the string stuff" with it if you like. Some kind of parsing and analyse must be done with it anyway as it may be expected data, unexpected data or just corrupted json string. All same as with function node and try - cach in it.