# MQTT SOM/EOM message formats. Problems
**URL:** https://discourse.nodered.org/t/mqtt-som-eom-message-formats-problems/10215
**Category:** General
**Created:** [16 April 2019 07:16 UTC](https://discourse.nodered.org/t/mqtt-som-eom-message-formats-problems/10215 "2019-04-16T07:16:22Z")
**Posts on this page:** 20
**Page:** 2
### Author:  [@bakman2](https://discourse.nodered.org/u/bakman2)
#### Post date: [19 April 2019 07:13 UTC](https://discourse.nodered.org/t/mqtt-som-eom-message-formats-problems/10215/21 "2019-04-19T07:13:56Z")
So instead of showing the debug output you give us a flow we cannot use.
---
### Author:  [@Trying\_to\_learn](https://discourse.nodered.org/u/Trying_to_learn)
#### Post date: [19 April 2019 07:17 UTC](https://discourse.nodered.org/t/mqtt-som-eom-message-formats-problems/10215/22 "2019-04-19T07:17:10Z")
I'm sure I have posted it here.
Since then it has been _hacked_ and modified with the new idea.
While it is being tested it has a lot of extra stuff in it and external (non-standard) nodes which cause problems for people.
I know that doesn't make it easy for people to debug/help, but I am kind of stuck between a rock and hard place in how to optimise it.
If you want to look at some of the ones I posted - this thread - it is basically two `MQTT` nodes connected to a `JSON` node.
---
### Author:  [@bakman2](https://discourse.nodered.org/u/bakman2)
#### Post date: [19 April 2019 07:28 UTC](https://discourse.nodered.org/t/mqtt-som-eom-message-formats-problems/10215/23 "2019-04-19T07:28:24Z")
It is easy if you explain your problem, you don't explain your problem you just ramble from Z to F, from D to P and from A to E.
I would like to help you but I am missing big parts in your story, especially the 'BirthCert, DeathCert' part.
Can you explain from A to Z how this is supposed to work ?
- device connects to DHCP
- ...
- ...
---
### Author:  [@Trying\_to\_learn](https://discourse.nodered.org/u/Trying_to_learn)
#### Post date: [19 April 2019 07:32 UTC](https://discourse.nodered.org/t/mqtt-som-eom-message-formats-problems/10215/24 "2019-04-19T07:32:42Z")
The problem is that I have a `JSON` node and it is **sometimes** not getting a `JSON` message.
**That** is the problem.
The idea works. It just throws up an error now and then and that sets off alarms which aren't really alarms.
I need(ed) a way to stop that happening.
---
### Author:  [@knolleary](https://discourse.nodered.org/u/knolleary)
#### Post date: [19 April 2019 07:37 UTC](https://discourse.nodered.org/t/mqtt-som-eom-message-formats-problems/10215/25 "2019-04-19T07:37:51Z")
> [@Trying\_to\_learn](#):
>
> The problem is that I have a `JSON` node and it is **sometimes** not getting a `JSON` message.
>
> **That** is the problem.
So you have two choices:
1. Ensure all messages you publish throughout your system are well formed JSON. Not much we can do to help with that as you control the messages.
2. Add some logic in this flow to filter out the messages that are not valid JSON and handle them differently - which is what @hotNipi was pointing towards.
The choice is yours.
---
### Author:  [@Trying\_to\_learn](https://discourse.nodered.org/u/Trying_to_learn)
#### Post date: [19 April 2019 07:43 UTC](https://discourse.nodered.org/t/mqtt-som-eom-message-formats-problems/10215/26 "2019-04-19T07:43:14Z")
Agreed.
But _how_ these incorrect messages are getting in the system (Correction: That MQTT channel/topic) is the sticky point.
HotiNipi may have solved the problem. As I can't control when these errors occur, I shall have to apply it and wait a while and see if the problem goes away.
Yes, I control the messages. But as I just mentioned I am seeing _erroneous_ messages which should not be in the BirtCert' or DeathCert' channel.
I shall have to look into how that is happening.
---
### Author:  [@Colin](https://discourse.nodered.org/u/Colin)
#### Post date: [19 April 2019 07:55 UTC](https://discourse.nodered.org/t/mqtt-som-eom-message-formats-problems/10215/27 "2019-04-19T07:55:01Z")
Show us what one of these messages looks like.
---
### Author:  [@Trying\_to\_learn](https://discourse.nodered.org/u/Trying_to_learn)
#### Post date: [19 April 2019 07:57 UTC](https://discourse.nodered.org/t/mqtt-som-eom-message-formats-problems/10215/28 "2019-04-19T07:57:54Z")
That isn't possible just now - though I have posted previous ones - as the errors are not controllable.
I've put the flow back to the _old_ layout before HotNipi offered his solution.
I'll leave it like that for a few days and see if any thing happens.
If I get an errors, I'll post what I see.
---
### Author:  [@hotNipi](https://discourse.nodered.org/u/hotNipi)
#### Post date: [19 April 2019 08:06 UTC](https://discourse.nodered.org/t/mqtt-som-eom-message-formats-problems/10215/29 "2019-04-19T08:06:48Z")
`{"topic":"EOM","payload":"'Tele Pi telemetry failure'","qos":0,"retain":true, .....`
Such kind of messages should not be feed directly into `json` node as the payload is plain string.
If you are controlling the creation of those messages, you should format them to be proper `JSON`. If not, you have option of error caching or you should check incoming message payload type before feeding them into `json` node.
---
### Author:  [@Trying\_to\_learn](https://discourse.nodered.org/u/Trying_to_learn)
#### Post date: [19 April 2019 08:10 UTC](https://discourse.nodered.org/t/mqtt-som-eom-message-formats-problems/10215/30 "2019-04-19T08:10:27Z")
Thanks.
(Obviously) that is a RPI.
It is from the `MQTT` node in the Birth and Death Certificates.
The GPS ones are from an Arduino.
There will be more Arduinos coming online at some stage.
So to check:
When MQTT connects to a server the Birth and Death Certificates are exchanged.
Would/should that be in `JSON` format?
---
### Author:  [@hotNipi](https://discourse.nodered.org/u/hotNipi)
#### Post date: [19 April 2019 08:11 UTC](https://discourse.nodered.org/t/mqtt-som-eom-message-formats-problems/10215/31 "2019-04-19T08:11:56Z")
Sounds right thing to do.
---
### Author:  [@knolleary](https://discourse.nodered.org/u/knolleary)
#### Post date: [19 April 2019 12:26 UTC](https://discourse.nodered.org/t/mqtt-som-eom-message-formats-problems/10215/32 "2019-04-19T12:26:47Z")
> [@Trying\_to\_learn](#):
>
> When MQTT connects to a server the Birth and Death Certificates are exchanged.
> Would/should that be in `JSON` format?
Mqtt does not require any particular format. It is up to you at the application layer to decide what format your payloads use.
If you want to be able to treat every payload as JSON then you need to make sure everything that gets published is in a JSON format.
If some things might not be JSON then you need to handle that. It may be you can identify the non JSON messages by virtue of what topic they are published on - in which case you can use a Switch node to direct those messages to a different flow.
Ultimately it comes down to what you want to do as, ultimately, you have to maintain it all.
---
### Author:  [@Colin](https://discourse.nodered.org/u/Colin)
#### Post date: [19 April 2019 12:43 UTC](https://discourse.nodered.org/t/mqtt-som-eom-message-formats-problems/10215/34 "2019-04-19T12:43:49Z")
> [@Trying\_to\_learn](#):
>
> When MQTT connects to a server the Birth and Death Certificates are exchanged.
> Would/should that be in `JSON` format?
The birth and death messages (they are messages not certificates) are under your control (or under the control of whoever configures the MQTT connection that is connecting or diconnecting), so probably you can put what you like there.
---
### Author:  [@Trying\_to\_learn](https://discourse.nodered.org/u/Trying_to_learn)
#### Post date: [19 April 2019 22:18 UTC](https://discourse.nodered.org/t/mqtt-som-eom-message-formats-problems/10215/35 "2019-04-19T22:18:41Z")
Nick/Colin.
Ok. Thanks. So their format is under my control.
(I'll look into that.)
Here are some messages/errors detected overnight.
```auto
{"topic":"EOM","payload":"GPS comms failure","qos":0,"retain":false,"_msgid":"c38fa4a4.e41c98","error":{"message":"Unexpected token G in JSON at position 0","source":{"id":"bf208502.79b7a","type":"json","count":1}},"ttl":0,"_queuetimestamp":1555704121328,"queueCount":3}
```
```auto
{"topic":"EOM","payload":"GPS comms failure","qos":0,"retain":false,"_msgid":"c38fa4a4.e41c98","error":{"message":"Unexpected token G in JSON at position 0","source":{"id":"c6927562.fa0738","type":"json","name":"tele-json","count":1}},"ttl":0,"_queuetimestamp":1555704121393,"queueCount":2}
```
This one is a bit weird, but.....
```auto
{"topic":"SOM","payload":"'Awaiting Pi Face'","qos":2,"retain":false,"_msgid":"812e07fb.b97408","error":{"message":"Unexpected token ' in JSON at position 0","source":{"id":"bf208502.79b7a","type":"json","count":1}},"ttl":0,"_queuetimestamp":1555711153860,"queueCount":1}
```
As is this one
```auto
{"topic":"SOM","payload":"'Awaiting Pi Face'","qos":2,"retain":false,"_msgid":"812e07fb.b97408","error":{"message":"Unexpected token ' in JSON at position 0","source":{"id":"c6927562.fa0738","type":"json","name":"tele-json","count":1}},"ttl":0,"_queuetimestamp":1555711153879,"queueCount":0}
```
Yes, it appears I shall have to go through and check a lot of my `MQTT` nodes.
Here's my take on what I am seeing:
The _GPS_ (WiFi link) failed a couple of times. I guess I need to humanise the `_queuetimestamp` to know when it happened.
Twice overnight? Ok. Strange. But why aren't I seeing the Birth Certificates?
The _GPS_ is connected and working.
_Awaiting Pi Face_ That's a bit of a doozy. He is wired - though wired BC and DC are detected, but anyway.....
(Via edit)
Ok, the _format_ of the packet out of the `MQTT` node.
It is set to `Auto-detect string or buffer`
So _ **shouldn't** _ that stop this kind of error happening anyway?
This is the code for the two nodes:
```auto
[{"id":"c9482822.f11c8","type":"mqtt in","z":"3913e6fb.8d0352","name":"","topic":"EOM","qos":"2","broker":"931f34a.34a47c8","x":210,"y":1250,"wires":[["5bfb5b93.bcc8b4"]]},{"id":"e48386f2.057dd","type":"mqtt in","z":"3913e6fb.8d0352","name":"","topic":"SOM","qos":"2","broker":"931f34a.34a47c8","x":210,"y":1300,"wires":[["5bfb5b93.bcc8b4"]]},{"id":"931f34a.34a47c8","type":"mqtt-broker","z":"","name":"MQTT host","broker":"192.168.0.99","port":"1883","clientid":"","usetls":false,"compatmode":false,"keepalive":"60","cleansession":true,"birthTopic":"","birthQos":"2","birthPayload":"","closeTopic":"","closePayload":"","willTopic":"","willQos":"0","willPayload":""}]
```
I'll stop now because I am not sure what else is needed.
---
### Author:  [@knolleary](https://discourse.nodered.org/u/knolleary)
#### Post date: [19 April 2019 22:33 UTC](https://discourse.nodered.org/t/mqtt-som-eom-message-formats-problems/10215/36 "2019-04-19T22:33:46Z")
All of those error messages you've shared are coming from a JSON node - you are passing text that isn't JSON to the JSON node and asking it to parse it.
> [@Trying\_to\_learn](#):
>
> Ok, the _format_ of the packet out of the `MQTT` node.
> It is set to `Auto-detect string or buffer`
> So _ **shouldn't** _ that stop this kind of error happening anyway?
The MQTT node has seen the payload is a string so it has converted the raw bytes to a string and passed it on. That has nothing to do with whether the string contains JSON or just plain text.
---
### Author:  [@Trying\_to\_learn](https://discourse.nodered.org/u/Trying_to_learn)
#### Post date: [19 April 2019 22:48 UTC](https://discourse.nodered.org/t/mqtt-som-eom-message-formats-problems/10215/37 "2019-04-19T22:48:09Z")
> [@knolleary](#):
>
> The MQTT node has seen the payload is a string so it has converted the raw bytes to a string and passed it on. That has nothing to do with whether the string contains JSON or just plain text.
Ok. My bad.
Here's where I am stuck though:
I'm just stating what has happened in the past.
That small part is to trap errors on **this** flow.
I have a _switch_ to stop it going any further - like when I have this kind of pesky bug.
If normal, this is sent on to a _bigger_ flow and it gets error reports from **all** machines.
I then want to `split` said things to filter by machine name.
So I have/had a `MQTT` receive node and then a `switch` node.
I was stuck (way back then) because I couldn't get the `switch` node to work correctly.
The data looked fine and I was trying to get my head around why it wasn't working.
I was trying to add/remove leading and trailing { } symbols.
You mentioned that I needed to put the message through a `JSON` node to make the `switch` node work/be able to _switch_/parse the data.
I seem to remember something about "As you are sending more than a simple msg.payload, when you use MQTT, it gets converted to a JSON package..... To get it back to the original format/type/style you need to send it through a `JSON` node."
Ok, I can't quote the exact reply. (Maybe I should search it before posting, but I am working in the "here and now".)
So, in light or recent information, I have deleted the original `JSON` node and left the bit that HotiNip mentioned.
I'll give that a spin around the block and see what happens.
Reading a bit deeper, I may not even need that. But: One step at a time.
Maybe off topic:
Ok, I did a bit of digging on the actual messages I posted.
The first error:
It was at: 2019-04-19T20:02:01.328Z
20:02:01Z + 10
30:02:01 (mod 24)
06:02 local.
Ok. I was asleep.
Reading MQTT stuff the Death and Birth Certificates are negotiated when the connection is set up and sent as needed.
I'm still (obviously) not quite getting that 100%.
But PiFace.... It is offline and no where to be found/seen.
Why would it's Certificate be broadcast?
(Yeah, ok: I put the wrong values in a node - probably)
How can I find **which node** is sending it?
There is only two devices connected.
TimePi and GPS.
---
### Author:  [@Trying\_to\_learn](https://discourse.nodered.org/u/Trying_to_learn)
#### Post date: [19 April 2019 23:56 UTC](https://discourse.nodered.org/t/mqtt-som-eom-message-formats-problems/10215/38 "2019-04-19T23:56:58Z")
I can be lucky.
Was poking around and saw this happen.
This is what I mean by saying I am seeing the wrong messages coming through in/on the wrong MQTT channels.
The _GPS_ decided it wasn't talking. You see two of those messages.
Then shortly after you get an _EOM_ (Which is what I call Death Certificates) come through with a payload of `X`.
`X` is sent on the `IFF` channel when a new WiFi connection is detected.
There is no reply. It is a simplex signal, published on the `IFF` channel.
Can someone help me with this?
(Is it helpful to find the _Gremlin_ which seems to still be here.)
---
### Author:  [@Colin](https://discourse.nodered.org/u/Colin)
#### Post date: [20 April 2019 07:41 UTC](https://discourse.nodered.org/t/mqtt-som-eom-message-formats-problems/10215/39 "2019-04-20T07:41:16Z")
Those messages must be setup in MQTT config nodes that you have configured, includeing the "X". Have you used the same topic for all the birth and death messages you have configured? If not then then have a look at what is coming from the MQTT nodes to see where they are coming from. Also if you look at the mosquitto log (assuming you are using mosquitto) you will see them connecting/disconnecting.
The "Error detected ..." message is coming from a warn message that you have put in a function node. Click on the node id above the message and it will show you which node.
---
### Author:  [@Trying\_to\_learn](https://discourse.nodered.org/u/Trying_to_learn)
#### Post date: [20 April 2019 09:20 UTC](https://discourse.nodered.org/t/mqtt-som-eom-message-formats-problems/10215/40 "2019-04-20T09:20:40Z")
Thanks Colin.
(I've tried searching "node-red mqtt log" and all I get is how to log data with MQTT. Not where the log is. I will try /var/log/mosquitto.log)
Looked at /var/log/mosquitto/...... Yeah. I can't see anything there of interest.
Then went to /var/lib/mosquitto/mosquitto.db
Wow! Big file.
Did `tail mosquitto.db` Alas not much interesting there either.
ITMT, I'll keep looking.
Can I first off ask for help on that.
The 'X' is inserted to MQTT channel 'IFF'.
(
The lay out is a bit "all over the shop" but I have cut out the nodes of importance for this.
Please excuse the `FAN` nodes that will probably be missing for you.
They just give me a neater layout. I know I could just use blank/empty `Function` nodes, but....
)
There is also a `GATE` node just near the start which is because I am testing this part and have the alternative part switched/gated from that point.
Again: Yes, I know. Messy. It is still a bit of "work in progress."
Code:
```auto
[{"id":"814858d2.003e58","type":"inject","z":"3913e6fb.8d0352","name":"IFF","topic":"","payload":"X","payloadType":"str","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":2020,"y":520,"wires":[["729709b0.71b9a"]]},{"id":"c9482822.f11c8","type":"mqtt in","z":"3913e6fb.8d0352","name":"","topic":"EOM","qos":"2","broker":"931f34a.34a47c8","x":230,"y":1250,"wires":[["5bfb5b93.bcc8b4"]]},{"id":"e48386f2.057dd","type":"mqtt in","z":"3913e6fb.8d0352","name":"","topic":"SOM","qos":"2","broker":"931f34a.34a47c8","x":230,"y":1300,"wires":[["5bfb5b93.bcc8b4"]]},{"id":"5bfb5b93.bcc8b4","type":"fan","z":"3913e6fb.8d0352","x":390,"y":1280,"wires":[["f654881.7ae28f8","4993a3ef.94fa3c"]]},{"id":"f654881.7ae28f8","type":"gate","z":"3913e6fb.8d0352","name":"A","controlTopic":"control","defaultState":"closed","openCmd":"A","closeCmd":"B","toggleCmd":"toggle","defaultCmd":"default","persist":false,"x":520,"y":1280,"wires":[["3847532e.ad6a1c"]]},{"id":"3847532e.ad6a1c","type":"fan","z":"3913e6fb.8d0352","x":680,"y":1280,"wires":[["1f93afa4.be3ee"]]},{"id":"1f93afa4.be3ee","type":"fan","z":"3913e6fb.8d0352","x":890,"y":1280,"wires":[["337780bd.50079","b3f64ec4.a83f78","5d4762e7.eac9d4","48c770ea.d0ec2"]]},{"id":"5d4762e7.eac9d4","type":"fan","z":"3913e6fb.8d0352","x":910,"y":940,"wires":[["92e82e4e.b3213"]]},{"id":"92e82e4e.b3213","type":"change","z":"3913e6fb.8d0352","name":"IFF Request command generator","rules":[{"t":"set","p":"payload","pt":"msg","to":"X","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":1080,"y":850,"wires":[["1a3b6900.7b01c7"]]},{"id":"1a3b6900.7b01c7","type":"link out","z":"3913e6fb.8d0352","name":"","links":["41f22411.72db44"],"x":1275,"y":850,"wires":[]},{"id":"41f22411.72db44","type":"link in","z":"3913e6fb.8d0352","name":"IFF request RX","links":["1a3b6900.7b01c7"],"x":2055,"y":850,"wires":[["729709b0.71b9a"]]},{"id":"729709b0.71b9a","type":"fan","z":"3913e6fb.8d0352","x":2160,"y":570,"wires":[["9ba1382f.d521a","3ab3e172.0b52be"]]},{"id":"3ab3e172.0b52be","type":"mqtt out","z":"3913e6fb.8d0352","name":"Identify (IFF)","topic":"IFF","qos":"2","retain":"","broker":"931f34a.34a47c8","x":2340,"y":520,"wires":[]},{"id":"931f34a.34a47c8","type":"mqtt-broker","z":"","name":"MQTT host","broker":"192.168.0.99","port":"1883","clientid":"","usetls":false,"compatmode":false,"keepalive":"60","cleansession":true,"birthTopic":"","birthQos":"2","birthPayload":"","closeTopic":"","closePayload":"","willTopic":"","willQos":"0","willPayload":""}]
```
---
### Author:  [@zenofmud](https://discourse.nodered.org/u/zenofmud)
#### Post date: [20 April 2019 09:39 UTC](https://discourse.nodered.org/t/mqtt-som-eom-message-formats-problems/10215/41 "2019-04-20T09:39:25Z")
What node - i.e. node-red-contrib-xxxxx is the `fan` node?
What node - i.e. node-red-contrib-xxxxx is the `gate` node?
Without any other info, it looks like you put the letter `X` into msg.payloaf and then send it thru the `fan` node and on to MQTT.
If the `fan` node is not doing anyting to msg.payload then you are sending a `string` into the MQTT-out node.
[Previous page](https://discourse.nodered.org/t/mqtt-som-eom-message-formats-problems/10215.md?page=1)
[Next page](https://discourse.nodered.org/t/mqtt-som-eom-message-formats-problems/10215.md?page=3)