MQTT SOM/EOM message formats. Problems

#1

"Originally" I had a bit of flow that looked at MQTT messages when devices connect to the WiFi network.

This is what I get:

{"topic":"STATUS/WIFIDEVICEID","payload":"{\"WIFI_DEVICE\":\"GPS\",\"IP_Address\":\"192.168.1.4\"}","qos":0,"retain":false,"_msgid":"fb4161b2.ecdcf","time":"2019-4-16 16:48:25","ttl":0,"_queuetimestamp":1555397305156,"queueCount":1}

Now, a few days ago it was ok.

Then, while working on something else, and the machine locked up, then when I got it working again it didn't work.

The switch node worked with something like the message.
Then to get it working I had to change the switch node settings from what ever they were to this:

(msg.)payload["IP Address"] In the property field.

And it worked, weirdly.

Today I reluctantly after finding out about node-red --safe I sat down and did the hard yard loading the flow which caused the problem.

Edited the offending part and what ever.

It is now working.

But here it gets interesting.

The WiFi connected part of the flow stopped working.

I also had to edit a function node from what it was to this:

var mpayload = msg.payload;
var mtopic = msg.topic;
var name = context.get('name');
var count = context.get('count') || 0;

if (mtopic === 'STATUS/WIFIDEVICEID')
{
    //
    //node.warn("Name recived is " + msg.payload["WIFI-DEVICE"]);
    context.set('name', msg.payload["WIFI-DEVICE"]);
    count = 1;
    context.set('count',count);
    node.status({fill:"yellow",shape:"dot",text:"WiFi Device received"});

}


if (msg.payload === 'On-line')
{
    //
    if (count === 1)
    {
        //
        node.status({fill:"green",shape:"dot",text:"message modified"});
        msg.topic = name;
        return msg;
    }
    return msg;
}
if (msg.payload === 'Off-line')
{
    //
    node.status({fill:"red",shape:"dot",text:"Off Line"});
    count = 0;
    context.set('count',count);
    return msg;
}

Note the line(s):

    //node.warn("Name recived is " + msg.payload["WIFI-DEVICE"]);
    context.set('name', msg.payload["WIFI-DEVICE"]);

Now, I am not getting something - again.

It was working.

I didn't change anything to do with this and suddenly it stopped working.

It did work a week ago. In that time the machine had the big crash and I had to spend a lot of time getting it working again.

Now I have to change things back to what they were before. (A week ago)

I made the line (from the function node) to:
context.set('name', msg.payload["WIFI_DEVICE"]);

And the switch from:
(msg.)payload["IP Address"] In the property field.
to payload.IP_Address

Why is this happening?

0 Likes

#2

You need to work on your questions.

What is it what you are trying to accomplish and start from there, there is no context here.

0 Likes

#3

As I said in another thread, just post the bit of flow that is not working, show the input to it, and explain what output you expect and what you are getting. We do not need your life story in every post.

0 Likes

#4

Obviously the code you have is from a crash restore and half correct.

The reason values are in payload.IP_Address and msg.payload["WIFI_DEVICE"] are because (as you said) that's how they arrive!

There is a fine line between not enough information and too much. You are a little on the on the too much side of the fence but it's better than too little :slight_smile:

Oh and one last thing... Try to be consistent. You use payload.IP_Address and msg.payload["WIFI_DEVICE"]I would suggest you stick with dot notation e.g.payload.IP_Addressand msg.payload.WIFI_DEVICE

Edit...
Forgot to say, your data
"payload":"{"WIFI_DEVICE":"GPS","IP_Address":"192.168.1.4"}" looks like it's a string not an object. You may need to send the payload through a JSON node or call var myPayload = JSON.parse(msg.payload); if you aren't already.

0 Likes

#5

Hey Steve.

Thanks. And very much thanks for reading that long post.

I think the message got lost in translation. My fault.

Originally (to call it something) the msg.payload was like: (say) "payload":"{"WIFI_DEVICE":"GPS","IP_Address":"192.168.1.4"}"

That goes through a switch node then through a function node.

You will see I have a json node in the flow.

There is two parts to the code/flow. (Two different parts of this code on different tabs)
Here they are:

[{"id":"acc9188c.41e128","type":"mqtt in","z":"3913e6fb.8d0352","name":"IFF","topic":"STATUS/WIFIDEVICEID","qos":"2","datatype":"auto","broker":"931f34a.34a47c8","x":130,"y":670,"wires":[["91c63083.8badd8","e5b68b69.8a5b5"]]},{"id":"91c63083.8badd8","type":"json","z":"3913e6fb.8d0352","name":"","property":"payload","action":"","pretty":false,"x":370,"y":670,"wires":[["cb167a65.72496","19b3623.e9f039e","9ef9e107.309728","788d1112.1456e8"]]},{"id":"788d1112.1456e8","type":"link out","z":"3913e6fb.8d0352","name":"Arduino test link","links":["5b53b5b6.d608ec"],"x":725,"y":670,"wires":[]},{"id":"931f34a.34a47c8","type":"mqtt-broker","z":"","name":"MQTT host","broker":"192.168.0.99","port":"1883","clientid":"","usetls":false,"compatmode":false,"keepalive":"60","cleansession":true,"birthTopic":"","birthQos":"2","birthPayload":"","closeTopic":"","closePayload":"","willTopic":"","willQos":"0","willPayload":""}]

Then there is this:

[{"id":"b0130538.b638e8","type":"function","z":"c636aa5a.cc34","name":".1.4","func":"var mpayload = msg.payload;\nvar mtopic = msg.topic;\nvar name = context.get('name');\nvar count = context.get('count') || 0;\n\nif (mtopic === 'STATUS/WIFIDEVICEID')\n{\n    //\n    context.set('name', msg.payload[\"WIFI_DEVICE\"]);\n    count = 1;\n    context.set('count',count);\n    node.status({fill:\"yellow\",shape:\"dot\",text:\"WiFi Device received\"});\n\n}\n\n\nif (msg.payload === 'On-line')\n{\n    //\n    if (count === 1)\n    {\n        //\n        node.status({fill:\"green\",shape:\"dot\",text:\"message modified\"});\n        msg.topic = name;\n        return msg;\n    }\n    return msg;\n}\nif (msg.payload === 'Off-line')\n{\n    //\n    node.status({fill:\"red\",shape:\"dot\",text:\"Off Line\"});\n    count = 0;\n    context.set('count',count);\n    return msg;\n}","outputs":1,"noerr":0,"x":2810,"y":760,"wires":[["796a929c.de4bd4","26d836fe.91b592","dcaec98b.2c53d8"]]},{"id":"9632566f.44483","type":"switch","z":"c636aa5a.cc34","name":"","property":"payload.IP_Address","propertyType":"msg","rules":[{"t":"cont","v":"192.168.1.1","vt":"str"},{"t":"cont","v":"192.168.1.2","vt":"str"},{"t":"cont","v":"192.168.1.3","vt":"str"},{"t":"cont","v":"192.168.1.4","vt":"str"},{"t":"cont","v":"192.168.1.5","vt":"str"},{"t":"cont","v":"192.168.1.6","vt":"str"},{"t":"cont","v":"192.168.1.7","vt":"str"},{"t":"cont","v":"192.168.1.8","vt":"str"},{"t":"cont","v":"192.168.1.9","vt":"str"}],"checkall":"true","repair":false,"outputs":9,"x":2310,"y":950,"wires":[[],["1b8f26d4.926bd9"],["a254b746.1244c8","e63a5ce1.3a1778","8f2ca8a4.57558"],["b0130538.b638e8","4c9f5938.62b81","8ca81413.798288"],["bd03d38e.249458"],["a4b5c8dd.95de8"],["81d452a8.20a048"],["64b8ac8a.3d768c"],["8c1bd9be.34a278"]]},{"id":"5b53b5b6.d608ec","type":"link in","z":"c636aa5a.cc34","name":"Arduino test link","links":["65774363.a187d4","5f843fa2.736b88","1e0934ef.a7a493","c016f0fc.671588","11aa545d.8cb9dc","788d1112.1456e8"],"x":2115,"y":950,"wires":[["9632566f.44483","8fc197d3.a94338","f25b2d49.78ae5"]]}]

That part of the code was never changed. When the big problem happened and I kind of got it working, the entire format of said messages changed to the other format.
context.set('name', msg.payload["WIFI_DEVICE"]);

So suddenly it stopped working. I had to go in and change it to the latter format.

Then, after all that fun, suddenly it needed to be changed back.

I really don't know what is happening or why.

Basically this is the plan for what I am wanting to do:
It did work at one stage, stopped, I got it working again, it stopped again and I got it working again - to the best of my knowledge that is where I am at now.

As it is, I have a couple of devices that connect to my WiFi which support what I call 'iff' (Identify - Friend or Foe)

So, when a device connects, all is good, but as it uses DHCP, I don't know "who is who" on the list of "connected devices" I see.
I made an "IFF button" which is sent to all devices via MQTT. When received, the device transmits an "IFF packet" back.

This has the IP address of the device and its name in the packet.
That data is slit up and the name sent to a flow which renames the device with the name, rather than the IP.

The SOM/EOM bit is used to detect a new connection. That generates a pules to trigger the "IFF button" being pressed.

0 Likes

#6

Today I logged in to the machine and I was told there are some errors.

These are to do with the SOM/EOM part I posted above.

I have a MQTT node and that goes to a JSON node to allow parsing. As I was told and thought, anyway.

But I see two errors:

{"topic":"EOM","payload":"BedPi Offline (WiFi)","qos":0,"retain":false,"_msgid":"99e67bfd.fb8578","error":{"message":"Unexpected token B in JSON at position 0","source":{"id":"bf208502.79b7a","type":"json","count":1}},"ttl":0,"_queuetimestamp":1555536468953,"queueCount":1}

and

{"topic":"SOM","payload":"'Awaiting Pi Face'","qos":2,"retain":false,"_msgid":"5aebae98.99872","error":{"message":"Unexpected token ' in JSON at position 0","source":{"id":"bf208502.79b7a","type":"json","count":1}},"ttl":0,"_queuetimestamp":1555577375459,"queueCount":0}

I don't remember these errors happening before.
To me it implies there is something happening with the SOM/EOM handlers.
But if it used to work and I haven't done anything to/with it, why is it suddenly not working?

0 Likes

#7

I am looking at your flows and your comments above.

What is SOM/EOM ?
Your errors mention topic:EOM/SOM, your function node is not handling those.

Where are the "'On-line/Offline" coming from ?

You still didn't explain what you are trying to accomplish

0 Likes

#8

Sorry....

SOM - EOM: They are the MQTT "Birth Certificate" and "Death Certificate".

The idea is that when a device connects MQTT gets the Birth Certificate and it is sent to a function to then force the bit of code to then get the "IFF" request to be published to devices.

Their reply is then used to modify a node so rather than the IP number, you get the device's name.

0 Likes

#9

Still no answer on the questions asked.

But ok in terms of your switch node, is not easier to use an array with definitions ? eg;

ip = msg.payload

var devices = [
  { "ipaddress": "192.168.1.1", "deviceName": "Raspberry" },
  { "ipaddress": "192.168.1.2", "deviceName": "GPS" }
];

device = devices.filter(i => i.ipaddress == ip );

return {payload:device[0].deviceName}

flow

[{"id":"56d33e80.b88ac","type":"inject","z":"811bc2a8.14b62","name":"","topic":"","payload":"192.168.1.1","payloadType":"str","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":150,"y":740,"wires":[["7f948b7c.42be54"]]},{"id":"7f948b7c.42be54","type":"function","z":"811bc2a8.14b62","name":"","func":"ip = msg.payload\n\nvar devices = [\n  { \"ipaddress\": \"192.168.1.1\", \"deviceName\": \"Raspberry\" },\n  { \"ipaddress\": \"192.168.1.2\", \"deviceName\": \"GPS\" }\n];\n\ndevice = devices.filter(i => i.ipaddress == ip );\n\nreturn {payload:device[0].deviceName}","outputs":1,"noerr":0,"x":290,"y":740,"wires":[["ffa72574.fea6f"]]},{"id":"ffa72574.fea6f","type":"debug","z":"811bc2a8.14b62","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","x":450,"y":740,"wires":[]}]
0 Likes

#10

Again: Thanks. It isn't I am deliberately avoiding your questions. (Ok, I'll do this bit again)
(Dont' know how to "cross out" text)
Please understand I want to answer your questions.
It is me who is not understanding them, and I am not doing it on purpose.

I shall look at the flow you posted here tomorrow. It is 22:05 local time here.

Ok, another attempt to answer the question/s:

The SOM/EOM are the connection messages when the WiFi device connects to the WAP.
It is the MQTT birth and death certificates, in reality. So: OK, there may be a bit of conflict when a non wifi device connects. But let's (for now) look at the bigger picture.

The idea is when a device connects, a trigger is sent (from the "SOM/EOM" bit of code) to the bit of code/method (?) that requests/published IFF via MQTT.

When the WiFi device received that message, it publishes a fixed packet/message.
The device's name (limited to 5 letters for screen neatness) and the IP address.

That is then JSON'ed to something when it is sent over MQTT. Then when received and put through the JSON node it can be used and I switch it and send the name to a node that replaces the IP address with the name.

(Just going back to something you said now. Just mentioning)

Yeah. I fully agree.

I did do that - from memory - and it worked.
The message was constructed on the Arduino and I did a similar thing when I wrote the code if it is a RPI (Node-Red) connecting.
Then, one day it decided it wanted the other. Somehow when I got a debug node and looked at the output and copied the path, it wasn't what I had. It was the other format/structure of how the msg.payload was constructed.
This meant editing the switch node and a heap of other nodes which do the IP address/Name replacements.

Then, the next time, (or maybe 2 times down the time line) it decided it wanted to go back to the original way.

Considering I hadn't done anything to those parts of the flows, messages or .... what ever.
It confuses me why/how that happened.

So, you then mentioned:

Again: No problems.

I have put a JSON node after the MQTT receive node.

But, I think that is catching me out.

Somehow/sometimes, something comes out of there which makes the error about

"Unexpected token B in JSON at position 0"

Great. But all the messages sent to that should comply with that is being parsed, and they should be JSON packets, as they are coming out of a MQTT RX node.

When I catch the error and try to drill down on WHO is doing it, I just get pointed back to the JSON node just after the MQTT RX node.
There are debug nodes, but what they show me doesn't help.

I shall try to capture that and post it. (If I haven't already) But I guess two copies of it won't/can't hurt.

It is just inducing it to happen is not something I know how to do.
So it isn't I can force the error and work on it.
It happens sporadically.

Bed is calling.

Again: Thanks. I hope this helped with what is happening.

0 Likes

#11

We need a Rosetta Stone. Sorry, I have read this a few times and I really do not understand what your problem is!!!

1 Like

#12

ghayne.... :slight_smile: Yeah, I'm not the best at explaining things I don't understand.

Here goes again:

Put what you have read aside and try this version:

I am trying to use MQTT Birth and Death certificates (more so Birth, I think) to trigger a signal to invoke another MQTT publication.

WHY?

I have some devices which connect via WiFi and use DHCP to obtain their IP address.

That's all well and good. That side of it works. The device/s connect and work.

I have a flow which shows the status of the IP addresses. As most of my network is FIXED IP, that isn't a problem. I know who is who at that level, but the DHCP isn't fixed.
Granted there is a long lease time on IP addresses, but anyway......

When a device connects, I have two MQTT nodes which catch the Birth & Death certificates.
When a Brith certificate is received, it triggers another part of the flow to send/publish an IFF request via MQTT.

The newly connected device then replies with its own unique reply. This contains the IP number (for identification) and its name.

That packet is received and dealt with through a SWITCH node and sent to a FUNCTION node to modify the display so the name is displayed rather than the IP address.

All well and good at this stage.

So: What's the problem?

Well, putting aside the recent problem: I am getting JSON ERRORS like this:

{"topic":"EOM","payload":"'Tele Pi telemetry failure'","qos":0,"retain":true,"_msgid":"d4895511.66ba18","error":{"message":"Unexpected token ' in JSON at position 0","source":{"id":"bf208502.79b7a","type":"json","count":1}},"ttl":0,"_queuetimestamp":1555587482644,"queueCount":5}

I seem to remember being told by Nick (and more recently having it mentioned) that I need to use a JSON node after a MQTT node if I want to easily parse the message.

So that is there.

But! (Here's the rub): Now and then for no apparent reason I get that error.

Why?

0 Likes

#13

Yes who doesn't ? This is normal networking. Still doesn't explain why/what you are doing.

What are you trying to accomplish ? Building a DHCP server in node-red ? What is the purpose ?

0 Likes

#14

Because ofmsg.payload sent to json node is plain string. Not a valid JSON.
I have same problem with tasmota LWT messages but there is workaround:

[{"id":"bd6e4368.4cf46","type":"mqtt in","z":"ba86d109.1f011","name":"","topic":"tele/#","qos":"1","broker":"63c80901.f61868","x":150,"y":340,"wires":[["d7f2018b.7622f"]]},{"id":"d7f2018b.7622f","type":"json","z":"ba86d109.1f011","name":"tele-json","property":"payload","action":"","pretty":false,"x":280,"y":340,"wires":[["73e49156.2b0bc","f843708a.20385"]]},{"id":"28d0d9f6.c994e6","type":"comment","z":"ba86d109.1f011","name":"Telemetry","info":"Listen telemetry data from sonoff devices.\nParse into inividual registered device objects\nby registered parameters.","x":160,"y":260,"wires":[]},{"id":"9931ca77.f651b8","type":"catch","z":"ba86d109.1f011","name":"","scope":["d7f2018b.7622f"],"x":150,"y":300,"wires":[["348add0.38b9224"]]},{"id":"3067a8fb.344568","type":"comment","z":"ba86d109.1f011","name":"Catch","info":"Not all telemetry messages have payload in JSON\nformat. For example **LWT** (Last Will & Testament)\nIt still catched here and forwarded to parsing.\nOnly LWT is currently in scope. Others (if any) dropped","x":280,"y":260,"wires":[]},{"id":"348add0.38b9224","type":"switch","z":"ba86d109.1f011","name":"LWT ↓","property":"topic","propertyType":"msg","rules":[{"t":"jsonata_exp","v":"$not($contains($split(topic, \"/\")[2] , \"LWT\"))\t","vt":"jsonata"},{"t":"else"}],"checkall":"true","repair":false,"outputs":2,"x":270,"y":300,"wires":[["f22b91ff.9c13a"],["73e49156.2b0bc"]]},{"id":"63c80901.f61868","type":"mqtt-broker","z":"","name":"keldriarvuti broker","broker":"192.168.1.111","port":"1883","tls":"","clientid":"","usetls":false,"compatmode":false,"keepalive":"60","cleansession":true,"birthTopic":"","birthQos":"0","birthPayload":"","closeTopic":"","closeQos":"0","closePayload":"","willTopic":"","willQos":"0","willPayload":""}]

This problem rises if you can't be sure what kind of incoming messages to expect. Usually you do and then you use json node only for messages where it guaranteed to be JSON
I listen full telemetry of all my devices with one MQTT node and therefore both types of messages (JSON and String) expected.

1 Like

#15

Thanks hotNipi

So, not wanting to sound stupid, I'll ask.
(Nah, I'm stupid.)

The output of the switch (LWT) node..... They get stuck together, right?
And that is the message what arrived from the MQTT node.

0 Likes

#16

After catching the error of json node, the switch node validates if the msg contains LWT. (I just ignore any other string type messages if such even exist) LWT msg is then routed to same flow where regular messages go.

1 Like

#17

I have just applied it and shall wait and see.

I think I will have further use for that bit as I have this problem else where.

0 Likes

#18

No, it isn't a DHCP server.

What I want is: On my screen which shows the devices connected, rather than just their IP, it has their name.

See attached.

Screenshot%20from%202019-04-19%2017-01-05

0 Likes

#19

I should mention that "Error-driven solution" is not the most practical one. It may seem to be the easiest one to figure out, but really should be the last choice to actually develop.

1 Like

#20

I agree.
Just at this stage, I need a solution to a false problem that is annoying at this point.

What I have noticed is that I get some weird messages out of the MQTT node.

[{"id":"c9482822.f11c8","type":"mqtt in","z":"3913e6fb.8d0352","name":"","topic":"EOM","qos":"2","broker":"931f34a.34a47c8","x":210,"y":1250,"wires":[["5bfb5b93.bcc8b4"]]},{"id":"e48386f2.057dd","type":"mqtt in","z":"3913e6fb.8d0352","name":"","topic":"SOM","qos":"2","broker":"931f34a.34a47c8","x":210,"y":1300,"wires":[["5bfb5b93.bcc8b4"]]},{"id":"931f34a.34a47c8","type":"mqtt-broker","z":"","name":"MQTT host","broker":"192.168.0.99","port":"1883","clientid":"","usetls":false,"compatmode":false,"keepalive":"60","cleansession":true,"birthTopic":"","birthQos":"2","birthPayload":"","closeTopic":"","closePayload":"","willTopic":"","willQos":"0","willPayload":""}]

I send other messages and I see them coming back on the BirthCert' or DeathCert' channel.

So: If that is happening, there is another Gremlin hiding in there somewhere.

I will try to catch examples and post. How it is happening is beyond me.

0 Likes