So instead of showing the debug output you give us a flow we cannot use.
I'm sure I have posted it here.
Since then it has been hacked and modified with the new idea.
While it is being tested it has a lot of extra stuff in it and external (non-standard) nodes which cause problems for people.
I know that doesn't make it easy for people to debug/help, but I am kind of stuck between a rock and hard place in how to optimise it.
If you want to look at some of the ones I posted - this thread - it is basically two MQTT nodes connected to a JSON node.
It is easy if you explain your problem, you don't explain your problem you just ramble from Z to F, from D to P and from A to E.
I would like to help you but I am missing big parts in your story, especially the 'BirthCert, DeathCert' part.
Can you explain from A to Z how this is supposed to work ?
- device connects to DHCP
- ...
- ...
The problem is that I have a JSON node and it is sometimes not getting a JSON message.
That is the problem.
The idea works. It just throws up an error now and then and that sets off alarms which aren't really alarms.
I need(ed) a way to stop that happening.
So you have two choices:
-
Ensure all messages you publish throughout your system are well formed JSON. Not much we can do to help with that as you control the messages.
-
Add some logic in this flow to filter out the messages that are not valid JSON and handle them differently - which is what @hotNipi was pointing towards.
The choice is yours.
1 Like
Agreed.
But how these incorrect messages are getting in the system (Correction: That MQTT channel/topic) is the sticky point.
HotiNipi may have solved the problem. As I can't control when these errors occur, I shall have to apply it and wait a while and see if the problem goes away.
Yes, I control the messages. But as I just mentioned I am seeing erroneous messages which should not be in the BirtCert' or DeathCert' channel.
I shall have to look into how that is happening.
Show us what one of these messages looks like.
That isn't possible just now - though I have posted previous ones - as the errors are not controllable.
I've put the flow back to the old layout before HotNipi offered his solution.
I'll leave it like that for a few days and see if any thing happens.
If I get an errors, I'll post what I see.
{"topic":"EOM","payload":"'Tele Pi telemetry failure'","qos":0,"retain":true, .....
Such kind of messages should not be feed directly into json node as the payload is plain string.
If you are controlling the creation of those messages, you should format them to be proper JSON. If not, you have option of error caching or you should check incoming message payload type before feeding them into json node.
1 Like
Thanks.
(Obviously) that is a RPI.
It is from the MQTT node in the Birth and Death Certificates.
The GPS ones are from an Arduino.
There will be more Arduinos coming online at some stage.
So to check:
When MQTT connects to a server the Birth and Death Certificates are exchanged.
Would/should that be in JSON format?
Sounds right thing to do.
Mqtt does not require any particular format. It is up to you at the application layer to decide what format your payloads use.
If you want to be able to treat every payload as JSON then you need to make sure everything that gets published is in a JSON format.
If some things might not be JSON then you need to handle that. It may be you can identify the non JSON messages by virtue of what topic they are published on - in which case you can use a Switch node to direct those messages to a different flow.
Ultimately it comes down to what you want to do as, ultimately, you have to maintain it all.
The birth and death messages (they are messages not certificates) are under your control (or under the control of whoever configures the MQTT connection that is connecting or diconnecting), so probably you can put what you like there.
Nick/Colin.
Ok. Thanks. So their format is under my control.
(I'll look into that.)
Here are some messages/errors detected overnight.
{"topic":"EOM","payload":"GPS comms failure","qos":0,"retain":false,"_msgid":"c38fa4a4.e41c98","error":{"message":"Unexpected token G in JSON at position 0","source":{"id":"bf208502.79b7a","type":"json","count":1}},"ttl":0,"_queuetimestamp":1555704121328,"queueCount":3}
{"topic":"EOM","payload":"GPS comms failure","qos":0,"retain":false,"_msgid":"c38fa4a4.e41c98","error":{"message":"Unexpected token G in JSON at position 0","source":{"id":"c6927562.fa0738","type":"json","name":"tele-json","count":1}},"ttl":0,"_queuetimestamp":1555704121393,"queueCount":2}
This one is a bit weird, but.....
{"topic":"SOM","payload":"'Awaiting Pi Face'","qos":2,"retain":false,"_msgid":"812e07fb.b97408","error":{"message":"Unexpected token ' in JSON at position 0","source":{"id":"bf208502.79b7a","type":"json","count":1}},"ttl":0,"_queuetimestamp":1555711153860,"queueCount":1}
As is this one
{"topic":"SOM","payload":"'Awaiting Pi Face'","qos":2,"retain":false,"_msgid":"812e07fb.b97408","error":{"message":"Unexpected token ' in JSON at position 0","source":{"id":"c6927562.fa0738","type":"json","name":"tele-json","count":1}},"ttl":0,"_queuetimestamp":1555711153879,"queueCount":0}
Yes, it appears I shall have to go through and check a lot of my MQTT nodes.
Here's my take on what I am seeing:
The GPS (WiFi link) failed a couple of times. I guess I need to humanise the _queuetimestamp to know when it happened.
Twice overnight? Ok. Strange. But why aren't I seeing the Birth Certificates?
The GPS is connected and working.
Awaiting Pi Face That's a bit of a doozy. He is wired - though wired BC and DC are detected, but anyway.....
(Via edit)
Ok, the format of the packet out of the MQTT node.
It is set to Auto-detect string or buffer
So shouldn't that stop this kind of error happening anyway?
This is the code for the two nodes:
[{"id":"c9482822.f11c8","type":"mqtt in","z":"3913e6fb.8d0352","name":"","topic":"EOM","qos":"2","broker":"931f34a.34a47c8","x":210,"y":1250,"wires":[["5bfb5b93.bcc8b4"]]},{"id":"e48386f2.057dd","type":"mqtt in","z":"3913e6fb.8d0352","name":"","topic":"SOM","qos":"2","broker":"931f34a.34a47c8","x":210,"y":1300,"wires":[["5bfb5b93.bcc8b4"]]},{"id":"931f34a.34a47c8","type":"mqtt-broker","z":"","name":"MQTT host","broker":"192.168.0.99","port":"1883","clientid":"","usetls":false,"compatmode":false,"keepalive":"60","cleansession":true,"birthTopic":"","birthQos":"2","birthPayload":"","closeTopic":"","closePayload":"","willTopic":"","willQos":"0","willPayload":""}]
I'll stop now because I am not sure what else is needed.
All of those error messages you've shared are coming from a JSON node - you are passing text that isn't JSON to the JSON node and asking it to parse it.
The MQTT node has seen the payload is a string so it has converted the raw bytes to a string and passed it on. That has nothing to do with whether the string contains JSON or just plain text.
1 Like
Ok. My bad.
Here's where I am stuck though:
I'm just stating what has happened in the past.
That small part is to trap errors on this flow.
I have a switch to stop it going any further - like when I have this kind of pesky bug.
If normal, this is sent on to a bigger flow and it gets error reports from all machines.
I then want to split said things to filter by machine name.
So I have/had a MQTT receive node and then a switch node.
I was stuck (way back then) because I couldn't get the switch node to work correctly.
The data looked fine and I was trying to get my head around why it wasn't working.
I was trying to add/remove leading and trailing { } symbols.
You mentioned that I needed to put the message through a JSON node to make the switch node work/be able to switch/parse the data.
I seem to remember something about "As you are sending more than a simple msg.payload, when you use MQTT, it gets converted to a JSON package..... To get it back to the original format/type/style you need to send it through a JSON node."
Ok, I can't quote the exact reply. (Maybe I should search it before posting, but I am working in the "here and now".)
So, in light or recent information, I have deleted the original JSON node and left the bit that HotiNip mentioned.
I'll give that a spin around the block and see what happens.
Reading a bit deeper, I may not even need that. But: One step at a time.
Maybe off topic:
Ok, I did a bit of digging on the actual messages I posted.
The first error:
It was at: 2019-04-19T20:02:01.328Z
20:02:01Z + 10
30:02:01 (mod 24)
06:02 local.
Ok. I was asleep.
Reading MQTT stuff the Death and Birth Certificates are negotiated when the connection is set up and sent as needed.
I'm still (obviously) not quite getting that 100%.
But PiFace.... It is offline and no where to be found/seen.
Why would it's Certificate be broadcast?
(Yeah, ok: I put the wrong values in a node - probably)
How can I find which node is sending it?
There is only two devices connected.
TimePi and GPS.
I can be lucky.
Was poking around and saw this happen.
This is what I mean by saying I am seeing the wrong messages coming through in/on the wrong MQTT channels.
The GPS decided it wasn't talking. You see two of those messages.
Then shortly after you get an EOM (Which is what I call Death Certificates) come through with a payload of X.
X is sent on the IFF channel when a new WiFi connection is detected.
There is no reply. It is a simplex signal, published on the IFF channel.
Can someone help me with this?
(Is it helpful to find the Gremlin which seems to still be here.)
Those messages must be setup in MQTT config nodes that you have configured, includeing the "X". Have you used the same topic for all the birth and death messages you have configured? If not then then have a look at what is coming from the MQTT nodes to see where they are coming from. Also if you look at the mosquitto log (assuming you are using mosquitto) you will see them connecting/disconnecting.
The "Error detected ..." message is coming from a warn message that you have put in a function node. Click on the node id above the message and it will show you which node.
Thanks Colin.
(I've tried searching "node-red mqtt log" and all I get is how to log data with MQTT. Not where the log is. I will try /var/log/mosquitto.log)
Looked at /var/log/mosquitto/...... Yeah. I can't see anything there of interest.
Then went to /var/lib/mosquitto/mosquitto.db
Wow! Big file.
Did tail mosquitto.db Alas not much interesting there either.
ITMT, I'll keep looking.
Can I first off ask for help on that.
The 'X' is inserted to MQTT channel 'IFF'.
(
The lay out is a bit "all over the shop" but I have cut out the nodes of importance for this.
Please excuse the FAN nodes that will probably be missing for you.
They just give me a neater layout. I know I could just use blank/empty Function nodes, but....
)
There is also a GATE node just near the start which is because I am testing this part and have the alternative part switched/gated from that point.
Again: Yes, I know. Messy. It is still a bit of "work in progress."
Code:
[{"id":"814858d2.003e58","type":"inject","z":"3913e6fb.8d0352","name":"IFF","topic":"","payload":"X","payloadType":"str","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":2020,"y":520,"wires":[["729709b0.71b9a"]]},{"id":"c9482822.f11c8","type":"mqtt in","z":"3913e6fb.8d0352","name":"","topic":"EOM","qos":"2","broker":"931f34a.34a47c8","x":230,"y":1250,"wires":[["5bfb5b93.bcc8b4"]]},{"id":"e48386f2.057dd","type":"mqtt in","z":"3913e6fb.8d0352","name":"","topic":"SOM","qos":"2","broker":"931f34a.34a47c8","x":230,"y":1300,"wires":[["5bfb5b93.bcc8b4"]]},{"id":"5bfb5b93.bcc8b4","type":"fan","z":"3913e6fb.8d0352","x":390,"y":1280,"wires":[["f654881.7ae28f8","4993a3ef.94fa3c"]]},{"id":"f654881.7ae28f8","type":"gate","z":"3913e6fb.8d0352","name":"A","controlTopic":"control","defaultState":"closed","openCmd":"A","closeCmd":"B","toggleCmd":"toggle","defaultCmd":"default","persist":false,"x":520,"y":1280,"wires":[["3847532e.ad6a1c"]]},{"id":"3847532e.ad6a1c","type":"fan","z":"3913e6fb.8d0352","x":680,"y":1280,"wires":[["1f93afa4.be3ee"]]},{"id":"1f93afa4.be3ee","type":"fan","z":"3913e6fb.8d0352","x":890,"y":1280,"wires":[["337780bd.50079","b3f64ec4.a83f78","5d4762e7.eac9d4","48c770ea.d0ec2"]]},{"id":"5d4762e7.eac9d4","type":"fan","z":"3913e6fb.8d0352","x":910,"y":940,"wires":[["92e82e4e.b3213"]]},{"id":"92e82e4e.b3213","type":"change","z":"3913e6fb.8d0352","name":"IFF Request command generator","rules":[{"t":"set","p":"payload","pt":"msg","to":"X","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":1080,"y":850,"wires":[["1a3b6900.7b01c7"]]},{"id":"1a3b6900.7b01c7","type":"link out","z":"3913e6fb.8d0352","name":"","links":["41f22411.72db44"],"x":1275,"y":850,"wires":[]},{"id":"41f22411.72db44","type":"link in","z":"3913e6fb.8d0352","name":"IFF request RX","links":["1a3b6900.7b01c7"],"x":2055,"y":850,"wires":[["729709b0.71b9a"]]},{"id":"729709b0.71b9a","type":"fan","z":"3913e6fb.8d0352","x":2160,"y":570,"wires":[["9ba1382f.d521a","3ab3e172.0b52be"]]},{"id":"3ab3e172.0b52be","type":"mqtt out","z":"3913e6fb.8d0352","name":"Identify (IFF)","topic":"IFF","qos":"2","retain":"","broker":"931f34a.34a47c8","x":2340,"y":520,"wires":[]},{"id":"931f34a.34a47c8","type":"mqtt-broker","z":"","name":"MQTT host","broker":"192.168.0.99","port":"1883","clientid":"","usetls":false,"compatmode":false,"keepalive":"60","cleansession":true,"birthTopic":"","birthQos":"2","birthPayload":"","closeTopic":"","closePayload":"","willTopic":"","willQos":"0","willPayload":""}]
What node - i.e. node-red-contrib-xxxxx is the fan node?
What node - i.e. node-red-contrib-xxxxx is the gate node?
Without any other info, it looks like you put the letter X into msg.payloaf and then send it thru the fan node and on to MQTT.
If the fan node is not doing anyting to msg.payload then you are sending a string into the MQTT-out node.
