Issue retrieving json from mqtt

#1

Hi there,
I'm starting to use nodered for the first time and I'm trying to get data from a mqtt device for further processing (output on dashboard, save on mongodb and so on)
I built the flow with dummy json object and worked fine but now I struggle to data from the real device (it's a Bosch XDK board)

The debug from the mqtt module show raw and string data: in the string I see the json object but in the raw there is a list of hex numbers and the json node throw me the error "Ignored non-object payload"

How I can get the payload string and not the raw data?

#2

Is that debug straight out of the mqtt node? If so then you are not getting json but a buffer. I didn't know that mqtt could give a buffer. It looks as if it might be a string as a buffer, feed it into a function node containing

msg.payload = msg.payload.toString('utf8');
return msg;

and then to a debug node and see what you get.

#3

Currently (Node-RED v0.19) the MQTT node outputs either a buffer or string... it looks to see if it's a utf8 string and if so sends a string... if not it sends a buffer - so I guess this is not a UTF8 string.

Form v0.20 there is an extra option to fix the output to be buffer, text, or parsed json (as well as the existing auto mode).

#4

How does it determine whether it is a string, the start looks ok.

#5

Thanks for the reply @Colin and @dceejay ,

I've tried to add a node func and convert the payload to a string, now the output is a string msg.payload : string[800] with an apparently valid JSON, but I get an error from the json node "Unexpected token in JSON at position753"
I thought that maybe I don't need anymore the JSON but still I'm unable to get any variable out with a debug node

Then I've installed the 0.20 beta, the mqtt node indeed offer differents output (buffer, string, parsed json object, base64 encoded string)

  1. Parsed JSON: node mqtt return a msg : string[19] "Invalid JSON string"
  2. Buffer: the default output, I get the hex code.
  3. String: get again the JSON in string format but then I've the same issue had with Colin solution ( json node throw a: "Unexpected token in JSON at position753" )

I'm a little at loss of idea because I thought that the JSON node has able to parsing a json string to msg.payload...am I missing something?

#6

I think they key word here is 'apparently valid'. If the JSON node is complaining, then it is not valid JSON. Have a look at what byte is at position 753 in the buffer you have. My guess is there's some non-utf8 byte there which is why the MQTT node is emitting it as a Buffer and not a String in the first place.

#7

This pasted below is the string I get from the mqtt node.
I've tried to put into a validator and the json seems fine (no errors)

Also If I take that json string and manually inject to a new JSON node I get the JSON object without issue...so the problem is still around the mqtt node I think

{
   "xdk2mam":[
      {
         "sensorType":"Environmental",
         "data":[
            {
               "name":"Pressure",
               "value":"100091"
            },
            {
               "name":"Temperature",
               "value":"27290"
            },
            {
               "name":"Humidity",
               "value":"45"
            }
         ]
      },
      {
         "sensorType":"Accelerometer",
         "data":[
            {
               "name":"x",
               "value":"30"
            },
            {
               "name":"y",
               "value":"-131"
            },
            {
               "name":"z",
               "value":"992"
            }
         ]
      },
      {
         "sensorType":"Gyroscope",
         "data":[
            {
               "name":"x",
               "value":"0"
            },
            {
               "name":"y",
               "value":"3295"
            },
            {
               "name":"z",
               "value":"4394"
            }
         ]
      },
      {
         "sensorType":"Inertial",
         "data":[
            {
               "name":"x",
               "value":"-305"
            },
            {
               "name":"y",
               "value":"-854"
            },
            {
               "name":"z",
               "value":"-122"
            }
         ]
      },
      {
         "sensorType":"Light",
         "data":[
            {
               "name":"milliLux",
               "value":"12150"
            }
         ]
      },
      {
         "sensorType":"Magnetometer",
         "data":[
            {
               "name":"x",
               "value":"27"
            },
            {
               "name":"y",
               "value":"13"
            },
            {
               "name":"z",
               "value":"7"
            }
         ]
      }
   ],
   "device":"248CE70556960C54",
   "timestamp":"1551949151"
}
#8

If you've just copy and pasted the text then sure, it's valid JSON. But that isn't what I suggested - because by copying the text you will have left behind any stray non-utf8 bytes behind.

What byte is at position 753 (or whatever position the error reports for a given message) of the buffer you receive?

#9

The byte position 763 (it change since the content is dinamic) is right the last character

#10

In your original post you were getting a Buffer from the MQTT node. Look at the raw buffer you get from the MQTT node.

#11

The mqtt node right now is configured as string output, so I guess the error is regarding the string...should I still try to set back to buffer?

#12

If you want to know what the bad byte is in the buffer then yes you should.

#13

Uhm I'm not completely sure about that, I mean with the mqtt node in buffer output I don't know anymore which exact position the JSON throw the error

Anyway with the output buffer I see the raw data in the debug tab but it stop to [500 … 800] (can't navigate into it)
I tried to dumping the mqtt buffer in a txt file but return me just the json string

#14

Feed the Buffer from the MQTT node into a Function node with:

var newBuffer = Buffer.allocUnsafe(5);
msg.payload.copy(newBuffer,0,msg.payload.length-5,msg.payload.length)
msg.payload = newBuffer;
return msg;

and pass the result to a Debug node. That will give you the last 5 bytes. I bet you'll have a 0 value in there.

#15

Thanks for the support @knolleary :slight_smile: it's the first time I'm working with binary data and this is driving me crazy
Yes it output all 0

image

#16

Ok - so you have quite a few 0 at the end there. Presumably whatever device is sending the data isn't quite getting its length calculation right and is sending more than it should.

The following Function will create a Buffer containing your data up to (but not including!) the first 0 it finds and then convert it to a String, so you can then pass it to the JSON node and, hopefully, get your actual data.

var zeroIndex = msg.payload.indexOf(0)
var newBuffer = Buffer.allocUnsafe(zeroIndex);
msg.payload.copy(newBuffer,0,0,zeroIndex)
msg.payload = newBuffer.toString();
return msg;
#17

Just tried your js snippet and it works! Without the zeros the JSON node is running fine.
To recap :MQTT node (output a buffer) -> func node(rebuild buffer without zeros)-> JSON node(output convert to json obj)

I've also did some further search and found a stackoverflow question about a similar issue

As alternative to manipulate the buffer is also possible to remove the zero from the string with regex (just tested)
msg.payload = msg.payload.trim().replace(/\0/g, '');

Thanks for the patience and the help @knolleary really appreciate it, and I've also learned something new today :smiley:

3 Likes