Issue retrieving json from mqtt

General
1

Hi there,
I'm starting to use nodered for the first time and I'm trying to get data from a mqtt device for further processing (output on dashboard, save on mongodb and so on)
I built the flow with dummy json object and worked fine but now I struggle to data from the real device (it's a Bosch XDK board)

The debug from the mqtt module show raw and string data: in the string I see the json object but in the raw there is a list of hex numbers and the json node throw me the error "Ignored non-object payload"

How I can get the payload string and not the raw data?

Cattura

2

Is that debug straight out of the mqtt node? If so then you are not getting json but a buffer. I didn't know that mqtt could give a buffer. It looks as if it might be a string as a buffer, feed it into a function node containing

msg.payload = msg.payload.toString('utf8');
return msg;

and then to a debug node and see what you get.

3

Currently (Node-RED v0.19) the MQTT node outputs either a buffer or string... it looks to see if it's a utf8 string and if so sends a string... if not it sends a buffer - so I guess this is not a UTF8 string.

Form v0.20 there is an extra option to fix the output to be buffer, text, or parsed json (as well as the existing auto mode).

4

How does it determine whether it is a string, the start looks ok.

5

Thanks for the reply @Colin and @dceejay ,

I've tried to add a node func and convert the payload to a string, now the output is a string msg.payload : string[800] with an apparently valid JSON, but I get an error from the json node "Unexpected token in JSON at position753"
I thought that maybe I don't need anymore the JSON but still I'm unable to get any variable out with a debug node

Cattura

Then I've installed the 0.20 beta, the mqtt node indeed offer differents output (buffer, string, parsed json object, base64 encoded string)

  1. Parsed JSON: node mqtt return a msg : string[19] "Invalid JSON string"
  2. Buffer: the default output, I get the hex code.
  3. String: get again the JSON in string format but then I've the same issue had with Colin solution ( json node throw a: "Unexpected token in JSON at position753" )

I'm a little at loss of idea because I thought that the JSON node has able to parsing a json string to msg.payload...am I missing something?

6

I think they key word here is 'apparently valid'. If the JSON node is complaining, then it is not valid JSON. Have a look at what byte is at position 753 in the buffer you have. My guess is there's some non-utf8 byte there which is why the MQTT node is emitting it as a Buffer and not a String in the first place.

7

This pasted below is the string I get from the mqtt node.
I've tried to put into a validator and the json seems fine (no errors)

Also If I take that json string and manually inject to a new JSON node I get the JSON object without issue...so the problem is still around the mqtt node I think

{
   "xdk2mam":[
      {
         "sensorType":"Environmental",
         "data":[
            {
               "name":"Pressure",
               "value":"100091"
            },
            {
               "name":"Temperature",
               "value":"27290"
            },
            {
               "name":"Humidity",
               "value":"45"
            }
         ]
      },
      {
         "sensorType":"Accelerometer",
         "data":[
            {
               "name":"x",
               "value":"30"
            },
            {
               "name":"y",
               "value":"-131"
            },
            {
               "name":"z",
               "value":"992"
            }
         ]
      },
      {
         "sensorType":"Gyroscope",
         "data":[
            {
               "name":"x",
               "value":"0"
            },
            {
               "name":"y",
               "value":"3295"
            },
            {
               "name":"z",
               "value":"4394"
            }
         ]
      },
      {
         "sensorType":"Inertial",
         "data":[
            {
               "name":"x",
               "value":"-305"
            },
            {
               "name":"y",
               "value":"-854"
            },
            {
               "name":"z",
               "value":"-122"
            }
         ]
      },
      {
         "sensorType":"Light",
         "data":[
            {
               "name":"milliLux",
               "value":"12150"
            }
         ]
      },
      {
         "sensorType":"Magnetometer",
         "data":[
            {
               "name":"x",
               "value":"27"
            },
            {
               "name":"y",
               "value":"13"
            },
            {
               "name":"z",
               "value":"7"
            }
         ]
      }
   ],
   "device":"248CE70556960C54",
   "timestamp":"1551949151"
}
8

If you've just copy and pasted the text then sure, it's valid JSON. But that isn't what I suggested - because by copying the text you will have left behind any stray non-utf8 bytes behind.

What byte is at position 753 (or whatever position the error reports for a given message) of the buffer you receive?

9

The byte position 763 (it change since the content is dinamic) is right the last character

image
image.png1304Ă—1244 203 KB

10

In your original post you were getting a Buffer from the MQTT node. Look at the raw buffer you get from the MQTT node.

11

The mqtt node right now is configured as string output, so I guess the error is regarding the string...should I still try to set back to buffer?

12

If you want to know what the bad byte is in the buffer then yes you should.

13

Uhm I'm not completely sure about that, I mean with the mqtt node in buffer output I don't know anymore which exact position the JSON throw the error

Anyway with the output buffer I see the raw data in the debug tab but it stop to [500 … 800] (can't navigate into it)
I tried to dumping the mqtt buffer in a txt file but return me just the json string

image

14

Feed the Buffer from the MQTT node into a Function node with:

var newBuffer = Buffer.allocUnsafe(5);
msg.payload.copy(newBuffer,0,msg.payload.length-5,msg.payload.length)
msg.payload = newBuffer;
return msg;

and pass the result to a Debug node. That will give you the last 5 bytes. I bet you'll have a 0 value in there.

1 Like
15

Thanks for the support @knolleary :slight_smile: it's the first time I'm working with binary data and this is driving me crazy
Yes it output all 0

image

16

Ok - so you have quite a few 0 at the end there. Presumably whatever device is sending the data isn't quite getting its length calculation right and is sending more than it should.

The following Function will create a Buffer containing your data up to (but not including!) the first 0 it finds and then convert it to a String, so you can then pass it to the JSON node and, hopefully, get your actual data.

var zeroIndex = msg.payload.indexOf(0)
var newBuffer = Buffer.allocUnsafe(zeroIndex);
msg.payload.copy(newBuffer,0,0,zeroIndex)
msg.payload = newBuffer.toString();
return msg;
1 Like
17

Just tried your js snippet and it works! Without the zeros the JSON node is running fine.
To recap :MQTT node (output a buffer) -> func node(rebuild buffer without zeros)-> JSON node(output convert to json obj)

I've also did some further search and found a stackoverflow question about a similar issue

As alternative to manipulate the buffer is also possible to remove the zero from the string with regex (just tested)
msg.payload = msg.payload.trim().replace(/\0/g, '');

Thanks for the patience and the help @knolleary really appreciate it, and I've also learned something new today :smiley:

3 Likes