# Issue retrieving json from mqtt

**URL:** <https://discourse.nodered.org/t/issue-retrieving-json-from-mqtt/8680>\
**Category:** General\
**Created:** [6 March 2019 19:39 UTC](https://discourse.nodered.org/t/issue-retrieving-json-from-mqtt/8680 "2019-03-06T19:39:10Z")\
**Posts on this page:** 17\
**Page:** 1

<div class="post-metadata">

**Author:** ![Deca](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/deca/32/6515_2.png) [@Deca](https://discourse.nodered.org/u/Deca)\
**Post date:** [6 March 2019 19:39 UTC](https://discourse.nodered.org/t/issue-retrieving-json-from-mqtt/8680/1 "2019-03-06T19:39:11Z")

</div>

Hi there,  
I'm starting to use nodered for the first time and I'm trying to get data from a mqtt device for further processing (output on dashboard, save on mongodb and so on)  
I built the flow with dummy json object and worked fine but now I struggle to data from the real device (it's a Bosch XDK board)

The debug from the mqtt module show raw and string data: in the string I see the json object but in the raw there is a list of hex numbers and the json node throw me the error "Ignored non-object payload"

How I can get the payload string and not the raw data?

![Cattura](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/2X/9/94764140b930a10fdc34b208e503b54f98b462a7.jpeg)

---

<div class="post-metadata">

**Author:** ![Colin](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/colin/32/17040_2.png) [@Colin](https://discourse.nodered.org/u/Colin)\
**Post date:** [6 March 2019 20:53 UTC](https://discourse.nodered.org/t/issue-retrieving-json-from-mqtt/8680/2 "2019-03-06T20:53:16Z")

</div>

Is that debug straight out of the mqtt node? If so then you are not getting json but a buffer. I didn't know that mqtt could give a buffer. It looks as if it might be a string as a buffer, feed it into a function node containing

```auto
msg.payload = msg.payload.toString('utf8');
return msg;

```

and then to a debug node and see what you get.

---

<div class="post-metadata">

**Author:** ![dceejay](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/dceejay/32/38_2.png) [@dceejay](https://discourse.nodered.org/u/dceejay)\
**Post date:** [6 March 2019 22:03 UTC](https://discourse.nodered.org/t/issue-retrieving-json-from-mqtt/8680/3 "2019-03-06T22:03:40Z")

</div>

Currently (Node-RED v0.19) the MQTT node outputs either a buffer or string... it looks to see if it's a utf8 string and if so sends a string... if not it sends a buffer - so I guess this is not a UTF8 string.

Form v0.20 there is an extra option to fix the output to be buffer, text, or parsed json (as well as the existing auto mode).

---

<div class="post-metadata">

**Author:** ![Colin](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/colin/32/17040_2.png) [@Colin](https://discourse.nodered.org/u/Colin)\
**Post date:** [7 March 2019 07:24 UTC](https://discourse.nodered.org/t/issue-retrieving-json-from-mqtt/8680/4 "2019-03-07T07:24:48Z")

</div>

How does it determine whether it is a string, the start looks ok.

---

<div class="post-metadata">

**Author:** ![Deca](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/deca/32/6515_2.png) [@Deca](https://discourse.nodered.org/u/Deca)\
**Post date:** [7 March 2019 08:08 UTC](https://discourse.nodered.org/t/issue-retrieving-json-from-mqtt/8680/5 "2019-03-07T08:08:48Z")

</div>

Thanks for the reply @Colin and @dceejay ,

I've tried to add a node func and convert the payload to a string, now the output is a string `msg.payload : string[800]` with an apparently valid JSON, but I get an error from the json node `"Unexpected token in JSON at position753"`  
I thought that maybe I don't need anymore the JSON but still I'm unable to get any variable out with a debug node

![Cattura](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/2X/c/c11686a8e7400389aa0aca29689ac2d7aab61d66.jpeg)

Then I've installed the 0.20 beta, the mqtt node indeed offer differents output (buffer, string, parsed json object, base64 encoded string)

1. Parsed JSON: node mqtt return a `msg : string[19] "Invalid JSON string"`
2. Buffer: the default output, I get the hex code.
3. String: get again the JSON in string format but then I've the same issue had with Colin solution ( json node throw a: `"Unexpected token in JSON at position753"` )

I'm a little at loss of idea because I thought that the JSON node has able to parsing a json string to msg.payload...am I missing something?

---

<div class="post-metadata">

**Author:** ![knolleary](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/knolleary/32/3_2.png) [@knolleary](https://discourse.nodered.org/u/knolleary)\
**Post date:** [7 March 2019 08:22 UTC](https://discourse.nodered.org/t/issue-retrieving-json-from-mqtt/8680/6 "2019-03-07T08:22:13Z")

</div>

> [@Deca](#):
>
> the output is a string `msg.payload : string[800]` with an apparently valid JSON, but I get an error from the json node `"Unexpected token in JSON at position753"`

I think they key word here is 'apparently valid'. If the JSON node is complaining, then it is not valid JSON. Have a look at what byte is at position 753 in the buffer you have. My guess is there's some non-utf8 byte there which is why the MQTT node is emitting it as a Buffer and not a String in the first place.

---

<div class="post-metadata">

**Author:** ![Deca](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/deca/32/6515_2.png) [@Deca](https://discourse.nodered.org/u/Deca)\
**Post date:** [7 March 2019 09:15 UTC](https://discourse.nodered.org/t/issue-retrieving-json-from-mqtt/8680/7 "2019-03-07T09:15:13Z")

</div>

This pasted below is the string I get from the mqtt node.  
I've tried to put into a validator and the json seems fine (no errors)

Also If I take that json string and manually inject to a new JSON node I get the JSON object without issue...so the problem is still around the mqtt node I think

```auto
{
   "xdk2mam":[
      {
         "sensorType":"Environmental",
         "data":[
            {
               "name":"Pressure",
               "value":"100091"
            },
            {
               "name":"Temperature",
               "value":"27290"
            },
            {
               "name":"Humidity",
               "value":"45"
            }
         ]
      },
      {
         "sensorType":"Accelerometer",
         "data":[
            {
               "name":"x",
               "value":"30"
            },
            {
               "name":"y",
               "value":"-131"
            },
            {
               "name":"z",
               "value":"992"
            }
         ]
      },
      {
         "sensorType":"Gyroscope",
         "data":[
            {
               "name":"x",
               "value":"0"
            },
            {
               "name":"y",
               "value":"3295"
            },
            {
               "name":"z",
               "value":"4394"
            }
         ]
      },
      {
         "sensorType":"Inertial",
         "data":[
            {
               "name":"x",
               "value":"-305"
            },
            {
               "name":"y",
               "value":"-854"
            },
            {
               "name":"z",
               "value":"-122"
            }
         ]
      },
      {
         "sensorType":"Light",
         "data":[
            {
               "name":"milliLux",
               "value":"12150"
            }
         ]
      },
      {
         "sensorType":"Magnetometer",
         "data":[
            {
               "name":"x",
               "value":"27"
            },
            {
               "name":"y",
               "value":"13"
            },
            {
               "name":"z",
               "value":"7"
            }
         ]
      }
   ],
   "device":"248CE70556960C54",
   "timestamp":"1551949151"
}

```

---

<div class="post-metadata">

**Author:** ![knolleary](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/knolleary/32/3_2.png) [@knolleary](https://discourse.nodered.org/u/knolleary)\
**Post date:** [7 March 2019 09:19 UTC](https://discourse.nodered.org/t/issue-retrieving-json-from-mqtt/8680/8 "2019-03-07T09:19:42Z")

</div>

If you've just copy and pasted the text then sure, it's valid JSON. But that isn't what I suggested - because by copying the text you will have left behind any stray non-utf8 bytes behind.

What byte is at position 753 (or whatever position the error reports for a given message) of the buffer you receive?

---

<div class="post-metadata">

**Author:** ![Deca](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/deca/32/6515_2.png) [@Deca](https://discourse.nodered.org/u/Deca)\
**Post date:** [7 March 2019 09:23 UTC](https://discourse.nodered.org/t/issue-retrieving-json-from-mqtt/8680/9 "2019-03-07T09:23:39Z")

</div>

The byte position 763 (it change since the content is dinamic) is right the last character

 ![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/2X/8/8fb80a124422557d3954f680b15a3ec16d214bfe.png)

---

<div class="post-metadata">

**Author:** ![knolleary](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/knolleary/32/3_2.png) [@knolleary](https://discourse.nodered.org/u/knolleary)\
**Post date:** [7 March 2019 09:25 UTC](https://discourse.nodered.org/t/issue-retrieving-json-from-mqtt/8680/10 "2019-03-07T09:25:13Z")

</div>

In your original post you were getting a Buffer from the MQTT node. Look at the raw buffer you get from the MQTT node.

---

<div class="post-metadata">

**Author:** ![Deca](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/deca/32/6515_2.png) [@Deca](https://discourse.nodered.org/u/Deca)\
**Post date:** [7 March 2019 09:26 UTC](https://discourse.nodered.org/t/issue-retrieving-json-from-mqtt/8680/11 "2019-03-07T09:26:35Z")

</div>

The mqtt node right now is configured as string output, so I guess the error is regarding the string...should I still try to set back to buffer?

---

<div class="post-metadata">

**Author:** ![knolleary](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/knolleary/32/3_2.png) [@knolleary](https://discourse.nodered.org/u/knolleary)\
**Post date:** [7 March 2019 09:29 UTC](https://discourse.nodered.org/t/issue-retrieving-json-from-mqtt/8680/12 "2019-03-07T09:29:05Z")

</div>

If you want to know what the bad byte is in the buffer then yes you should.

---

<div class="post-metadata">

**Author:** ![Deca](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/deca/32/6515_2.png) [@Deca](https://discourse.nodered.org/u/Deca)\
**Post date:** [7 March 2019 09:43 UTC](https://discourse.nodered.org/t/issue-retrieving-json-from-mqtt/8680/13 "2019-03-07T09:43:00Z")

</div>

Uhm I'm not completely sure about that, I mean with the mqtt node in buffer output I don't know anymore which exact position the JSON throw the error

Anyway with the output buffer I see the raw data in the debug tab but it stop to [500 … 800] (can't navigate into it)  
I tried to dumping the mqtt buffer in a txt file but return me just the json string

![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/2X/4/4d4a927a4227c7cd5e097a49cb1268de2b9b95b1.png)

---

<div class="post-metadata">

**Author:** ![knolleary](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/knolleary/32/3_2.png) [@knolleary](https://discourse.nodered.org/u/knolleary)\
**Post date:** [7 March 2019 09:58 UTC](https://discourse.nodered.org/t/issue-retrieving-json-from-mqtt/8680/14 "2019-03-07T09:58:28Z")

</div>

Feed the Buffer from the MQTT node into a Function node with:

```auto
var newBuffer = Buffer.allocUnsafe(5);
msg.payload.copy(newBuffer,0,msg.payload.length-5,msg.payload.length)
msg.payload = newBuffer;
return msg;

```

and pass the result to a Debug node. That will give you the last 5 bytes. I bet you'll have a `0` value in there.

---

<div class="post-metadata">

**Author:** ![Deca](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/deca/32/6515_2.png) [@Deca](https://discourse.nodered.org/u/Deca)\
**Post date:** [7 March 2019 11:44 UTC](https://discourse.nodered.org/t/issue-retrieving-json-from-mqtt/8680/15 "2019-03-07T11:44:13Z")

</div>

Thanks for the support @knolleary 🙂 it's the first time I'm working with binary data and this is driving me crazy  
Yes it output all 0

![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/2X/e/ea4871e27c259944c3a9cad2765ef384eafd0bcb.png)

---

<div class="post-metadata">

**Author:** ![knolleary](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/knolleary/32/3_2.png) [@knolleary](https://discourse.nodered.org/u/knolleary)\
**Post date:** [7 March 2019 12:46 UTC](https://discourse.nodered.org/t/issue-retrieving-json-from-mqtt/8680/16 "2019-03-07T12:46:36Z")

</div>

Ok - so you have quite a few `0` at the end there. Presumably whatever device is sending the data isn't quite getting its length calculation right and is sending more than it should.

The following Function will create a Buffer containing your data up to (but not including!) the first `0` it finds and then convert it to a String, so you can then pass it to the JSON node and, hopefully, get your actual data.

```auto
var zeroIndex = msg.payload.indexOf(0)
var newBuffer = Buffer.allocUnsafe(zeroIndex);
msg.payload.copy(newBuffer,0,0,zeroIndex)
msg.payload = newBuffer.toString();
return msg;

```

---

<div class="post-metadata">

**Author:** ![Deca](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/deca/32/6515_2.png) [@Deca](https://discourse.nodered.org/u/Deca)\
**Post date:** [7 March 2019 19:59 UTC](https://discourse.nodered.org/t/issue-retrieving-json-from-mqtt/8680/17 "2019-03-07T19:59:10Z")

</div>

Just tried your js snippet and it works! Without the zeros the JSON node is running fine.  
To recap :MQTT node (output a buffer) -\> func node(rebuild buffer without zeros)-\> JSON node(output convert to json obj)

I've also did some further search and found a stackoverflow question about a similar issue

> <https://stackoverflow.com/questions/22206426/parsing-string-to-json-using-node-gives-unexpected-token-validator-says-ok>

As alternative to manipulate the buffer is also possible to remove the zero from the string with regex (just tested)  
`msg.payload = msg.payload.trim().replace(/\0/g, '');`

Thanks for the patience and the help @knolleary really appreciate it, and I've also learned something new today 😃