# Flow to decode binary data and a walk-through on how to install the npm module

**URL:** <https://discourse.nodered.org/t/flow-to-decode-binary-data-and-a-walk-through-on-how-to-install-the-npm-module/5174>\
**Category:** General\
**Created:** [24 November 2018 23:20 UTC](https://discourse.nodered.org/t/flow-to-decode-binary-data-and-a-walk-through-on-how-to-install-the-npm-module/5174 "2018-11-24T23:20:28Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![Andrei](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/andrei/32/10446_2.png) [@Andrei](https://discourse.nodered.org/u/Andrei)\
**Post date:** [24 November 2018 23:20 UTC](https://discourse.nodered.org/t/flow-to-decode-binary-data-and-a-walk-through-on-how-to-install-the-npm-module/5174/1 "2018-11-24T23:20:28Z")

</div>

I just wanted to share a simple way to parse and decode a binary stream of data using the npm module [binary-parser-encoder](https://www.npmjs.com/package/binary-parser-encoder). This is an alternative to the [node-red-contrib-binary node](https://flows.nodered.org/node/node-red-contrib-binary)

1- Install the module , in my case in Windows by typing:

```auto
npm install binary-parser

```

2- Amend settings.js config file by adding the `binary_parser` line as shown below

```auto
 functionGlobalContext: {
        jsonata:require('jsonata'),
        moment:require('moment'),
        chance:require('chance'),
        binary_parser:require('binary-parser').Parser,

```

Note that the module name uses a, `hyphen` in the name whereas the property name in `functionGlobalContext` will use `underscore`

3- Start (or restart) Node-RED. Hopefully there will be no errors, otherwise the initialization will not proceed( in which case you need to fallback and analyse).

**Using the module.**

As usual in Node-RED you need to use `global.get` to require the module. See below how the function node will looks like. This will be the core of the flow. I am using it here to decode a proprietary packet format for a Lidar scanner (but using random test data).

```auto
var Packet = global.get('binary_parser');
var buf = msg.payload;

var typea = new Packet()
  .endianess("little")
  .uint16("magic")
  .uint16("packet_type")
  .uint32("packet_size")
  .uint16("header_size")
  .uint16("scan_number")
  .uint16("packet_number")
  .double("timestamp_raw")
  .double("timestamp_syncw")
  .uint32("status_flags")
  .uint32("scan_frequencye")
  .uint16("num_points_scan")
  .uint16("num_points_packet")
  .uint16("first_index")
  .uint32("first_angle")
  .uint32("angular_increment")
  .uint32("iq_input")
  .uint32("iq_overload")
  .double("iq_timestamp_raw")
  .double("iq_timestamp_sync")
  .uint8("header_padding");
  
msg.payload = typea.parse(buf);
return msg;

```

Flow:

```auto
[{"id":"16dd9094.027e5f","type":"tab","label":"Flow 1","disabled":false,"info":""},{"id":"c899e7c.87f7618","type":"inject","z":"16dd9094.027e5f","name":"Go","topic":"","payload":"","payloadType":"str","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":150,"y":180,"wires":[["1ccf13aa.f5370c"]]},{"id":"1ccf13aa.f5370c","type":"function","z":"16dd9094.027e5f","name":"Dataset Buffer","func":"msg.payload = Buffer.from([0xa2, 0x5c, 0x01, 0x80, 0x02, 0x10, 0x02, 0x10, 0x02, 0x10, 0x10, 0x02, 0x10, 0x03, 0x80, 0x02, 0x10, 0x02, 0x10, 0x02, 0x15, 0x95, 0x02, 0x10, 0x02, 0x10, 0x02, 0x10, 0x10, 0x02, 0x10, 0x03, 0x01, 0x80, 0x02, 0x10, 0x02, 0x10, 0x02, 0x10, 0x10, 0x02, 0x10, 0x03, 0x80, 0x02, 0x10, 0x02, 0x10, 0x02, 0x15, 0x95, 0x02, 0x10, 0x02, 0x10, 0x02, 0x10, 0x10, 0x02, 0x10, 0x03, 0x03, 0x80, 0x02, 0x10, 0x02, 0x10, 0x02, 0x15, 0x95, 0x02, 0x10, 0x02, 0x10, 0x02, 0x10, 0x10, 0x02, 0x10, 0x03, 0x01, 0x80, 0x02, 0x10, 0x02, 0x10, 0x02, 0x10, 0x10, 0x02, 0x10, 0x03, 0x80, 0x02, 0x10, 0x02, 0x10, 0x02, 0x15, 0x95, 0x02, 0x10, 0x02, 0x10, 0x02, 0x10, 0x10, 0x02, 0x10, 0x03]);\nreturn msg;","outputs":1,"noerr":0,"x":320,"y":180,"wires":[["7df2c612.8102a8"]]},{"id":"7df2c612.8102a8","type":"function","z":"16dd9094.027e5f","name":"binary_parser","func":"var Packet = global.get('binary_parser');\nvar buf = msg.payload;\n\nvar typea = new Packet()\n .endianess(\"little\")\n .uint16(\"magic\")\n .uint16(\"packet_type\")\n .uint32(\"packet_size\")\n .uint16(\"header_size\")\n .uint16(\"scan_number\")\n .uint16(\"packet_number\")\n .double(\"timestamp_raw\")\n .double(\"timestamp_syncw\")\n .uint32(\"status_flags\")\n .uint32(\"scan_frequencye\")\n .uint16(\"num_points_scan\")\n .uint16(\"num_points_packet\")\n .uint16(\"first_index\")\n .uint32(\"first_angle\")\n .uint32(\"angular_increment\")\n .uint32(\"iq_input\")\n .uint32(\"iq_overload\")\n .double(\"iq_timestamp_raw\")\n .double(\"iq_timestamp_sync\")\n .uint8(\"header_padding\");\n \nmsg.payload = typea.parse(buf);\nreturn msg;\n ","outputs":1,"noerr":0,"x":520,"y":180,"wires":[["92b3fe6d.84a35"]]},{"id":"92b3fe6d.84a35","type":"debug","z":"16dd9094.027e5f","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","x":690,"y":180,"wires":[]}]

```

As the binary data structure is fixed the flow will be quite simple.

 ![r-01](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/2X/0/0a4f6369b0d4fe9cdaed5bccac43e6804b660544.png)

 ![r-02](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/2X/0/0efc2c6d7a19bc1ba5d8ccf07f74ea50e8026fec.png)

---

<div class="post-metadata">

**Author:** ![Colin](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/colin/32/17040_2.png) [@Colin](https://discourse.nodered.org/u/Colin)\
**Post date:** [25 November 2018 07:10 UTC](https://discourse.nodered.org/t/flow-to-decode-binary-data-and-a-walk-through-on-how-to-install-the-npm-module/5174/2 "2018-11-25T07:10:34Z")

</div>

What is the advantage over using the contrib node?

---

<div class="post-metadata">

**Author:** ![Andrei](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/andrei/32/10446_2.png) [@Andrei](https://discourse.nodered.org/u/Andrei)\
**Post date:** [25 November 2018 13:22 UTC](https://discourse.nodered.org/t/flow-to-decode-binary-data-and-a-walk-through-on-how-to-install-the-npm-module/5174/3 "2018-11-25T13:22:42Z")

</div>

HI @Colin,

I found that the size of the Pattern field provided in the config dialog of the contrib-node-binary is a major constraint. The expressions required to parse a binary data are normally quite large and it is cumbersome to be limited to work on one single line. I prefer to work in the larger room in the function node.

Also the binary-parser-encoder module seems to be updated more frequently and have more contributors to the code. Finally i liked more the chainable syntax. The syntax used by the node-red-contrib-binary is somehow obscure (or more likely I did not spend sufficient time working on it to get acquainted).

![r-01](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/2X/8/86d36acb8a6173129aa664b5edd53996ab630b1c.png)

---

<div class="post-metadata">

**Author:** ![dceejay](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/dceejay/32/38_2.png) [@dceejay](https://discourse.nodered.org/u/dceejay)\
**Post date:** [25 November 2018 14:29 UTC](https://discourse.nodered.org/t/flow-to-decode-binary-data-and-a-walk-through-on-how-to-install-the-npm-module/5174/4 "2018-11-25T14:29:21Z")

</div>

always happy to look at a Pull Request that changes that field into a larger edit field if you like...

---

<div class="post-metadata">

**Author:** ![BartButenaers](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/bartbutenaers/32/10476_2.png) [@BartButenaers](https://discourse.nodered.org/u/BartButenaers)\
**Post date:** [25 November 2018 15:35 UTC](https://discourse.nodered.org/t/flow-to-decode-binary-data-and-a-walk-through-on-how-to-install-the-npm-module/5174/5 "2018-11-25T15:35:54Z")

</div>

Hi Andrei,

Thanks for the info! I haven't used the node-red-contrib-binary before.  
Perhaps my questions are a bit off-topic, but now you have triggered my brain...

When users get an image from their IP camera, they **have to** set the output of the httprequest node to '_buffer_' instead of '_utf8 string_'. Otherwise the image becomes corrupt. But do you perhaps have an understanding of what is really going wrong in that case? In other words, what means 'corrupt'.

And a second question. Suppose they would have an image as utf8 string (which will have bad influence on performance anyway, due to the string conversion), but can we convert the string back to a valid image buffer afterwards? E.g. using the node-red-contrib-binary. Or is the image data corrupt until ethernity?

Thanks !  
Bart

---

<div class="post-metadata">

**Author:** ![dceejay](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/dceejay/32/38_2.png) [@dceejay](https://discourse.nodered.org/u/dceejay)\
**Post date:** [25 November 2018 16:01 UTC](https://discourse.nodered.org/t/flow-to-decode-binary-data-and-a-walk-through-on-how-to-install-the-npm-module/5174/6 "2018-11-25T16:01:20Z")

</div>

corrupt mean that it converted what was a binary payload into a utf-8 string - which expands many higher numbered characters into two bytes. It may be possible to un-mangle it but I wouldn't rely on it. Rather than try to guess if it is text or binary - it's safer for the http-request node to be explicit.

The contrib-binary node is more for dissecting packets - eg unpacking C struct like objects that many low level protocols used in the days before we didn't care about bandwidth and the verbosity of json, xml, etc 🙂

The binary parser module may indeed be a better choice as it does at least seem to be maintained - even if the syntax is slightly different... (of course) - May be worth a breaking change PR if the owner can be persuaded. Failing that a new node that did this would be useful (especially for all those playing with modbus, plcs etc)

---

<div class="post-metadata">

**Author:** ![Andrei](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/andrei/32/10446_2.png) [@Andrei](https://discourse.nodered.org/u/Andrei)\
**Post date:** [26 November 2018 01:24 UTC](https://discourse.nodered.org/t/flow-to-decode-binary-data-and-a-walk-through-on-how-to-install-the-npm-module/5174/7 "2018-11-26T01:24:41Z")

</div>

Hi Bart,

Dave already replied to question #1 so let me just add my view on question #2.

I strongly believe that the conversion binary \> UTF-8 \> binary is not possible at all. If you have a look on the [character set for UTF-8](https://www.fileformat.info/info/charset/UTF-8/list.htm) you will see that many bytes would be converted to the very same character (mainly in the range 0 to 31 ) in the trip back. So it is not possible a round-trip conversion. Secondly, as you are working with an image data it may contain any possible byte (from 0 to 255). Note that many bytes are invalid in UTF-8 so they break the code or, most likely, the encoder will convert these bytes to the **replacement character** �  
I am afraid that the answer is that the image, if converted that way, is lost forever. RIP data.

Edit: another issue is the fact already pointed out by Dave that the conversion from binary to UTF-8 will eventually generate extra bytes since UTF-8 is a variable-length encoding.

---

<div class="post-metadata">

**Author:** ![lr\_oliveira](https://avatars.discourse-cdn.com/v4/letter/l/f6c823/32.png) [@lr\_oliveira](https://discourse.nodered.org/u/lr_oliveira)\
**Post date:** [26 November 2018 15:09 UTC](https://discourse.nodered.org/t/flow-to-decode-binary-data-and-a-walk-through-on-how-to-install-the-npm-module/5174/8 "2018-11-26T15:09:53Z")

</div>

I'm having the same job and were having trouble sending your email a lot. No you're not able to get an array of measured points that my case is 90. Do you have any idea to do this? Here's how my data is returning and the line I've added to your code to get the cloud of points. I think I have to do an array of 90 positions but when tntei an error occurred in the function. Also I could not make a "for" to fill the Array. Anyway, I believe that I am not very experienced in javascript and so I am having these difficulties.

 ![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/2X/e/ea0e107d772c853afdff7fb0ccecf792c277e0e1.png)  
 ![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/2X/a/abcfced9b67bbbb6eecb6be5431660d8b9a4fef8.png)  
 ![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/2X/0/05f70bff965b51c06ccb8366ec1bdfd3c3880f82.png)

---

<div class="post-metadata">

**Author:** ![nenadvkc](https://avatars.discourse-cdn.com/v4/letter/n/90ced4/32.png) [@nenadvkc](https://discourse.nodered.org/u/nenadvkc)\
**Post date:** [21 July 2020 17:48 UTC](https://discourse.nodered.org/t/flow-to-decode-binary-data-and-a-walk-through-on-how-to-install-the-npm-module/5174/9 "2020-07-21T17:48:51Z")

</div>

I am getting error message when using this flow, anybody knows why?  
"TypeError: Packet is not a constructor"

---

<div class="post-metadata">

**Author:** ![Steve-Mcl](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/steve-mcl/32/4826_2.png) [@Steve-Mcl](https://discourse.nodered.org/u/Steve-Mcl)\
**Post date:** [21 July 2020 18:02 UTC](https://discourse.nodered.org/t/flow-to-decode-binary-data-and-a-walk-through-on-how-to-install-the-npm-module/5174/10 "2020-07-21T18:02:24Z")

</div>

1. Did you ready the very first post where it tells you to install `binary-parser` and modify setings.js?
2. what are you trying to achieve? Things have changed a lot since Nov 2018 - there may well be a suitable node to do your type conversions.

PS, you _really_ should start a new thread & if necessary link back to this one.

---

<div class="post-metadata">

**Author:** ![nenadvkc](https://avatars.discourse-cdn.com/v4/letter/n/90ced4/32.png) [@nenadvkc](https://discourse.nodered.org/u/nenadvkc)\
**Post date:** [21 July 2020 18:56 UTC](https://discourse.nodered.org/t/flow-to-decode-binary-data-and-a-walk-through-on-how-to-install-the-npm-module/5174/11 "2020-07-21T18:56:08Z")

</div>

1.I did not modify settings.js I missed that step, now after an edit, I am getting another error  
"TypeError: (intermediate value).endianess(...).uint16(...).uint16(...).uint32(...).uint16(...).uint16(...).uint16(...).double is not a function"  
2. I would like to parse a binary .txt file

---

<div class="post-metadata">

**Author:** ![Steve-Mcl](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/steve-mcl/32/4826_2.png) [@Steve-Mcl](https://discourse.nodered.org/u/Steve-Mcl)\
**Post date:** [21 July 2020 19:06 UTC](https://discourse.nodered.org/t/flow-to-decode-binary-data-and-a-walk-through-on-how-to-install-the-npm-module/5174/12 "2020-07-21T19:06:38Z")

</div>

> [@nenadvkc](#):
>
> would like to parse a binary .txt file

You could probably achieve that like this...

trigger (i.e. inject / cron) --\> file-in node (return buffer) --\> [node-red-contrib-buffer-parser](https://flows.nodered.org/node/node-red-contrib-buffer-parser)

---

<div class="post-metadata">

**Author:** ![nenadvkc](https://avatars.discourse-cdn.com/v4/letter/n/90ced4/32.png) [@nenadvkc](https://discourse.nodered.org/u/nenadvkc)\
**Post date:** [21 July 2020 19:16 UTC](https://discourse.nodered.org/t/flow-to-decode-binary-data-and-a-walk-through-on-how-to-install-the-npm-module/5174/13 "2020-07-21T19:16:44Z")

</div>

Thanks , this looks promising