# Newbie question about extracting the data from buffer

**URL:** <https://discourse.nodered.org/t/newbie-question-about-extracting-the-data-from-buffer/50753>\
**Category:** General\
**Created:** [8 September 2021 10:41 UTC](https://discourse.nodered.org/t/newbie-question-about-extracting-the-data-from-buffer/50753 "2021-09-08T10:41:46Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![Volt](https://avatars.discourse-cdn.com/v4/letter/v/9de0a6/32.png) [@Volt](https://discourse.nodered.org/u/Volt)\
**Post date:** [8 September 2021 10:41 UTC](https://discourse.nodered.org/t/newbie-question-about-extracting-the-data-from-buffer/50753/1 "2021-09-08T10:41:46Z")

</div>

Hello to all fellow forum members - this is my first post.

I have a problem regarding reading data from a device.  
I have a device that provides information in the form of data frames - for example:

```auto
{"topic":"","payload":[126,1,12,0,33,4,0,0,0,0,4,118,5,0,8,0,0,0,0,215,145,165,25,5,1,8,1,1,75,7,165,137,87,0,5,2,8,112,217,147,25,167,97,47,21,213,99,126,1,12,0,11,4,0,0,0,0,183,124,7,131,8,0,0,0,0,0,0,0,0,129,182,126,1,12,0,29,4,0,0,0,0,32,25,7,128,4,12,9,63,0,7,128,8,165,4,0,0,174,137,56,97,7,130],"_session":{"type":"tcp","id":"baff59f996d58e61"},"_msgid":"f1b4160dfdea5014"}

```

Received data are OK - they are clear to me.

 ![Clipboard02](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/f/5/f5fe771d3cdb23abccbb3d3753ccc3da0e2919fa.jpeg)

I've divided it to three lines, but it was transmitted in one message.

In grey there are data I don't need.  
I need to extract only the data in white.  
The white data have always the same structure (I'll call it "data frame"):  
 ![Clipboard03](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/c/9/c94eb680819d8907210451ebc1d1c8b7b81d9cb1.jpeg)  
There are two bytes of Identifier, next one byte of length of the data and then the data.  
I would like to extract the identifier and the data to pass them to the database.

The problem is:

- the order of the "data frames" in the message is not the same every time;
- the length of the "data field" in the "data frame" could be different for different Identifiers;

I will be grateful for any help in extracting the data.

Kind regards,  
Dawid

---

<div class="post-metadata">

**Author:** ![Volt](https://avatars.discourse-cdn.com/v4/letter/v/9de0a6/32.png) [@Volt](https://discourse.nodered.org/u/Volt)\
**Post date:** [8 September 2021 12:54 UTC](https://discourse.nodered.org/t/newbie-question-about-extracting-the-data-from-buffer/50753/2 "2021-09-08T12:54:50Z")

</div>

I'll partly answer to myself:  
the original data is in DEC format - I can easily translate it to HEX and join to have one string:  
Using node - function:

```auto
msg.payload = msg.payload.toString('hex');
return msg;

```

and then the msg.payload looks like these:

```auto
{"topic":"","payload":"7e010c00580400000000090805000800000000756a331a05010806014b07ce6d5900050208be5cb21265673300050308ba1ce16aa868333e050408105c806a25689d3d05050812dc56e2a473330105060824196ba9a6602f0205070886984939a7602f036d127e01","_session":{"type":"tcp","id":"9c5bb6cd66442a9d"},"_msgid":"d1483e5dbf19868e"}

```

Now - we have one string containing all information - every byte of data is represented by two signs.  
Now (in the string) we have to find the Identifier (4 signs), read the next 2 signs - data Length, read data - next XX signs (length based on the Length value).  
For example: in the string find the Identifier: 0500, read the next signs - Length value: 08, read the next signs - the Data: 00000000756a331a

This is the point where I stop and cannot go any further.  
Would be grateful for help.

Kind regards,  
Dawid

---

<div class="post-metadata">

**Author:** ![Steve-Mcl](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/steve-mcl/32/4826_2.png) [@Steve-Mcl](https://discourse.nodered.org/u/Steve-Mcl)\
**Post date:** [8 September 2021 13:05 UTC](https://discourse.nodered.org/t/newbie-question-about-extracting-the-data-from-buffer/50753/3 "2021-09-08T13:05:09Z")

</div>

[Buffer-parser](https://flows.nodered.org/node/node-red-contrib-buffer-parser) can help you with extracting and converting.

---

<div class="post-metadata">

**Author:** ![Volt](https://avatars.discourse-cdn.com/v4/letter/v/9de0a6/32.png) [@Volt](https://discourse.nodered.org/u/Volt)\
**Post date:** [8 September 2021 13:16 UTC](https://discourse.nodered.org/t/newbie-question-about-extracting-the-data-from-buffer/50753/4 "2021-09-08T13:16:05Z")

</div>

Hi @Steve-Mcl,  
thanks for Your replay.

I've already tried the buffer-parser, but I can't set there the flexible position / size of the interested data - as I've wrote in the first post:

- the order of the "data frames" in the message is not the same every time;
- the length of the "data field" in the "data frame" could be different for different Identifiers;

Or maybe I can't set it up properly?

Kind regards,  
Dawid

---

<div class="post-metadata">

**Author:** ![Steve-Mcl](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/steve-mcl/32/4826_2.png) [@Steve-Mcl](https://discourse.nodered.org/u/Steve-Mcl)\
**Post date:** [8 September 2021 13:20 UTC](https://discourse.nodered.org/t/newbie-question-about-extracting-the-data-from-buffer/50753/5 "2021-09-08T13:20:46Z")

</div>

No, what you do is parse the fixed elements (like id and length) then pass that to a 2nd buffer parser.

If you look at the built in help, you can actually provide a specification as the input - i.e. you can create a msg with a dynamic spec built from previous details parsed.

I dont have time to do a demo for you until later so have a go yourself if you feel up to it.

---

<div class="post-metadata">

**Author:** ![Volt](https://avatars.discourse-cdn.com/v4/letter/v/9de0a6/32.png) [@Volt](https://discourse.nodered.org/u/Volt)\
**Post date:** [8 September 2021 13:24 UTC](https://discourse.nodered.org/t/newbie-question-about-extracting-the-data-from-buffer/50753/6 "2021-09-08T13:24:37Z")

</div>

Thanks - I'll try.

---

<div class="post-metadata">

**Author:** ![Steve-Mcl](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/steve-mcl/32/4826_2.png) [@Steve-Mcl](https://discourse.nodered.org/u/Steve-Mcl)\
**Post date:** [8 September 2021 20:36 UTC](https://discourse.nodered.org/t/newbie-question-about-extracting-the-data-from-buffer/50753/7 "2021-09-08T20:36:29Z")

</div>

Did you get anywhere?

Here is a low code solution...

 ![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/9/6/96b83a790093e038e91a895c7cc092edb3143a99.png)

```auto
[{"id":"1ed34be2ac16ffbe","type":"inject","z":"62a0e5a5.fc59bc","name":"Your data","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"[126,1,12,0,33,4,0,0,0,0,4,118,5,0,8,0,0,0,0,215,145,165,25,5,1,8,1,1,75,7,165,137,87,0,5,2,8,112,217,147,25,167,97,47,21,213,99,126,1,12,0,11,4,0,0,0,0,183,124,7,131,8,0,0,0,0,0,0,0,0,129,182,126,1,12,0,29,4,0,0,0,0,32,25,7,128,4,12,9,63,0,7,128,8,165,4,0,0,174,137,56,97,7,130]","payloadType":"bin","x":180,"y":860,"wires":[["30ac9422f5925948"]]},{"id":"30ac9422f5925948","type":"buffer-parser","z":"62a0e5a5.fc59bc","name":"get header","data":"payload","dataType":"msg","specification":"spec","specificationType":"ui","items":[{"type":"byte","name":"start","offset":0,"length":1,"offsetbit":0,"scale":"1","mask":""},{"type":"int16be","name":"messageLength","offset":3,"length":1,"offsetbit":0,"scale":"1","mask":""},{"type":"int16be","name":"headerCRC","offset":10,"length":1,"offsetbit":0,"scale":"1","mask":""},{"type":"buffer","name":"nextPart","offset":12,"length":-1,"offsetbit":0,"scale":"1","mask":""}],"swap1":"","swap2":"","swap3":"","swap1Type":"swap","swap2Type":"swap","swap3Type":"swap","msgProperty":"payload","msgPropertyType":"str","resultType":"keyvalue","resultTypeType":"output","multipleResult":false,"fanOutMultipleResult":false,"setTopic":true,"outputs":1,"x":370,"y":860,"wires":[["cdcbf31114649e96"]]},{"id":"8ef8e2e92f47ec1f","type":"buffer-parser","z":"62a0e5a5.fc59bc","name":"get part length","data":"payload","dataType":"msg","specification":"spec","specificationType":"ui","items":[{"type":"uint8","name":"length","offset":2,"length":1,"offsetbit":0,"scale":"1","mask":""},{"type":"buffer","name":"packetData","offset":0,"length":-1,"offsetbit":0,"scale":"1","mask":""}],"swap1":"","swap2":"","swap3":"","swap1Type":"swap","swap2Type":"swap","swap3Type":"swap","msgProperty":"payload","msgPropertyType":"str","resultType":"keyvalue","resultTypeType":"output","multipleResult":false,"fanOutMultipleResult":false,"setTopic":true,"outputs":1,"x":660,"y":1020,"wires":[["2ddaabb1ca92e1da"]]},{"id":"725be264fa2bf8c0","type":"debug","z":"62a0e5a5.fc59bc","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":1050,"y":1100,"wires":[]},{"id":"2ddaabb1ca92e1da","type":"function","z":"62a0e5a5.fc59bc","name":"build spec","func":"msg.spec = { \n \"options\": { \n \"byteSwap\": [],\n \"resultType\": \"keyvalue\",\n \"singleResult\": true,\n \"msgProperty\": \"payload\",\n \"setTopic\": true },\n \"items\": [\n { \"type\": \"uint16be\",\n \"name\": \"ident\",\n \"offset\": 0,\n \"length\": 1\n }, \n { \"type\": \"uint8\",\n \"name\": \"length\",\n \"offset\": 2,\n \"length\": 1\n }, \n { \"type\": \"buffer\",\n \"name\": \"data\",\n \"offset\": 3,\n \"length\": msg.payload.length\n }, \n { \n \"type\": \"buffer\", \n \"name\": \"nextPart\", \n \"offset\": 3+msg.payload.length,\n \"length\": -1\n }\n] \n }\n\n\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":860,"y":1020,"wires":[["acce00c0633a994d","9eadd537281269a4"]]},{"id":"acce00c0633a994d","type":"buffer-parser","z":"62a0e5a5.fc59bc","name":"","data":"payload.packetData","dataType":"msg","specification":"spec","specificationType":"msg","items":[{"type":"buffer","name":"data","offset":0,"length":1,"offsetbit":0,"scale":"1","mask":""},{"type":"buffer","name":"buffer","offset":99,"length":-1,"offsetbit":0,"scale":"1","mask":""}],"swap1":"","swap2":"","swap3":"","swap1Type":"swap","swap2Type":"swap","swap3Type":"swap","msgProperty":"payload","msgPropertyType":"str","resultType":"keyvalue","resultTypeType":"output","multipleResult":false,"fanOutMultipleResult":false,"setTopic":true,"outputs":1,"x":650,"y":1100,"wires":[["725be264fa2bf8c0","4ed279831786f0c1"]]},{"id":"4ed279831786f0c1","type":"switch","z":"62a0e5a5.fc59bc","name":"more/crc/err","property":"payload.nextPart.length","propertyType":"msg","rules":[{"t":"gt","v":"2","vt":"num"},{"t":"eq","v":"2","vt":"num"},{"t":"else"}],"checkall":"true","repair":false,"outputs":3,"x":370,"y":1160,"wires":[["f1ac096e0cf679b4"],["3445ee8d514a6ce7"],["331ddd85e302c671"]]},{"id":"3445ee8d514a6ce7","type":"buffer-parser","z":"62a0e5a5.fc59bc","name":"get crc","data":"payload.nextPart","dataType":"msg","specification":"spec","specificationType":"ui","items":[{"type":"int16be","name":"crc","offset":0,"length":1,"offsetbit":0,"scale":"1","mask":""}],"swap1":"","swap2":"","swap3":"","swap1Type":"swap","swap2Type":"swap","swap3Type":"swap","msgProperty":"payload","msgPropertyType":"str","resultType":"keyvalue","resultTypeType":"output","multipleResult":false,"fanOutMultipleResult":false,"setTopic":true,"outputs":1,"x":630,"y":1180,"wires":[["3a522af689fecdb3"]]},{"id":"331ddd85e302c671","type":"function","z":"62a0e5a5.fc59bc","name":"raise error","func":"node.error(\"Expected 2 or more bytes - data may be corrupt\", msg);\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":420,"y":1220,"wires":[[]]},{"id":"3a522af689fecdb3","type":"debug","z":"62a0e5a5.fc59bc","name":"","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":1050,"y":1180,"wires":[]},{"id":"cdcbf31114649e96","type":"function","z":"62a0e5a5.fc59bc","name":"build spec","func":"msg.spec = { \n \"options\": { \n \"byteSwap\": [],\n \"resultType\": \"keyvalue\",\n \"singleResult\": false,\n \"msgProperty\": \"payload\",\n \"setTopic\": true \n },\n \"items\": [\n { \n \"type\": \"buffer\", \n \"name\": \"packetData\", \n \"topic\": \"packetData\",\n \"offset\": 0,\n \"length\": msg.payload.messageLength + 2//CRC is 2 long\n },\n {\n \"type\": \"buffer\",\n \"name\": \"nextPart\",\n \"topic\": \"nextPart\",\n \"offset\": msg.payload.messageLength + 2,\n \"length\": -1\n }\n] \n}\n\n\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":580,"y":860,"wires":[["2f408bafcf485bc3"]]},{"id":"2f408bafcf485bc3","type":"buffer-parser","z":"62a0e5a5.fc59bc","name":"get message data","data":"payload.nextPart","dataType":"msg","specification":"spec","specificationType":"msg","items":[{"type":"buffer","name":"data","offset":0,"length":1,"offsetbit":0,"scale":"1","mask":""},{"type":"buffer","name":"buffer","offset":99,"length":-1,"offsetbit":0,"scale":"1","mask":""}],"swap1":"","swap2":"","swap3":"","swap1Type":"swap","swap2Type":"swap","swap3Type":"swap","msgProperty":"payload","msgPropertyType":"str","resultType":"keyvalue","resultTypeType":"output","multipleResult":false,"fanOutMultipleResult":false,"setTopic":true,"outputs":1,"x":790,"y":860,"wires":[["5234ab5b302e7b64","763db38bb4b571d2"]]},{"id":"5234ab5b302e7b64","type":"debug","z":"62a0e5a5.fc59bc","name":"","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":1030,"y":860,"wires":[]},{"id":"9eadd537281269a4","type":"debug","z":"62a0e5a5.fc59bc","name":"","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":1030,"y":1020,"wires":[]},{"id":"f1ac096e0cf679b4","type":"change","z":"62a0e5a5.fc59bc","name":"move nextPart to payload","rules":[{"t":"set","p":"payload","pt":"msg","to":"payload.nextPart","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":410,"y":1020,"wires":[["8ef8e2e92f47ec1f"]]},{"id":"763db38bb4b571d2","type":"switch","z":"62a0e5a5.fc59bc","name":"nextpart/packetData","property":"topic","propertyType":"msg","rules":[{"t":"eq","v":"nextPart","vt":"str"},{"t":"eq","v":"packetData","vt":"str"}],"checkall":"true","repair":false,"outputs":2,"x":400,"y":940,"wires":[["db3a9c7325629428"],["8ef8e2e92f47ec1f"]]},{"id":"db3a9c7325629428","type":"switch","z":"62a0e5a5.fc59bc","name":"Length>0","property":"payload.length","propertyType":"msg","rules":[{"t":"gt","v":"0","vt":"num"}],"checkall":"true","repair":false,"outputs":1,"x":640,"y":940,"wires":[["30ac9422f5925948"]]}]

```

---

<div class="post-metadata">

**Author:** ![Volt](https://avatars.discourse-cdn.com/v4/letter/v/9de0a6/32.png) [@Volt](https://discourse.nodered.org/u/Volt)\
**Post date:** [9 September 2021 05:30 UTC](https://discourse.nodered.org/t/newbie-question-about-extracting-the-data-from-buffer/50753/8 "2021-09-09T05:30:04Z")

</div>

Hi @Steve-Mcl,

I've tried myself, but no success.  
Your solution looks like a little bit of magic! 😮

I'll test it!

Kind regards,  
Dawid

---

<div class="post-metadata">

**Author:** ![Volt](https://avatars.discourse-cdn.com/v4/letter/v/9de0a6/32.png) [@Volt](https://discourse.nodered.org/u/Volt)\
**Post date:** [9 September 2021 09:35 UTC](https://discourse.nodered.org/t/newbie-question-about-extracting-the-data-from-buffer/50753/9 "2021-09-09T09:35:49Z")

</div>

Hi @Steve-Mcl,

I am very grateful for your help!  
One thing I am absolutely sure of - I would not have come up with this on my own!  
What is more - I'm just realizing how much I still have to learn.

I have one small question:  
for example - for now I would like to use only one object (Identifier: 1923)  
 ![Clipboard04](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/3/8/38cbcb226435d42a170507a64fd6a84505a3cbac.jpeg)  
and wrote the Data to the database.  
What should I use - switch node?

Kind regards,  
Dawid

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/1X/d073cd938eafa2e558d7c2cd59003b3ef4963033.png) [@system](https://discourse.nodered.org/u/system)\
**Post date:** [8 November 2021 09:36 UTC](https://discourse.nodered.org/t/newbie-question-about-extracting-the-data-from-buffer/50753/10 "2021-11-08T09:36:10Z")

</div>

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.