Newbie question about extracting the data from buffer

General
1

Hello to all fellow forum members - this is my first post.

I have a problem regarding reading data from a device.
I have a device that provides information in the form of data frames - for example:

{"topic":"","payload":[126,1,12,0,33,4,0,0,0,0,4,118,5,0,8,0,0,0,0,215,145,165,25,5,1,8,1,1,75,7,165,137,87,0,5,2,8,112,217,147,25,167,97,47,21,213,99,126,1,12,0,11,4,0,0,0,0,183,124,7,131,8,0,0,0,0,0,0,0,0,129,182,126,1,12,0,29,4,0,0,0,0,32,25,7,128,4,12,9,63,0,7,128,8,165,4,0,0,174,137,56,97,7,130],"_session":{"type":"tcp","id":"baff59f996d58e61"},"_msgid":"f1b4160dfdea5014"}

Received data are OK - they are clear to me.

Clipboard02
Clipboard021310Ă—183 99.1 KB

I've divided it to three lines, but it was transmitted in one message.

In grey there are data I don't need.
I need to extract only the data in white.
The white data have always the same structure (I'll call it "data frame"):
Clipboard03
There are two bytes of Identifier, next one byte of length of the data and then the data.
I would like to extract the identifier and the data to pass them to the database.

The problem is:

  • the order of the "data frames" in the message is not the same every time;
  • the length of the "data field" in the "data frame" could be different for different Identifiers;

I will be grateful for any help in extracting the data.

Kind regards,
Dawid

2

I'll partly answer to myself:
the original data is in DEC format - I can easily translate it to HEX and join to have one string:
Using node - function:

msg.payload = msg.payload.toString('hex');
return msg;

and then the msg.payload looks like these:

{"topic":"","payload":"7e010c00580400000000090805000800000000756a331a05010806014b07ce6d5900050208be5cb21265673300050308ba1ce16aa868333e050408105c806a25689d3d05050812dc56e2a473330105060824196ba9a6602f0205070886984939a7602f036d127e01","_session":{"type":"tcp","id":"9c5bb6cd66442a9d"},"_msgid":"d1483e5dbf19868e"}

Now - we have one string containing all information - every byte of data is represented by two signs.
Now (in the string) we have to find the Identifier (4 signs), read the next 2 signs - data Length, read data - next XX signs (length based on the Length value).
For example: in the string find the Identifier: 0500, read the next signs - Length value: 08, read the next signs - the Data: 00000000756a331a

This is the point where I stop and cannot go any further.
Would be grateful for help.

Kind regards,
Dawid

3

Buffer-parser can help you with extracting and converting.

4

Hi @Steve-Mcl,
thanks for Your replay.

I've already tried the buffer-parser, but I can't set there the flexible position / size of the interested data - as I've wrote in the first post:

  • the order of the "data frames" in the message is not the same every time;
  • the length of the "data field" in the "data frame" could be different for different Identifiers;

Or maybe I can't set it up properly?

Kind regards,
Dawid

5

No, what you do is parse the fixed elements (like id and length) then pass that to a 2nd buffer parser.

If you look at the built in help, you can actually provide a specification as the input - i.e. you can create a msg with a dynamic spec built from previous details parsed.

I dont have time to do a demo for you until later so have a go yourself if you feel up to it.

6

Thanks - I'll try.

7

Did you get anywhere?

Here is a low code solution...

image
image1158Ă—804 73.1 KB 

[{"id":"1ed34be2ac16ffbe","type":"inject","z":"62a0e5a5.fc59bc","name":"Your data","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"[126,1,12,0,33,4,0,0,0,0,4,118,5,0,8,0,0,0,0,215,145,165,25,5,1,8,1,1,75,7,165,137,87,0,5,2,8,112,217,147,25,167,97,47,21,213,99,126,1,12,0,11,4,0,0,0,0,183,124,7,131,8,0,0,0,0,0,0,0,0,129,182,126,1,12,0,29,4,0,0,0,0,32,25,7,128,4,12,9,63,0,7,128,8,165,4,0,0,174,137,56,97,7,130]","payloadType":"bin","x":180,"y":860,"wires":[["30ac9422f5925948"]]},{"id":"30ac9422f5925948","type":"buffer-parser","z":"62a0e5a5.fc59bc","name":"get header","data":"payload","dataType":"msg","specification":"spec","specificationType":"ui","items":[{"type":"byte","name":"start","offset":0,"length":1,"offsetbit":0,"scale":"1","mask":""},{"type":"int16be","name":"messageLength","offset":3,"length":1,"offsetbit":0,"scale":"1","mask":""},{"type":"int16be","name":"headerCRC","offset":10,"length":1,"offsetbit":0,"scale":"1","mask":""},{"type":"buffer","name":"nextPart","offset":12,"length":-1,"offsetbit":0,"scale":"1","mask":""}],"swap1":"","swap2":"","swap3":"","swap1Type":"swap","swap2Type":"swap","swap3Type":"swap","msgProperty":"payload","msgPropertyType":"str","resultType":"keyvalue","resultTypeType":"output","multipleResult":false,"fanOutMultipleResult":false,"setTopic":true,"outputs":1,"x":370,"y":860,"wires":[["cdcbf31114649e96"]]},{"id":"8ef8e2e92f47ec1f","type":"buffer-parser","z":"62a0e5a5.fc59bc","name":"get part length","data":"payload","dataType":"msg","specification":"spec","specificationType":"ui","items":[{"type":"uint8","name":"length","offset":2,"length":1,"offsetbit":0,"scale":"1","mask":""},{"type":"buffer","name":"packetData","offset":0,"length":-1,"offsetbit":0,"scale":"1","mask":""}],"swap1":"","swap2":"","swap3":"","swap1Type":"swap","swap2Type":"swap","swap3Type":"swap","msgProperty":"payload","msgPropertyType":"str","resultType":"keyvalue","resultTypeType":"output","multipleResult":false,"fanOutMultipleResult":false,"setTopic":true,"outputs":1,"x":660,"y":1020,"wires":[["2ddaabb1ca92e1da"]]},{"id":"725be264fa2bf8c0","type":"debug","z":"62a0e5a5.fc59bc","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":1050,"y":1100,"wires":[]},{"id":"2ddaabb1ca92e1da","type":"function","z":"62a0e5a5.fc59bc","name":"build spec","func":"msg.spec = { \n    \"options\": { \n        \"byteSwap\": [],\n         \"resultType\": \"keyvalue\",\n         \"singleResult\": true,\n         \"msgProperty\": \"payload\",\n         \"setTopic\": true },\n         \"items\": [\n            { \"type\": \"uint16be\",\n                \"name\": \"ident\",\n                \"offset\": 0,\n                \"length\": 1\n            }, \n            { \"type\": \"uint8\",\n                \"name\": \"length\",\n                \"offset\": 2,\n                \"length\": 1\n            }, \n            { \"type\": \"buffer\",\n                \"name\": \"data\",\n                \"offset\": 3,\n                \"length\": msg.payload.length\n            }, \n            { \n                \"type\": \"buffer\", \n                \"name\": \"nextPart\", \n                \"offset\": 3+msg.payload.length,\n                \"length\": -1\n            }\n        ] \n    }\n\n\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":860,"y":1020,"wires":[["acce00c0633a994d","9eadd537281269a4"]]},{"id":"acce00c0633a994d","type":"buffer-parser","z":"62a0e5a5.fc59bc","name":"","data":"payload.packetData","dataType":"msg","specification":"spec","specificationType":"msg","items":[{"type":"buffer","name":"data","offset":0,"length":1,"offsetbit":0,"scale":"1","mask":""},{"type":"buffer","name":"buffer","offset":99,"length":-1,"offsetbit":0,"scale":"1","mask":""}],"swap1":"","swap2":"","swap3":"","swap1Type":"swap","swap2Type":"swap","swap3Type":"swap","msgProperty":"payload","msgPropertyType":"str","resultType":"keyvalue","resultTypeType":"output","multipleResult":false,"fanOutMultipleResult":false,"setTopic":true,"outputs":1,"x":650,"y":1100,"wires":[["725be264fa2bf8c0","4ed279831786f0c1"]]},{"id":"4ed279831786f0c1","type":"switch","z":"62a0e5a5.fc59bc","name":"more/crc/err","property":"payload.nextPart.length","propertyType":"msg","rules":[{"t":"gt","v":"2","vt":"num"},{"t":"eq","v":"2","vt":"num"},{"t":"else"}],"checkall":"true","repair":false,"outputs":3,"x":370,"y":1160,"wires":[["f1ac096e0cf679b4"],["3445ee8d514a6ce7"],["331ddd85e302c671"]]},{"id":"3445ee8d514a6ce7","type":"buffer-parser","z":"62a0e5a5.fc59bc","name":"get crc","data":"payload.nextPart","dataType":"msg","specification":"spec","specificationType":"ui","items":[{"type":"int16be","name":"crc","offset":0,"length":1,"offsetbit":0,"scale":"1","mask":""}],"swap1":"","swap2":"","swap3":"","swap1Type":"swap","swap2Type":"swap","swap3Type":"swap","msgProperty":"payload","msgPropertyType":"str","resultType":"keyvalue","resultTypeType":"output","multipleResult":false,"fanOutMultipleResult":false,"setTopic":true,"outputs":1,"x":630,"y":1180,"wires":[["3a522af689fecdb3"]]},{"id":"331ddd85e302c671","type":"function","z":"62a0e5a5.fc59bc","name":"raise error","func":"node.error(\"Expected 2 or more bytes - data may be corrupt\", msg);\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":420,"y":1220,"wires":[[]]},{"id":"3a522af689fecdb3","type":"debug","z":"62a0e5a5.fc59bc","name":"","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":1050,"y":1180,"wires":[]},{"id":"cdcbf31114649e96","type":"function","z":"62a0e5a5.fc59bc","name":"build spec","func":"msg.spec = { \n    \"options\": { \n        \"byteSwap\": [],\n        \"resultType\": \"keyvalue\",\n        \"singleResult\": false,\n        \"msgProperty\": \"payload\",\n        \"setTopic\": true \n    },\n    \"items\": [\n        { \n            \"type\": \"buffer\", \n            \"name\": \"packetData\", \n            \"topic\": \"packetData\",\n            \"offset\": 0,\n            \"length\": msg.payload.messageLength + 2//CRC is 2 long\n        },\n        {\n            \"type\": \"buffer\",\n            \"name\": \"nextPart\",\n            \"topic\": \"nextPart\",\n            \"offset\": msg.payload.messageLength + 2,\n            \"length\": -1\n        }\n    ] \n}\n\n\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":580,"y":860,"wires":[["2f408bafcf485bc3"]]},{"id":"2f408bafcf485bc3","type":"buffer-parser","z":"62a0e5a5.fc59bc","name":"get message data","data":"payload.nextPart","dataType":"msg","specification":"spec","specificationType":"msg","items":[{"type":"buffer","name":"data","offset":0,"length":1,"offsetbit":0,"scale":"1","mask":""},{"type":"buffer","name":"buffer","offset":99,"length":-1,"offsetbit":0,"scale":"1","mask":""}],"swap1":"","swap2":"","swap3":"","swap1Type":"swap","swap2Type":"swap","swap3Type":"swap","msgProperty":"payload","msgPropertyType":"str","resultType":"keyvalue","resultTypeType":"output","multipleResult":false,"fanOutMultipleResult":false,"setTopic":true,"outputs":1,"x":790,"y":860,"wires":[["5234ab5b302e7b64","763db38bb4b571d2"]]},{"id":"5234ab5b302e7b64","type":"debug","z":"62a0e5a5.fc59bc","name":"","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":1030,"y":860,"wires":[]},{"id":"9eadd537281269a4","type":"debug","z":"62a0e5a5.fc59bc","name":"","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":1030,"y":1020,"wires":[]},{"id":"f1ac096e0cf679b4","type":"change","z":"62a0e5a5.fc59bc","name":"move nextPart to payload","rules":[{"t":"set","p":"payload","pt":"msg","to":"payload.nextPart","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":410,"y":1020,"wires":[["8ef8e2e92f47ec1f"]]},{"id":"763db38bb4b571d2","type":"switch","z":"62a0e5a5.fc59bc","name":"nextpart/packetData","property":"topic","propertyType":"msg","rules":[{"t":"eq","v":"nextPart","vt":"str"},{"t":"eq","v":"packetData","vt":"str"}],"checkall":"true","repair":false,"outputs":2,"x":400,"y":940,"wires":[["db3a9c7325629428"],["8ef8e2e92f47ec1f"]]},{"id":"db3a9c7325629428","type":"switch","z":"62a0e5a5.fc59bc","name":"Length>0","property":"payload.length","propertyType":"msg","rules":[{"t":"gt","v":"0","vt":"num"}],"checkall":"true","repair":false,"outputs":1,"x":640,"y":940,"wires":[["30ac9422f5925948"]]}]
1 Like
8

Hi @Steve-Mcl,

I've tried myself, but no success.
Your solution looks like a little bit of magic! :open_mouth:

I'll test it!

Kind regards,
Dawid

9

Hi @Steve-Mcl,

I am very grateful for your help!
One thing I am absolutely sure of - I would not have come up with this on my own!
What is more - I'm just realizing how much I still have to learn.

I have one small question:
for example - for now I would like to use only one object (Identifier: 1923)
Clipboard04
and wrote the Data to the database.
What should I use - switch node?

Kind regards,
Dawid

10

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.