[pre-release] node-red-contrib-dataparser (or whatever you guys think is a better name)

Hi all,

I have a private node i use for converting (mostly PLC) data into usable values for various projects. I would like some fresh eyes and thoughts.

FIRSTLY: about that name - as you read through the below & understand its capabilities - if you can think of a much nicer name - i'm open to suggestions

Foreword, background & reasoning...

  • I work with PLCs and industrial devices and much of the type the data I get is a buffer or array of 16bit integer
  • Often, I need to convert various parts of this data to BCD, UINT16LE, FLOATLE etc.
  • I must have read 50 times (especially MODBUS threads) "how do I get a 32 bit integer" "but its not the right way round" etc etc
  • Depending on the equipment, the data is often byteswapped, sometimes it is even necessary to swap32 followed by swap16 to sort the data
  • Many times I need to get flags (bits/booleans) from a WORD but also need the whole word as an integer too
  • sometimes...
    • I need a simple array of values created from the data
    • I need a object with named properties (for easy access)
    • I need individual topic/payloads generated (for easy sending to MQTT)
    • other times i just want the data as a buffer in the right order

Inputs...

  • data - a buffer (of bytes) or an array (of 16bit integer)
  • specification - an object specifying what you want to do with it

Options...

  • The specification can instruct the node to
    • byteswap 16, 32, 64. Not only once but you can actually tell it to swap32, then swap64, then swap16 (and so on) if thats what is required
    • extract bits, booleans, ints, uints, floats, doubles all with BE and LE options
    • can re-use any part of the buffer e.g.
      • take 2 bytes starting at offset 0 & create a UINT16BE and also take the same 2 bytes starting at offset 0 & create a UINT16LE and also take 4 bytes starting at offset 0 & create a INT32BE and so on
      • can get a bit (or bit array) from data starting at a byte offset & bit offset

Outputs 4 formats...

  • value - just the values you specified in an array
  • object - a named object contaiing .vaue and other parts of your specification to help you correlate the value (provides context to your value)
  • array - a combination of above 2 (an array of context objects)
  • buffer - no processing other than byteSwap(s). This is useful for ...
    • turning an array into a buffer
    • simple byteswaps

Outputs options...

  • can return a single result with all conversions
  • can return multiple results with a topic & payload for each item in your specification

Some Samples

Typical PLC Data (array 16bit) converted to various formatted values

The data...

[25185,25699,26213,26727,27241,27755,28013,28783,29297,29811,30325,30839,31353,256,512,768,1024,1280,1536,1792,2048,2304,2560,2816,3072,3597]

specification...

{
    "options": {
        "byteSwap": [
            "swap16"
        ],
        "resultType": "value",
        "singleResult": false,
        "msgProperty": "data"
    },
    "items": [
        {
            "name": "plc1/production/alphabet",
            "type": "string",
            "offset": 0,
            "length": 26
        },
        {
            "name": "plc1/production/status/counts",
            "type": "int",
            "offset": 4
        },
        {
            "name": "plc1/production/status/sequence",
            "type": "bcd",
            "offset": 4,
            "length": 5
        },
        {
            "name": "plc1/machine/status/runners/temperature",
            "type": "int16le",
            "offset": 26,
            "length": 6
        },
        {
            "name": "plc1/machine/status/runners/speed",
            "type": "int16be",
            "offset": 26,
            "length": 6
        },
        {
            "name": "plc1/machine/status/flags",
            "type": "bool",
            "offset": 0,
            "length": 32
        }
    ]
}

Output...
image

Using same PLC Data Sample...

different specification...

{
    "options": {
        "byteSwap": [
            "swap16"
        ],
        "resultType": "object",
        "singleResult": true,
        "msgProperty": "data"
    },
    "items": [
        {
            "name": "alphabet",
            "type": "string",
            "offset": 0,
            "length": 26
        },
        {
            "name": "single byte pos 4",
            "type": "int",
            "offset": 4
        },
        {
            "name": "bcd equiv",
            "type": "bcd",
            "offset": 4,
            "length": 5
        },
        {
            "name": "Array[6] of int16le",
            "type": "int16le",
            "offset": 26,
            "length": 6
        },
        {
            "name": "Array[6] of int16be",
            "type": "int16be",
            "offset": 26,
            "length": 6
        },
        {
            "name": "32 bools",
            "type": "bool",
            "offset": 0,
            "length": 32
        },
        {
            "name": "Array[4] of 16bits",
            "type": "16bit",
            "offset": 0,
            "length": 4
        }
    ]
}

Output...

The built in help


continued

Wrap up

Please feel free to chip in.

  • A nicer name suggestion?
  • better option names?
  • additional functionality?
  • better help info?

final notes

  • i realise a UI editor would improve this (an when I get time I may add that) but for now, I have made it so the specification can be typed in or passed in (via msg, flow, global etc) - at some point in the future, these can/will be mapped to a UI editor
2 Likes

I think that is a brilliant idea. Though I think we will still spend a fair amount of time trying to guide users to the correct options.

I agree so I intend on including built in import examples.

Should probably make reference to that in the built in help.

Indeed however it should reduce the need to teach people how to code js around the buffer object in a function node right?

When a UI is created (to simplify the specification) that should help somewhat BUT data conversion is a necessary evil and not something to easily abstract.

Any other thoughts?

So how does it compare to node-red-contrib-binary that also provide templates to pack and unpack buffers ? That allows bit wise packing as well.

I tried the binary package Dave. There are a number of things they do differently.

Issue with binary node (see end of post for more details)

  • Couldn't get individual bits to work (always returned wrong values (possibly my lack of understanding))
  • No BCD support
  • Can't seem to rewind or restart the buffer or random access (might be my lack of understanding)
  • A few other things it can't do (i forget now - was over 1 year ago I evaluated it) & some issues (like not putting value against name if only one item in specification (mentioned in a thread on this forum somewhere, you pointed to the underlying lib as the issue) but mostly these are due to the 4 year old packet lib

What i like about node-red-contrib-binary...

  • Simpler specification

What this does that binary doesn't (as far as I can see)...

  • specification is dynamic & can be sent in as a msg/flow/global property - permitting fully dynamic setup (e.g. via a dashboard)
  • output multiple as well as single msg style output
  • one or more byte swaps of 16, 32 or 64 (sometimes necessary to do multiple swaps) on the buffer (before accessing any LE or BE functions)
  • output to any msg property
  • input data can come from not only a msg property but also a flow or global property
  • accepts input of 16bit array (common plc data format) - something that buffer should really handle - but it doesnt do this simply
  • can specify the same offset many times
  • specification format permits random access (e.g. no need for any skips when accessing only first and last elements)

All this in one node designed around simplification of data conversions that currently (as far as I know) would require a bunch of (sometimes) specialised operations (function nodes, buffer manipulation etc) both before and after to achieve things like the 2 examples I posted above.

Those examples (in the original post) were created like this - one node...

image

A good use is when used to take PLC data & send individual messages as topic/payload to an MQTT broker for example - it greatly reduces the coding necessary to process the raw DATA


**** node-red-contrib-binary issues I mentioned****
NOTE: Some of these might be my lack of understanding

[{"id":"1f6f379a.1832d8","type":"binary","z":"c70ba4a4.e7fb58","name":"bits working if x8=>empty added but values are incorrect","property":"payload","pattern":"b8{b1 => bit0, b1 => bit1, b1 => bit2, x5}, x8 => empty","x":1050,"y":1060,"wires":[["9ab3533.89b20b"]]},{"id":"b5d33983.eaafd8","type":"inject","z":"c70ba4a4.e7fb58","name":"inject [0010,0010]   ( [2,2] )","topic":"","payload":"[2,2]","payloadType":"bin","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":690,"y":1000,"wires":[["1f6f379a.1832d8","3db42fba.fb599","190a229a.ccd43d"]]},{"id":"9ab3533.89b20b","type":"debug","z":"c70ba4a4.e7fb58","name":"3rd","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":1150,"y":1100,"wires":[]},{"id":"3db42fba.fb599","type":"binary","z":"c70ba4a4.e7fb58","name":"1st byte named - not working","property":"payload","pattern":"b8 => fisrt_byte","x":960,"y":940,"wires":[["d58986b4.7ed988"]]},{"id":"190a229a.ccd43d","type":"binary","z":"c70ba4a4.e7fb58","name":"bits - not working","property":"payload","pattern":"b8{b1 => bit0, b1 => bit1, b1 => bit2, x5}","x":930,"y":1000,"wires":[["96a9d250.79ad5"]]},{"id":"96a9d250.79ad5","type":"debug","z":"c70ba4a4.e7fb58","name":"2nd","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":1150,"y":1000,"wires":[]},{"id":"d58986b4.7ed988","type":"debug","z":"c70ba4a4.e7fb58","name":"1st","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":1150,"y":940,"wires":[]}]

Looks good - couple of comments on comments ...

The author of the packet library is back working on it so there could be a big bump in functionality. any day now... https://github.com/bigeasy/packet (or not)

You say input data can come from not only a msg property but also a flow or global property - hopefully you only mean the (fairly static) template part - It is an anti-pattern to pass live data not via the msg.

Re name - as this is not a generic (in the widest sense) data parser - I would add in something about binary or buffer or something.

Thanks for taking the time Dave.

Would you mind explaining this part please? (With an example if you have time?)

There are times (due to other nodes lack of capabilities) I have had to put some data into a flow or global first - (fully understanding that it wasn't ideal due to concurrency etc) - but sometimes the only way (that I could see). An example of this is perhaps a cache - i might need to have a cache to process at a later time & I may need to replay that at some point. By not permitting flow or global as a data source, I would have to do change nodes. (I might not be thinking this through entirely)

If I am wrong or dont get your point, please explain a little further (I really would like to better understand the anti-pattern POV please)

Anyone want to assist in naming?

  • node-red-contrib-dataparser
  • node-red-contrib-data-parser
  • node-red-contrib-buffer-parser
  • node-red-contrib-buffer-processor
  • node-red-contrib-buffer-ops
  • other

0 voters

The whole point of Node-RED is to (hopefully) be able to understand where the data to going by following the wires - aka the flow of the data. So if I get something from my plc input node it should flow down the wire to your parser node... And then it is the act of that msg arriving that triggers the node to do something (and pass it on as a msg). Flow and Global context provide a way to park things for later. If you send your data to context - a node won't get triggered when it changes - only a msg can do that.
And yes the change node is the way into and out of context within a flow.

Also when if/when we ever get to distributed flows - it would only be the msg that connects things - contexts would be local to a domain.

This topic was automatically closed after 60 days. New replies are no longer allowed.