Buffer parser node drops all incoming data?

General
1

I send a message into buffer parser node. It sends one or more messages out. In the process, every data in the original msg is lost. I need some meta data for continuity purposes like timestamp etc. Nodes that discard all incoming data makes node red difficult. What's the easiest cleanest way of keeping data around such nodes?

Group join? How will that work if the node (like buffer parser) spits out more than one msg per incoming msg?

Context? Could work if buffer parser node only handles one msg at a time (not all nodes do).

Has someone made a generic flow to keep data like this?

2

It would be see worth submitting an issue against the node, it is actively maintained and might get enhanced to pass through unused properties.

3

it depends how you are using it / how it is set-up.

As far as I can remember, outputting multiple will NOT (deliberately) contain all of the original (since that would be superfluous duplication)

Please provide a small reproducible demo flow (with self contained data (i.e. use inject or template node)) and describe what your overarching goal is. I will try to recommend an alternative or better direction. If necessary, I may even be tempted to update it to based on how strong the argument is.

4

Sorry I don't have a good flow at the moment to illustrate this. Can attempt to describe with words instead:

Most typical case for me is generating statistics to be stored in database. Ie. to show on dashboard / graph, or do whatever logic later somewhere else. To generate this data, use a timer (for example some cron inject node) to send a msg at specific intervals. This will be the trigger to fetch some data to be stored as statistics / timeseries data.

At that exact trigger time I add msg.ts which is epoch / unix timestamp (ms since 1970). This is the timestamp for the data. Ideally this is done at the start of the flow, on trigger.

Then, the msg travels downstream the flow. In this process, we can encounter delays. Network delays, error-handling, x amounts of retries for example. I'm currently working on a modbus node which frequently fails due to timeout or complains about closed ports. So I add a retry mechanic.

Then, we use buffer parser node to extract data from modbus node. Now, that very specific msg.ts timestamp is lost. So we can re-apply it, right? But alas, time could potentially be 1 minute later. So will conflict with the next trigger.

Now this exact case could be handled by truncating ts after buffer node (ie. set minutes and/or seconds to 0 depending on the interval). But it's just extra hassle to handle this at a later point, when it is much easier and cleaner to deal with at start. I want a ts which is set precisely on trigger time and not set later when affected by various delays in the flow.

This is just one usecase. In general, I'd like to have the freedom to preserve any meta-data to support the possibility of some continuity when messages travel down the flow.

5

Same applies:

For example, you might be using a mode of the buffer parser unnecessarily.

like: inject (fake modbus data) -> change (add ts) -> buffer parser -> debug

6

This is how buffer parser is set up:

image
image884×772 15.6 KB

This is part of the flow:

image
image1699×450 73 KB

Sorry it is messy, WIP.

7

I have played around with inject and buffer parser as you suggest, and it always removed all data on output. Unfortunately I deleted it. Will make a new example.

8

Here is an example:

[
    {
        "id": "5e1ea70e4ed1df8a",
        "type": "inject",
        "z": "02f78812994b35d8",
        "name": "",
        "props": [],
        "repeat": "",
        "crontab": "",
        "once": false,
        "onceDelay": 0.1,
        "topic": "",
        "x": 850,
        "y": 1540,
        "wires": [
            [
                "43e2a9f5ea06e926"
            ]
        ]
    },
    {
        "id": "43e2a9f5ea06e926",
        "type": "function",
        "z": "02f78812994b35d8",
        "name": "add ts",
        "func": "const now = new Date();\nnow.setSeconds(0,0);\nnow.setMinutes(0);\nmsg.ts = now.getTime();\nreturn msg;",
        "outputs": 1,
        "timeout": 0,
        "noerr": 0,
        "initialize": "",
        "finalize": "",
        "libs": [],
        "x": 970,
        "y": 1540,
        "wires": [
            [
                "4f102ab31c47bebf"
            ]
        ]
    },
    {
        "id": "f927b58ed38a3e4e",
        "type": "buffer-parser",
        "z": "02f78812994b35d8",
        "name": "power",
        "data": "payload",
        "dataType": "msg",
        "specification": "spec",
        "specificationType": "ui",
        "items": [
            {
                "type": "floatbe",
                "name": "active_power_1",
                "offset": 0,
                "length": 1,
                "offsetbit": 0,
                "scale": "1",
                "mask": ""
            },
            {
                "type": "floatbe",
                "name": "active_power_2",
                "offset": 4,
                "length": 1,
                "offsetbit": 0,
                "scale": "1",
                "mask": ""
            },
            {
                "type": "floatbe",
                "name": "active_power_3",
                "offset": 8,
                "length": 1,
                "offsetbit": 0,
                "scale": "1",
                "mask": ""
            },
            {
                "type": "floatbe",
                "name": "power_consumption",
                "offset": 12,
                "length": 1,
                "offsetbit": 0,
                "scale": "1",
                "mask": ""
            }
        ],
        "swap1": "",
        "swap2": "",
        "swap3": "",
        "swap1Type": "swap",
        "swap2Type": "swap",
        "swap3Type": "swap",
        "msgProperty": "payload",
        "msgPropertyType": "str",
        "resultType": "value",
        "resultTypeType": "return",
        "multipleResult": true,
        "fanOutMultipleResult": false,
        "setTopic": true,
        "outputs": 1,
        "x": 1230,
        "y": 1540,
        "wires": [
            [
                "51f76b7beb2bfac4"
            ]
        ]
    },
    {
        "id": "51f76b7beb2bfac4",
        "type": "debug",
        "z": "02f78812994b35d8",
        "name": "debug",
        "active": true,
        "tosidebar": true,
        "console": false,
        "tostatus": false,
        "complete": "true",
        "targetType": "full",
        "statusVal": "",
        "statusType": "auto",
        "x": 1350,
        "y": 1540,
        "wires": []
    },
    {
        "id": "4f102ab31c47bebf",
        "type": "function",
        "z": "02f78812994b35d8",
        "name": "modbus",
        "func": "const ts = msg.ts;\nmsg = {\n  topic: \"power\",\n  payload: [\n    16572,\n    48928,\n    16570,\n    34562,\n    16551,\n    58778,\n    16775,\n    51951,\n    65472,\n    0,\n    65472,\n    0,\n    65472,\n    0,\n    16640,\n    22210,\n    65472,\n    0,\n    65472,\n    0,\n    65472,\n    0,\n    16790,\n    12527\n  ],\n  \"modbusRequest\": {\n    \"unitid\": null,\n    \"fc\": 3,\n    \"address\": 3053,\n    \"quantity\": 24,\n    \"emptyMsgOnFail\": false,\n    \"keepMsgProperties\": true,\n    \"messageId\": \"675c0f83747ecdc714a345fa\"\n  },\n  \"responseBuffer\": {\n    \"data\": [\n      16572,\n      48928,\n      16570,\n      34562,\n      16551,\n      58778,\n      16775,\n      51951,\n      65472,\n      0,\n      65472,\n      0,\n      65472,\n      0,\n      16640,\n      22210,\n      65472,\n      0,\n      65472,\n      0,\n      65472,\n      0,\n      16790,\n      12527\n    ],\n    \"buffer\": [\n      64,\n      188,\n      191,\n      32,\n      64,\n      186,\n      135,\n      2,\n      64,\n      167,\n      229,\n      154,\n      65,\n      135,\n      202,\n      239,\n      255,\n      192,\n      0,\n      0,\n      255,\n      192,\n      0,\n      0,\n      255,\n      192,\n      0,\n      0,\n      65,\n      0,\n      86,\n      194,\n      255,\n      192,\n      0,\n      0,\n      255,\n      192,\n      0,\n      0,\n      255,\n      192,\n      0,\n      0,\n      65,\n      150,\n      48,\n      239\n    ]\n  }\n};\n\nmsg.ts = ts;\nreturn msg;",
        "outputs": 1,
        "timeout": 0,
        "noerr": 0,
        "initialize": "",
        "finalize": "",
        "libs": [],
        "x": 1100,
        "y": 1540,
        "wires": [
            [
                "f927b58ed38a3e4e"
            ]
        ]
    }
]
9

Oh - can send as single message instead! And it then passes on incoming data - great! Problem solved haha.

Thanks for pushing me to look once more at the options.

10

As I said earlier, the node does not copy all props when outputing multiple (to avoid expensive duplication)

There are several ways to solve this, this is the most straightforward:

image
image976×190 28.1 KB

11

You mean you are intentionally saving the data with the wrong timestamp, rather than the time it was actually read? That seems very odd to me.

12

Yup. The data goes into Thingsboard database and all sorts of aggregations are wrong if values are not having the exact right timestamp. Further, having precise timestamps removes possibility of duplication. I look at ts as sort of the "id" of the job the msg represents. Each job now has ts working as unique id.

13

It is all a matter of what is defined as the right timestamp. It sounds to me as if the
Thingsboard database tools are somewhat deficient if you have to lie about the timestamp in order to make it work. Influxdb and grafana, for example, have no such problems. However, if that is what one is stuck with then one has to make the best use of it one can.

14

It's a simplification which got some pros and cons. Pros is handling is super easy. Con is you may get some inaccuracies in terms of seconds (for minutely data) or minutes (in terms of hourly data) depending on how it is implemented.

It's not really a matter of the db itself, I'm sure whatever database is used in thingsboard can handle this and much more (postgres and cassandra?). But when using Thingsboard cloud, we have zero direct access to any databases beneath. For example we're stuck with widget interface for time filter, and Thingsboard has no support for showing future data (!), for example weather forecasts or electricity prices for the next day. Again, it's pros and cons. We can't do everything we'd like. But at the same time we don't have to build that system from scratch, or go through all configs and settings.

We got much bigger problem at the moment in terms of data disappearing due to faulty connections etc, and all kinds of ways error handling is implemented (comparing http result code in output msg, modbus throwing errors, bacnet etc.).