Trigger Node handling - limitations?

General
1

Hi All, we are using the trigger node as part of a watchdog flow to provide an alert (via webhook) when sensor data is not received after X minutes.

We are new to NodeRed and are learning as we go.
Originally we had a 'watchdog' flow (attached below) per sensor but as we started scaling up we started using the 'Handling' option on the unique sensor ID (devEUI) for the trigger node so we have a single watchdog flow for multiple sensors of the same type.
This appears to work as expected but as we continue to scale up we will be looking at this flow potentially monitoring hundreds of sensors. As I understand it, this will involve hundreds of counters running simultaneously and in this case being reset (via change node) every ~15 minutes providing data is present.

Would you have any performance concerns with this approach or other options/solutions that we've not yet considered for monitoring frequency of data?

Hopefully I have provided sufficient detail to understand our requirements but if not I'm happy to elaborate further.
Data is coming via MQTT, decode is either via a function node or directly on the gateway. The main flow also writes to DB, provides dashboarding, alerts on out of spec data and in some cases will provide some cause and effect actions.

Thanks in advance

Matt


[
    {
        "id": "9cb3c40960cdf070",
        "type": "group",
        "z": "8ff88037d9bc958d",
        "style": {
            "stroke": "#999999",
            "stroke-opacity": "1",
            "fill": "none",
            "fill-opacity": "1",
            "label": true,
            "label-position": "nw",
            "color": "#a4a4a4"
        },
        "nodes": [
            "e8a94db3fb960e72",
            "70ce5cc2692fe19a",
            "d50aa1520aa6a42c",
            "9c1bf22685b78818",
            "1a9c5ddc4080e7d5",
            "3e4271fad7812916"
        ],
        "x": 604,
        "y": 439,
        "w": 1092,
        "h": 202
    },
    {
        "id": "e8a94db3fb960e72",
        "type": "debug",
        "z": "8ff88037d9bc958d",
        "g": "9cb3c40960cdf070",
        "name": "debug 192",
        "active": false,
        "tosidebar": true,
        "console": false,
        "tostatus": false,
        "complete": "false",
        "statusVal": "",
        "statusType": "auto",
        "x": 1230,
        "y": 480,
        "wires": []
    },
    {
        "id": "70ce5cc2692fe19a",
        "type": "trigger",
        "z": "8ff88037d9bc958d",
        "g": "9cb3c40960cdf070",
        "name": "Watchdog",
        "op1": "",
        "op2": "timeout",
        "op1type": "nul",
        "op2type": "str",
        "duration": "1",
        "extend": true,
        "overrideDelay": false,
        "units": "hr",
        "reset": "",
        "bytopic": "topic",
        "topic": "payload.devEUI",
        "outputs": 1,
        "x": 890,
        "y": 600,
        "wires": [
            [
                "e8a94db3fb960e72",
                "d50aa1520aa6a42c",
                "9c1bf22685b78818"
            ]
        ]
    },
    {
        "id": "d50aa1520aa6a42c",
        "type": "template",
        "z": "8ff88037d9bc958d",
        "g": "9cb3c40960cdf070",
        "name": "No Data warning",
        "field": "payload",
        "fieldType": "msg",
        "format": "handlebars",
        "syntax": "mustache",
        "template": "{\n  \"@context\": \"https://schema.org/extensions\",\n  \"@type\": \"MessageCard\",\n  \"themeColor\": \"0072C6\",\n  \"title\": \"Milesight Fridge Timeout\",\n  \"subject\": \"Milesight Fridge Timeout\",\n  \"text\": \"Milesight Fridge Timeout has not received data for 1 hr\"\n}",
        "output": "str",
        "x": 1280,
        "y": 560,
        "wires": [
            [
                "3e4271fad7812916"
            ]
        ]
    },
    {
        "id": "9c1bf22685b78818",
        "type": "delay",
        "z": "8ff88037d9bc958d",
        "g": "9cb3c40960cdf070",
        "name": "",
        "pauseType": "delay",
        "timeout": "10",
        "timeoutUnits": "seconds",
        "rate": "1",
        "nbRateUnits": "1",
        "rateUnits": "second",
        "randomFirst": "1",
        "randomLast": "5",
        "randomUnits": "seconds",
        "drop": false,
        "allowrate": false,
        "outputs": 1,
        "x": 930,
        "y": 520,
        "wires": [
            [
                "1a9c5ddc4080e7d5"
            ]
        ]
    },
    {
        "id": "1a9c5ddc4080e7d5",
        "type": "change",
        "z": "8ff88037d9bc958d",
        "g": "9cb3c40960cdf070",
        "name": "",
        "rules": [
            {
                "t": "set",
                "p": "payload",
                "pt": "msg",
                "to": "0",
                "tot": "str"
            }
        ],
        "action": "",
        "property": "",
        "from": "",
        "to": "",
        "reg": false,
        "x": 710,
        "y": 520,
        "wires": [
            [
                "70ce5cc2692fe19a"
            ]
        ]
    },
    {
        "id": "3e4271fad7812916",
        "type": "http request",
        "z": "8ff88037d9bc958d",
        "g": "9cb3c40960cdf070",
        "name": "Teams Webhook Alerts",
        "method": "POST",
        "ret": "txt",
        "paytoqs": "ignore",
        "url": "https://webhook.office.com",
        "tls": "",
        "persist": false,
        "proxy": "",
        "insecureHTTPParser": false,
        "authType": "",
        "senderr": false,
        "headers": [],
        "x": 1570,
        "y": 560,
        "wires": [
            []
        ]
    }
]

NodeRed_Watchdog
NodeRed_Watchdog1703Ă—406 81.5 KB

2

You don't show how this little flow gets initiated. However: infinite loops have the potential to be troublesome.

What about this?

image
image815Ă—123 8.37 KB

The trigger node can be set to handle each msg.topic or other property separately so as long as your sensor data contains it's source id somewhere you only need one trigger node for all of your sensors.
The template clearly needs changing to be able to handle warnings for multiple devices.

3

It may pay at some point to query the db for any sensor where latest timestamp is greater than X minutes old. This may be more efficient as you scale up.

1 Like
4

Thanks Jbudd,

The flow is initiated either from the output of the payload decode function or an SQL INSERT function into the "Watchdog" trigger node.
The decoded object always contains a devEUI, MAC, serial number we can use for identification.

With my flow I had to use the change node to reset the counter on receipt of a new message. Your flow does not contain this so would the alert not be sent after the 1hour period configured in the trigger?
Guess I must be missing something here?

Matt

5

Try this flow:

image
image1196Ă—127 14.8 KB

[{"id":"a42431d234597e36","type":"trigger","z":"271fea779d53f1b1","name":"","op1":"","op2":"","op1type":"nul","op2type":"payl","duration":"10","extend":true,"overrideDelay":false,"units":"s","reset":"","bytopic":"topic","topic":"payload.id","outputs":1,"x":330,"y":80,"wires":[["2b2f3ad6bf57f2af"]]},{"id":"8d72200f4fe66170","type":"inject","z":"271fea779d53f1b1","name":"Inject after 4 sec","props":[{"p":"payload"},{"p":"timestamp","v":"","vt":"date"}],"repeat":"","crontab":"","once":true,"onceDelay":"4","topic":"","payload":"{\"id\": \"ABC\", \"Temperature\": 42}","payloadType":"json","x":130,"y":100,"wires":[["a42431d234597e36"]]},{"id":"332e5a96c2ef5471","type":"inject","z":"271fea779d53f1b1","name":"Inject at start","props":[{"p":"payload"},{"p":"timestamp","v":"","vt":"date"}],"repeat":"","crontab":"","once":true,"onceDelay":0.1,"topic":"","payload":"{\"id\": \"DEF\", \"Temperature\": 21}","payloadType":"json","x":110,"y":60,"wires":[["a42431d234597e36"]]},{"id":"2b2f3ad6bf57f2af","type":"template","z":"271fea779d53f1b1","name":"Construct alert message","field":"payload","fieldType":"msg","format":"handlebars","syntax":"mustache","template":"{{payload.id}} has not been heard of for at least 10 seconds!","output":"str","x":550,"y":80,"wires":[["3775f6c8c01f4e7c"]]},{"id":"3775f6c8c01f4e7c","type":"debug","z":"271fea779d53f1b1","name":"debug 195","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":770,"y":80,"wires":[]}]

It has :

  • Send nothing
  • Then wait for 10 seconds
  • Extend the delay if a new message arrives
  • Then send the latest message object
  • Have a separate countdown timer for each msg.payload.id.

Press the input buttons a few times. 10 seconds after you stop, the alert is sent.

1 Like
6

Even if it adds another moving part in your environment, I'd go for using redis for this kind of requirement.

Keeping potentially thousands of counters in NodeRed may work, but, I am not sure if it is the right tool for this particular job. NodeRed is the right tool for many, many things though !!

With redis, it is (reasonably) easy to write in the database eg. key=my_unique_sensor_id, value=timestamp with an expiry time of X minutes when you get data from that sensor_id.
When you receive new data for that same sensor_id, you update the key/value.
This expiry feature is built-in redis.

Then, you configure redis to send expiry notice. In NodeRed you have a flow triggered by this expiry notice and voilĂ . You'll have an alert giving you the silent sensor_id.

this Redis — Getting Notified When a Key is Expired or Changed | by Aditya Rama | Nerd For Tech | Medium gives you a quick overview of the expiry feature.

1 Like