Trying to understand the filter to remove some outlier values

I run Tellstick Duos, and sometimes they give crazy values on my Oregon THGR222 sensors. So I have been experimenting for a few hours now without understanding the filter node. I'm experimenting with only two sensors, so I can see that it works before I try with more. In I have the ID in the topic, like Tellstick-Z-Wave-Tellstick/12/humidity/state, with the payload array, where payload[0].temperature is what I'm interested in. First I use an exec node to make the ID's (10 & 12) into the topic of the message and the temperature into the only payload, as a number. Then I send that into the Filter node, set to filter out anything that's more than 10 percent different from the previous node. The weird thing is that it works for one node, but not for the other. 12 has a weird value every ten minutes or so, and that is filtered. But not 10. I have set it to "Apply mode separately for each topic".

Another thing is if I restart NR and it gets the outlier value first. I have no idea how to recover from that. I know that the correct values are at least five times as often as the wrong value. But I can't use the outlier node, since I can't make one flow for each sensor. Suggestions would be appreciated. Here's the part of the flow that is relevant:

[
    {
        "id": "0e9896deee4b0f81",
        "type": "switch",
        "z": "b61f4f80.052e5",
        "name": "Pass on sensor 10 & 12",
        "property": "topic",
        "propertyType": "msg",
        "rules": [
            {
                "t": "cont",
                "v": "12",
                "vt": "str"
            },
            {
                "t": "cont",
                "v": "10",
                "vt": "str"
            },
            {
                "t": "else"
            }
        ],
        "checkall": "true",
        "repair": false,
        "outputs": 3,
        "x": 530,
        "y": 240,
        "wires": [
            [
                "b8b69540deb8c1ae"
            ],
            [
                "b8b69540deb8c1ae"
            ],
            []
        ]
    },
    {
        "id": "b8b69540deb8c1ae",
        "type": "function",
        "z": "b61f4f80.052e5",
        "name": "Temp to payload",
        "func": "var id = msg.topic.split('/')[1];\nvar tmp = msg.payload[0].temperature;\nmsg.topic =  id;\nmsg.payload =  Number(tmp);\nreturn msg;",
        "outputs": 1,
        "timeout": "",
        "noerr": 0,
        "initialize": "",
        "finalize": "",
        "libs": [],
        "x": 220,
        "y": 320,
        "wires": [
            [
                "d3723f5d70cd0a9c",
                "ed99bdd73a146ccb"
            ]
        ]
    },
    {
        "id": "d3723f5d70cd0a9c",
        "type": "debug",
        "z": "b61f4f80.052e5",
        "name": "Before filter node",
        "active": true,
        "tosidebar": true,
        "console": false,
        "tostatus": false,
        "complete": "payload",
        "targetType": "msg",
        "statusVal": "",
        "statusType": "auto",
        "x": 490,
        "y": 320,
        "wires": []
    },
    {
        "id": "ed99bdd73a146ccb",
        "type": "rbe",
        "z": "b61f4f80.052e5",
        "name": "Remove outliers",
        "func": "narrowband",
        "gap": "10",
        "start": "",
        "inout": "out",
        "septopics": true,
        "property": "payload",
        "topi": "topic",
        "x": 480,
        "y": 360,
        "wires": [
            [
                "51cdc257669c334b"
            ]
        ]
    },
    {
        "id": "51cdc257669c334b",
        "type": "debug",
        "z": "b61f4f80.052e5",
        "name": "After filter node",
        "active": true,
        "tosidebar": true,
        "console": false,
        "tostatus": false,
        "complete": "payload",
        "targetType": "msg",
        "statusVal": "",
        "statusType": "auto",
        "x": 700,
        "y": 360,
        "wires": []
    }
]

It may help if you included a couple of complete messages received.

I sort of get what it is you are doing, but also not quite enough to be able to help at this point.

Thought/suggestion:

At the start (ie: after deploy) have an inject node with a nominal payload (and topic, etc) set to what you want and inject it.

I don't even remember why I didn't like simpler smoothing options but I made an outlier eliminator some day. There will be no arguments about if it is better or worse than any other such filters. Best will be to have reliable sensors. But as I'm lazy and some sensors are hard to access to replace ...

It uses link call in flow, every data stream must have different topic.

The example flow sends original and filtered data to chart. So the outliers make peaks but filtered data is smooth.

[{"id":"4bed14e387f0d1a7","type":"function","z":"f8b8c963bdc117f5","name":"outlier detector","func":"let detectors = context.get('detectors') || []\nlet detector = detectors.find(d => d.topic == msg.topic)\nif (!detector) {\n    detector = { topic: msg.topic, storage: [msg.payload] }\n    detectors.push(detector)\n} else {\n    detector.storage.push(msg.payload)\n}\nwhile (detector.storage.length > 3) {\n    detector.storage.shift()\n}\ncontext.set(\"detectors\", detectors);\n\nconst max = Math.max(...detector.storage)\nconst min = Math.min(...detector.storage)\nconst average = detector.storage.reduce((a, b) => a + b, 0) / detector.storage.length;\nconst last = detector.storage[detector.storage.length - 1]\nconst distanceTo = (val) => { return Math.abs(average - val) }\nif (distanceTo(min) <= distanceTo(max)) {\n    if (distanceTo(last) <= distanceTo(min)) {\n        msg.payload = last\n    }\n    else {\n        msg.payload = min\n    }\n}\nelse {\n    if (distanceTo(last) <= distanceTo(max)) {\n        msg.payload = last\n    }\n    else {\n        msg.payload = max\n    }\n}\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":580,"y":1640,"wires":[["66dd5e66b3a02d3e"]]},{"id":"d8944ab4830af2ac","type":"link in","z":"f8b8c963bdc117f5","name":"to-detector","links":[],"x":465,"y":1640,"wires":[["4bed14e387f0d1a7"]]},{"id":"66dd5e66b3a02d3e","type":"link out","z":"f8b8c963bdc117f5","name":"from-detector","mode":"return","links":[],"x":695,"y":1640,"wires":[]},{"id":"77ed00243b4c060a","type":"inject","z":"f8b8c963bdc117f5","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"1","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":150,"y":1840,"wires":[["ab0dd5319eaac7af","97961351f40e3e4b"]]},{"id":"ab0dd5319eaac7af","type":"function","z":"f8b8c963bdc117f5","name":"data-1 with outlier","func":"const arr = [17,17,18,19,18,17,17,117,17,18,19,20,21,22,21,22,22,5,22,23,22,21,22,21,20,19,18,17]\n\nlet idx = context.get('idx') || 0\nidx++;\nif(idx > arr.length -1){\n    idx = 0\n}\ncontext.set('idx',idx)\n\nmsg.payload = arr[idx]\nmsg.topic = 'data-1'\n\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":350,"y":1820,"wires":[["b4ca8f0bdafc1829","18abbbb48008783f"]]},{"id":"b4ca8f0bdafc1829","type":"link call","z":"f8b8c963bdc117f5","name":"filter","links":["d8944ab4830af2ac"],"linkType":"static","timeout":"30","x":570,"y":1740,"wires":[["98bb56f4c53e73ea"]]},{"id":"97961351f40e3e4b","type":"function","z":"f8b8c963bdc117f5","name":"data-2 with outlier","func":"const arr = [-17,-17,-18,-19,-18,-17,-17,-117,-17,-18,-19,-20,21,-22,-21,-22,-22,-5,-22,-23,-22,-21,-22,-21,-20,-19,-18,-17]\n\nlet idx = context.get('idx') || 0\nidx++;\nif(idx > arr.length -1){\n    idx = 0\n}\ncontext.set('idx',idx)\n\nmsg.payload = arr[idx]\nmsg.topic = 'data-2'\n\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":350,"y":1860,"wires":[["b4ca8f0bdafc1829","18abbbb48008783f"]]},{"id":"18abbbb48008783f","type":"ui_chart","z":"f8b8c963bdc117f5","name":"","group":"55888982bdd0637f","order":4,"width":0,"height":0,"label":"chart","chartType":"line","legend":"false","xformat":"HH:mm:ss","interpolate":"linear","nodata":"","dot":false,"ymin":"","ymax":"","removeOlder":1,"removeOlderPoints":"","removeOlderUnit":"60","cutout":0,"useOneColor":false,"useUTC":false,"colors":["#1f77b4","#aec7e8","#ff7f0e","#2ca02c","#98df8a","#d62728","#ff9896","#9467bd","#c5b0d5"],"outputs":1,"useDifferentColor":false,"className":"","x":890,"y":1840,"wires":[[]]},{"id":"98bb56f4c53e73ea","type":"change","z":"f8b8c963bdc117f5","name":"change topic","rules":[{"t":"change","p":"topic","pt":"msg","from":"-1","fromt":"str","to":"-1-filtered","tot":"str"},{"t":"change","p":"topic","pt":"msg","from":"-2","fromt":"str","to":"-2-fitered","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":730,"y":1740,"wires":[["18abbbb48008783f"]]},{"id":"86c4bc0b1fe89727","type":"inject","z":"f8b8c963bdc117f5","name":"reset","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"[]","payloadType":"json","x":730,"y":1900,"wires":[["18abbbb48008783f"]]},{"id":"55888982bdd0637f","type":"ui_group","name":"1. group","tab":"62083694d0eab7ca","order":1,"disp":true,"width":"6","collapse":false,"className":""},{"id":"62083694d0eab7ca","type":"ui_tab","name":"Home","icon":"dashboard","order":1,"disabled":false,"hidden":false}]

@Trying_to_learn Thank you, but I'm afraid doing an inject is quite impossible since these values come from temperature sensors. I would not know what value to inject the first time, that depends on the temperature in the room. But here is a view of one of the sensors:

The 9.2 temps are correct. Then suddenly it throws a totally random temp, 16.1 and 24.3. The pattern I see is that it also sends a message without a temp in the same burst. I hadn't noticed that before. So I am starting to think that I could add a two second delay before sending the temps on MQTT, put in a switch node that looks for the NaN and then send a message reset when that comes, since the waiting temperature will be wrong.

@hotNipi Thank you! I will look at that if the attempt at using a message reset does not work.

@hotNipi I don't think your setup will work, unfortunately. I have the ID of the temperature sensors in a totally different part of my system (EventGhost, to be precise - the script I'm using has served me for ten years and it would take far too many hours to change), and if I'm not mistaken your outlier detector needs the ID's in use coded in, right?

Weird, but I think I see the system in the weirdness. This seems to happen when the Tellstick gets what it interprets as either two temperature readings in the same array or no temperature, only humidity. So what I need is to filter out those instances. I can filter the number of arrays on a switch by using ($count(payload))=1 in the switch, but that will filter out the sensors I have that are pure temp sensors, not temp and humidity. So I need to find out how to compare two array paths and block if there are two temperature:. Then I can send a message reset for those values. The temps around 9.3 are correct:

Never mind, it seems like the problem was in another part of the flow. I combine temp and humidity on those who has both. But sometimes it seems that the humidity comes too late, so it mixes up two temps. I need to solve it that way. Thanks for the help and suggestions! :+1:

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.