I can access my heating system via RS232 and it will spit out some sensors' data. So I wrote a python script, which reads the values out periodically and sends them via MQTT in a JSON.
Nodered then analyzes the JSON and sends the values in separate topics.
Now, my heating system has a bug (which won't get fixed, since the hardware is kinda discontinued from the manufacturer) and that bug randomly sends unrealistic values. These values are mostly within "range", as the temperature can be reached, but just not at those "drops" or "spikes" (the abnormalities come in both extremes). I already ignore really unrealistic values like >120°C for (compressed) water. Or temperature below zero.
But where I am out of ideas is how can I just discard (or send the previous data) with this in mind:
I want the green spikes to still be counted and the red "known bugs" should be ignored or repeat the previous value.
For context: the readout from the heating system is every 2minutes and if for some reason one of my three heating possibilities start heating (it's solar thermic, wood furnace and/or gas heater) within two minutes there could be a certain spike in temperature, which could seem like the bug (green vs. red)
I tried with "block unless value change within 15%", but this still blocks the green spikes, and then there's no values, unless the temperature readings are within 15% again, which could be hours...
So I'm out of ideas how to handle that, is there anybody around with a similar issue and can point me to the right direction?
You could easily, of course, do that all in node-red
I couple of possible approaches come to mind.
The easiest approach would be to preserve the previous data point and compare the current point with the previous and pass the previous if the current is less than a set % difference. You could, of course, do that in the Python script which would be best because then your actual data will be reasonably clean. But it is easily done in NR as well, probably best done via a function node and use of a context variable. Though noting that the context variable will need to be an object that stores the last value against the sensor ID. e.g. { "sensor1": 24, "sensor2": 32, ... }.
In a similar approach. Looking at the data, it appears as though it is unlikely that you would get spikes on all inputs at the same time. So potentially, you could compare values against the other sensors and see if just 1 value drops but the other(s) don't. For that, you would use a similar object context variable keeping the latest values for each sensor.
You could, of course, also combine the two if you really needed to.
The final approach I can think of would be to smooth the data over multiple sensor values. But unless you are getting values well under a minute apart, I'm not sure that is really the best approach.
Those approaches pre-suppose that your sensor values are coming at a fast enough rate that having a gap wouldn't be an issue and that the interval isn't so large that a significant change of environment would be expected anyway.
oh! I guess, there was some addition to RBE since i set up my environment:
So, I'll just test this "last input value". If it works, it will eventually skip one "real" value and I'd get the increase/decrease like two minute later, which is ok for me. But if it's the bug, it'll just skip it...
