Anyone done much work with the Smooth Node?

General
1

Guys,

Still moving my heating project forward - currently on my picaxe system which does the temperature measurement, i have a rolling average of 10 temperatures, i do this by testing the incoming values against the average and throwing away any that are more than 20% different from the average - assuming they are bad reads from the DS18B20.

As i move all of this to an Arduino and Node-Red - i want to move all the logic into Node-red.

I have look at the PID nodes - but as this is a very slow moving temperature (hydronic inslab heating) - and a water store of 5000 litres - i do not need or want to adjust anything very minutely or quickly.

I am looking at the Smooth node which seems perfect - just wondering though how it works in terms of the high and low pass filter (these would seem to do my sanity checking portion for any bad sensor reads) - but how do i use them - do i stack a series of smooth nodes in the data stream and pass the message along ?

So

  1. MQTT incoming
  2. Smooth node with hipass filter
  3. SMooth node with lowpass filter
  4. Smooth node with mean

Or is there some other way to clean up the "rogue" values prior to passing them into the smooth node ?

regards

Craig

2

The rbe node may be better for dropping bad values

3

I am smoothing out light levels from a sensor in our hallway. Not sure if this is the best way to do it but it works. The following flow also shows how I worked out the best approach by charting the outputs from different smoothing options. Never quite got round to taking that out again.

[{"id":"7f81f115.807e1","type":"subflow","name":"COMMAND output","info":"### Take a COMMAND topic with the action in the payload. Send to DB & MQTT\n\nFeed in a msg with an appropriate COMMAND topic, e,g, COMMAND/SWITCH01\n\nThe payload must contain everything needed to save to MongoDB & everythng\nneeded to process the command elsewhere.\n\n### Input\n- msg with \n    - topic set to COMMAND/<switch-name> \n    - paload set to command to send (e.g \"On\" or \"Off\")\n\n### Output\n- Each input is sent to MongoDB\n   - Once to a collection matching the topic\n   - Once to a summary collection \"CurrentSwitchStatus\" with one record for each switch\n- Each input is sent to MQTT on COMMAND/#\n   - COMMAND/SWITCHnn message is sent with Retain=true, QoS=1\n   - So when NR restarts, it recieves back the last command and replays is\n   - This ensures all switches are reset according to the last setting BUT it will override any manual sets\n   - Obviously this may result in incorrect settings if a scheduled change event has been missed. (TODO improve scheduler)\n- Output 1 is for debugging the detail output\n- Output 2 is for debugging the summary output\n \n### Expected Input msg Format\n```JSON\n{\n\t\"topic\": \"HARDWARE-IN/TH1/0x3001\",\n\t\"payload\": {\n\t\t\"timestamp\": \"2016-01-09T17:46:17.016Z\",\n\t\t\"topic\": \"TH1/0x3001\",\n\t\t\"inputController\": \"RFX\",\n\t\t\"deviceFamily\": \"TH1\",\n\t\t\"id\": \"0x3001\",\n\t\t\"deviceFeatures\": \"TH\",\n\t\t\n\t\t\"temperature\": { \"value\": 19, \"unit\": \"degC\" },\n\t\t\"humidity\": { \"value\": 50, \"unit\": \"%\", \"status\": \"NORMAL\" },\n\t\t\"status\": { \"rssi\": 4, \"battery\": 9 },\n\t\t\"rssi\": 4,\n\t\t\"battery\": 9,\n\t\t\"Temperature\": 19,\n\t\t\"Humidity\": 50,\n\t\t\"Heat_Index\": 18.3,\n\t\t\"DewPoint\": 8.3\n\t}\n}\n```\n","in":[{"x":100,"y":160,"wires":[{"id":"13c42b13.ec3bd5"}]}],"out":[{"x":644,"y":100,"wires":[{"id":"13c42b13.ec3bd5","port":0}]},{"x":683,"y":209,"wires":[{"id":"13c42b13.ec3bd5","port":1}]}]},{"id":"13c42b13.ec3bd5","type":"function","z":"7f81f115.807e1","name":"Validate Input. 1) Device specific output, 2) Current Status","func":"/***\n * Make sure output message is ready to be recorded in MongoDB\n * INPUT: \n *   From MQTT HARDWARE-IN but for remote controls (and smart switches) only\n * OUTPUTS:\n *   1) Device specific, data is sent to both MongoDB and MQTT\n *   2) Current Status. Data is sent to the CurrentSwitchStatus MongoDB collection only\n ***/\n\nif ( !('topic' in msg) || msg.topic === '' ) {\n    node.warn( 'No topic! Dropping output' );\n    return null;\n}\n\nif ( (msg.topic.split('/'))[0] !== 'COMMAND' ) {\n    node.warn( 'Topic is not COMMAND/# Dropping Output' );\n    return null;\n}\n\nif ( !('collection' in msg) ) {\n    msg.collection  = msg.topic; \n}\n\nif ( !('_id' in msg) ) {\n    msg._id = new Date();\n}\n\n// Record what triggered this command msg if we can\nif ( !('inputHardware' in msg) ) {\n    msg.inputHardware = 'MANUAL_OR_NR';\n}\n\n// -- CHANGED TO A LISTENER ON COMMAND CHANNEL --\n//// Keep current switch status in global memory\n//var cmdStatus = global.get('cmdStatus') || {};\n//cmdStatus[msg.topic] = msg.payload;\n//global.set('cmdStatus', cmdStatus);\n\n// 2nd output is for the Current Switch status\nvar currMsg = {\n    'collection': 'CurrentSwitchStatus',\n    '_id': msg.topic,\n    'state': msg.payload,\n    'lastUpdate': msg._id,\n    'inputHardware': msg.inputHardware\n};\n\nreturn [msg, currMsg];","outputs":"2","noerr":0,"x":360,"y":160,"wires":[["6c18bd26.93e744"],[]]},{"id":"6c18bd26.93e744","type":"mqtt out","z":"7f81f115.807e1","name":"COMMAND/SWITCHnn (On/Off) [retained]","topic":"","qos":"1","retain":"true","broker":"3b2726f7.c4d8da","x":794,"y":154,"wires":[]},{"id":"8d3967a.f72c698","type":"comment","z":"7f81f115.807e1","name":"Take a COMMAND topic with the action in the payload. Send to DB & MQTT","info":"### Take a COMMAND topic with the action in the payload. Send to DB & MQTT\n\nFeed in a msg with an appropriate COMMAND topic, e,g, COMMAND/SWITCH01\n\nThe payload must contain everything needed to save to MongoDB & everythng\nneeded to process the command elsewhere.\n\n### Input\n- msg with \n    - topic set to COMMAND/<switch-name> \n    - paload set to command to send (e.g \"On\" or \"Off\")\n\n### Output\n- Each input is sent to MongoDB\n   - Once to a collection matching the topic\n   - Once to a summary collection \"CurrentSwitchStatus\" with one record for each switch\n- Each input is sent to MQTT on COMMAND/#\n   - COMMAND/SWITCHnn message is sent with Retain=true, QoS=1\n   - So when NR restarts, it recieves back the last command and replays is\n   - This ensures all switches are reset according to the last setting BUT it will override any manual sets\n   - Obviously this may result in incorrect settings if a scheduled change event has been missed. (TODO improve scheduler)\n- Output 1 is for debugging the detail output\n- Output 2 is for debugging the summary output\n \n### Expected Input msg Format\n```JSON\n{\n\t\"topic\": \"HARDWARE-IN/TH1/0x3001\",\n\t\"payload\": {\n\t\t\"timestamp\": \"2016-01-09T17:46:17.016Z\",\n\t\t\"topic\": \"TH1/0x3001\",\n\t\t\"inputController\": \"RFX\",\n\t\t\"deviceFamily\": \"TH1\",\n\t\t\"id\": \"0x3001\",\n\t\t\"deviceFeatures\": \"TH\",\n\t\t\n\t\t\"temperature\": { \"value\": 19, \"unit\": \"degC\" },\n\t\t\"humidity\": { \"value\": 50, \"unit\": \"%\", \"status\": \"NORMAL\" },\n\t\t\"status\": { \"rssi\": 4, \"battery\": 9 },\n\t\t\"rssi\": 4,\n\t\t\"battery\": 9,\n\t\t\"Temperature\": 19,\n\t\t\"Humidity\": 50,\n\t\t\"Heat_Index\": 18.3,\n\t\t\"DewPoint\": 8.3\n\t}\n}\n```\n","x":358.8957824707031,"y":23.888885498046875,"wires":[]},{"id":"e9288d30.e3c06","type":"comment","z":"7f81f115.807e1","name":"TODO: Save command updates to json file","info":"and reload on startup, passing to UI.","x":362,"y":311,"wires":[]},{"id":"a680238d.4782a","type":"comment","z":"a5dcc8f6.5a2338","name":"Switch on in low light but only between 6am and 1am","info":"This monitors light from an ESP8266 based sensor\nplatform containing an I2C light sensor.\n\nThe light output is in Lux.\n\nThe sensor is positioned above a lamp that should\ncome on automatically if the light level falls below\nthat produced by the lamp.\n\nHowever, this should only happen during the day\nand evening, not overnight.","x":411,"y":758,"wires":[]},{"id":"152b0840.102698","type":"function","z":"a5dcc8f6.5a2338","name":"Light & Time control","func":"// Turn on/off a light (by issuing a COMMAND/SWITCHnn MQTT msg)\n// depending on the lux level from a sensor (on/offThresh).\n// Except if the time is between two hours in early AM (start/endTime)\n// Don't repeat send the command (tracks whether light is already on/off)\n//\n// Recommended to feed in Averaged or Low bandwidth filter light\n// levels over about the last 10 minutes. \n// You can use the smooth node for this.\n\n/*\nglobal.set('switchLocations', {\n  \"SWITCH02\" : {location: \"HOME/IN/00/HALL\",     description: \"Hall standard lamp near kitchen\",  type: \"Siemens white remote plug\"},\n});\nglobal.set('lightLevels', {\n  \"HOME/IN/00/HALL\": [15,50]\n});\n*/\n\n// Only check during Day time not Night\nvar home = global.get('home');\nvar dayNight = home.daynight || 'NA';\nif ( dayNight === 'Day'  ) {\n\n    // Which switch are we controlling?\n    var mySwitch = 'SWITCH02';\n    var ctrlSwitch = (global.get('switchLocations'))[mySwitch];\n    var swLocation = ctrlSwitch.location;\n    \n    // ----------------------------------------------- //\n    // SETTINGS\n    // ----------------------------------------------- //\n    var lightLevels = global.get('lightLevels');\n    // What light thresholds do you want?\n    onThresh  = lightLevels[swLocation][0]; // Turn light on if less than this\n    offThresh = lightLevels[swLocation][1]; // Turn light off if greater than this\n    // ----------------------------------------------- //\n    // The topic is the lamp switch we want to control\n    msg.topic = 'COMMAND/' + mySwitch;\n    // ----------------------------------------------- //\n\n    // Comes in from MQTT in string form\n    var val = parseFloat(msg.payload);\n\n    node.status({fill:\"green\",shape:\"ring\",text:'Day: Lux=' + val + ', Threshold On=' + onThresh + ' Off=' + offThresh});\n    \n    // cmdStatus is set by \"COMMAND output\" subflow\n    // it keeps track of current command status\n    // and is indexed by command (MQTT) topic\n    // e.g. cmdStatus = { 'COMMAND/SWITCH02': 'On', 'COMMAND/SWITCH05': 'Off' } \n    var cmdStatus = global.get('cmdStatus') || {};\n    \n    msg.payload = cmdStatus[msg.topic];\n    \n    // Check light thresholds\n    if ( val < onThresh ) {\n        \n        // Light level below on threshold\n        msg.payload = 'On';\n        \n    } else if ( val > offThresh ) {\n        \n        // Light level above off threshold\n        msg.payload = 'Off';\n        \n    } // else light within thresholds, do nothing\n    \n    // If cmd has changed, send msg\n    if ( cmdStatus[msg.topic] !== msg.payload ) {\n        // NB: cmdStatus is updated in \"COMMAND output\" subflow\n        return msg;\n    } // otherwise, cmd not changed, nothing to do\n\n} else {\n    node.status({fill:\"red\",shape:\"dot\",text:\"Night\"});\n} // -- End of time check -- //\n\n// --- End of Light & Time Control --- //\n\n/*\n// Set the start and end hours (24hr clock) that you want IGNORED\n// No commands issued between these times.\n// NB: Assumes both hours are after midnight. If you need\n//     to span midnight, you will need more complex time check.\nvar startIgnore = 0;\nvar endIgnore   = 6;\n// ----------------------------------------------- //\n// These DO have to be all new Date()\n// Don't set start/end to nowTime as they would\n// be the SAME data (nowTime would be changed by setHours)\nvar nowTime = new Date();\nvar startTime = new Date();\nvar endTime = new Date();\nstartTime.setHours(startIgnore,0,0,0);\nendTime.setHours(endIgnore,0,0,0);\n\n// Check if we are within the ignore time\nif ( (nowTime > startTime) && (nowTime < endTime )  ) {\n    \n    // Within the ignore time so do nothing\n\n} else { // Not in ignore time, so process\n    \n} // -- End of time check -- //\n\n*/","outputs":"1","noerr":0,"x":540,"y":872,"wires":[["d82db943.105b68"]]},{"id":"e7462437.3acd28","type":"mqtt in","z":"a5dcc8f6.5a2338","name":"Hall Lux","topic":"LIGHT/HOME/IN/00/HALL/D1M02","qos":"2","broker":"3b2726f7.c4d8da","x":160,"y":872,"wires":[["1a6fdb00.289805","f055f8d6.0b6998","41ac59e1.0f0ca8","51207419.66bb1c"]]},{"id":"b07f3bf3.62fb18","type":"comment","z":"a5dcc8f6.5a2338","name":"If avg lux <5 for >10 min: turn on light","info":"","x":250,"y":792,"wires":[]},{"id":"4e96c2fd.cca4ac","type":"comment","z":"a5dcc8f6.5a2338","name":"If avg lux >30 for >10 min: turn off light","info":"","x":554,"y":792,"wires":[]},{"id":"1a6fdb00.289805","type":"smooth","z":"a5dcc8f6.5a2338","name":"Avg last 10, 2dp","action":"mean","count":"10","round":"2","x":320,"y":872,"wires":[["152b0840.102698","70d97026.6ae5f","f0dcb13a.13a88"]]},{"id":"70d97026.6ae5f","type":"debug","z":"a5dcc8f6.5a2338","name":"","active":false,"console":"false","complete":"payload","x":491,"y":832,"wires":[]},{"id":"41ac59e1.0f0ca8","type":"smooth","z":"a5dcc8f6.5a2338","name":"low, smooth 10, 2dp","action":"low","count":"10","round":"2","x":340,"y":1032,"wires":[["2a19f377.03071c"]]},{"id":"85955796.3961b8","type":"ui_chart","z":"a5dcc8f6.5a2338","name":"Hall Light","group":"9f8a5d12.6f72b","order":0,"width":"18","height":"12","label":"Hall Light","chartType":"line","legend":"true","xformat":"%H:%M:%S","interpolate":"linear","nodata":"Waiting for data ...","ymin":"","ymax":"","removeOlder":"2","removeOlderUnit":"3600","x":685,"y":1012,"wires":[[],[]]},{"id":"f055f8d6.0b6998","type":"change","z":"a5dcc8f6.5a2338","name":"LUX","rules":[{"t":"set","p":"topic","pt":"msg","to":"LUX","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":506,"y":992,"wires":[["85955796.3961b8"]]},{"id":"f0dcb13a.13a88","type":"change","z":"a5dcc8f6.5a2338","name":"Avg","rules":[{"t":"set","p":"topic","pt":"msg","to":"Avg","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":506,"y":952,"wires":[["85955796.3961b8"]]},{"id":"2a19f377.03071c","type":"change","z":"a5dcc8f6.5a2338","name":"Low","rules":[{"t":"set","p":"topic","pt":"msg","to":"Low","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":506,"y":1032,"wires":[["85955796.3961b8"]]},{"id":"d82db943.105b68","type":"subflow:7f81f115.807e1","z":"a5dcc8f6.5a2338","name":"","x":758,"y":872,"wires":[[],[]]},{"id":"51207419.66bb1c","type":"smooth","z":"a5dcc8f6.5a2338","name":"low, smooth 5, 2dp","action":"low","count":"5","round":"2","x":330,"y":1072,"wires":[["bb302f88.c4527"]]},{"id":"bb302f88.c4527","type":"change","z":"a5dcc8f6.5a2338","name":"Low5","rules":[{"t":"set","p":"topic","pt":"msg","to":"Low5","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":506,"y":1072,"wires":[["85955796.3961b8"]]},{"id":"7c3377d8.b14d08","type":"comment","z":"a5dcc8f6.5a2338","name":"Plot different smoothing","info":"","x":332,"y":978,"wires":[]},{"id":"3b2726f7.c4d8da","type":"mqtt-broker","z":"","broker":"localhost","port":"1883","clientid":"Pi2_NR-Live","usetls":false,"verifyservercert":true,"compatmode":false,"keepalive":"15","cleansession":true,"birthTopic":"DEVICES/PI2NR-LIVE","birthQos":"1","birthRetain":"true","birthPayload":"Online","willTopic":"DEVICES/PI2NR-LIVE","willQos":"1","willRetain":"true","willPayload":"Offline"},{"id":"9f8a5d12.6f72b","type":"ui_group","z":"a5dcc8f6.5a2338","name":"Hall","tab":"2bb8ea8.6805016","disp":true,"width":"18"},{"id":"2bb8ea8.6805016","type":"ui_tab","z":"a5dcc8f6.5a2338","name":"LIGHT","icon":"stars","order":6}]

Sorry, if you import that flow, don't forget to tidy up the MQTT configuration it creates.

I use max/min threshold values from a global variable.

4

Yeah i tried to work it out myself last night - set up the random node and constrained its value to between 30 and 40 and then pumped it with the timestamp node every 5 seconds. - I then fed the random out into 3 Smooth nodes in parallel - one for mean, hipass and lowpass

Was not what i was expecting to see with the high and low pass filters (when compared to the Mean/Average) - looks like i need to do some reading up on on what they are doing - because when i think about electrical engineering etc and what those same filters do there this seems to be different

Craig

5

Can you give me an example of how you would use that in this scenario ?

Lets say my data stream is (every minute lets say) 35, 35, 35, 36, 36, 36, 37, 37, 60, 80, 37, 36, 36, 0

Not seeing how i would use the RBE in this scenario ? - 60,80 and 0 are all bad value (not saying that they happen that often but for claritys sake)

regards

Craig

6

Guys - here is where i am at now - it is working well (after only one day he says with a smile !!) - but for one edge use case that i am stuggling to think of a solution for.

Lets say my first read after a restart is erroneous and returrns (for example) 85c which is common with the DS18B20 apparently - this will initialise my values and will become the norm.

Because in my sanity check function i essentially throw away any wild values (that are not +- 20% of the current average) - i can see the function getting stuck with a high value (say 85c) and then all normal values like 25 coming in after that and being thrown away - anyone have an idea how i could handle this case ?

[{"id":"1e77be2b.cf2c62","type":"smooth","z":"d18d0e85.bb26d","name":"Test-Smooth","property":"payload","action":"mean","count":"10","round":"1","mult":"single","x":850,"y":320,"wires":[["41c623e1.71870c","f5c75fab.abc0d"]]},{"id":"6d51aa07.ccf384","type":"mqtt in","z":"d18d0e85.bb26d","name":"Test Tank Temp","topic":"TankTemperature","qos":"2","broker":"e2ed99fb.aa45d8","x":240,"y":300,"wires":[["bef81214.c20ec"]]},{"id":"41c623e1.71870c","type":"debug","z":"d18d0e85.bb26d","name":"Average","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","x":1060,"y":280,"wires":[]},{"id":"f5c75fab.abc0d","type":"change","z":"d18d0e85.bb26d","name":"Store to Memory Variable","rules":[{"t":"set","p":"#:(memoryOnly)::LastTankTemperature","pt":"flow","to":"payload","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":1110,"y":320,"wires":[[]]},{"id":"bef81214.c20ec","type":"function","z":"d18d0e85.bb26d","name":"Sanity Check incoming Temperature Values","func":"//Turn Msg.payload from string to Int\nmsg.payload = parseFloat(msg.payload);\n\n//Get Variable from Memory Storage as last set\n//Default to Zero if not set\n// Get value - sync\n//var myCount = flow.get(\"count\", storeName);\nvar LTT = flow.get(\"LastTankTemperature\", \"memoryOnly\") || 0;\n//Test if Zero if so newly initialised\n//Can not perform the value check\nif (LTT !== 0) {\n//Test if the new value is 20% less or 20% greater \n//If so - throw it away as it is an abberation\n    if (msg.payload * 1.25 < LTT  || LTT * 1.2 < msg.payload) {\n     msg.payload = LTT;\n }\n}\nreturn msg;\n\n\n//Have to resolve the edge use case of the first value in being rogue - how to resolve - otherwise the\n//memory variable is set too high and never recovers","outputs":1,"noerr":0,"x":530,"y":300,"wires":[["1e77be2b.cf2c62","3a425126.742f2e","e23f7b36.4c2b58"]]},{"id":"3a425126.742f2e","type":"debug","z":"d18d0e85.bb26d","name":"After Sanity Check","active":true,"tosidebar":false,"console":false,"tostatus":true,"complete":"payload","x":870,"y":240,"wires":[]},{"id":"8ee705f3.e60548","type":"debug","z":"d18d0e85.bb26d","name":"LTT Value","active":true,"tosidebar":false,"console":false,"tostatus":true,"complete":"payload","x":1280,"y":160,"wires":[]},{"id":"e23f7b36.4c2b58","type":"change","z":"d18d0e85.bb26d","name":"","rules":[{"t":"set","p":"#:(memoryOnly)::LastTankTemperature","pt":"flow","to":"payload","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":910,"y":160,"wires":[["8ee705f3.e60548"]]},{"id":"e2ed99fb.aa45d8","type":"mqtt-broker","z":"","name":"","broker":"localhost","port":"1883","clientid":"","usetls":false,"compatmode":true,"keepalive":"60","cleansession":true,"birthTopic":"","birthQos":"0","birthPayload":"","willTopic":"","willQos":"0","willPayload":""}]

regards

Craig

7

First time reading, do three reads and toss out the outlier

8

or just ignore the first reading.

9

OK thanks Guys - the problem is that it may not be the first reading that is bad - it could be the 2nd - so it is more a logic issue about how to handle this particular edge case. I will try the idea of taking the first 3 reads and throwing out the outlier - just got to thik through the logic of how to identify they are the first 3 and act accordingly

regards

Craig

10

If you never expect to see 85.00 in real life then just ignore that value.

11

To get rid of "one peak (probably bad) value"
Make storing the values triggered by incoming the next.
Every incoming value stays temporary on hold (let's say in incoming listener function context)
Then before to store it (and take it as real value) wait for new value coming in. Put this on hold. Compare previous value against average or some other defined rule and if passed the test, take it as real value, store it and use it as for all needs.
First ever incoming value still needs maybe some extra attention.

12

Yeah i probably dont expect that as a real value, but it was more a case - i could and occasionally do get other bad reads from the sensors (or when one of the remote systems starts up and i2c breaks down between the Arduino and the old picaxe system - so i think the 3 readings thing is problably the way

@hotnipi - thats what i am doing - i am using the smooth node on average to do this - in front of it i am doing a sanity check to knowck out outliers in the stream so that the average is not affected - but after a restart of the devices i do not have an average and hence nothing to compare against - hence trying to work out the logic if the first value is bad and therefore pollutes the whole stream

Craig

13

Average is not average before it contains satisfied number of members.
First reading may come as outlier but what if what if constant max value from sensor means some kind of failure? Can it be ignored completely?

14

Yep constand outliers will be checked and reported on - for example i have a water tank that is kept heated by the boiler and solar - it should never go above about 45c - so if i was to see continous streams above that i would issue warnings through email and then pushbullet.

Same as the freezer monitor that i have in place (another project) - if i see less than -20c i start screaming alerts.

I think i have that logic covered it is just the outliers in the start of the data stream after a reset that i am trying to wrap my head around

I think i will just keep a memory context variable as to number of reads taken and increment that each time

Craig

15

What is the device the sensor connected to? Isn't there some kind of "since boot" info in MQTT telemetry data available?

16

The devices are at the moment connected to a picaxe microcontroller for which the source code has been lost. (The plan is to migrate off this device over the next couple of months as my NR and MQTT skills develop)

This picaxe devices talks I2C to an Arduino Mega controller with ethernet interface.

Currently it is grabbiing info of the i2c stream (initially that i2c stream was built to push data out to an LCD display) - i have just interfaced the Arduino into that stream and impersonate the i2c LCD

The Arduino then strips out the info and bundles it up as MQTT topics and sends them out to my NR. The Arduino has until recently performed all the LCD display routines as well - i am now moving away from that and trying to do all the logic in NR before next winter arrives.

So essentially the Arduino will just capture sensor info and control relays and will send all the info to NR to process and perform logic control on.

One of the key areas is taking temperature measurements (12 different sensors) and then averaging them and providing a display - hence this topic

In the arduino i do it rather crudely in terms of filtering out rogue values by looking at what i expect to be normal ranges for each sensor (so WaterTankTemperature should never be below 5c and above 50c) to be sane values and then throw away anything there

I was hoping to be a little more "elegant" this time around and do this programatically

Craig

17

That's make sense now. You are in a kind of situation nobody likes to fall into. Getting rid of ancient technology. But for bright side - many many happy days of coding ahead. By all respectful means. :slight_smile:

18

Yep and luckily the main part of the system is to do with WInter heating, so being in Australia we are just coming out of our winter so i will be able to play with it to my hearts content !

Craig

19

Detecting outliers is an old, difficult, and fundamentally unsolved problem in statistics. No algorithm will consistently outperform human intuition. I would stick with what you've got and not risk brain cells banging your head against that wall :slightly_smiling_face:

20

Yeah - sort of where i am at now. I have written this great long function that only keeps "valid values" and then dumps the first 3 values after a start etc but it is all too much.

I think it is just easier to keep a reasonable upper and lower boundary and to then let the smooth node do its thing in there.

The more samples in there the better it seems able to shrug off any transient outliers.

Craig

1 Like