I had the need to get the average value of a number of payloads, and hoped the Smooth node would let me do this. But I found it emits a message for every incoming message once the previous message buffer is full, resulting in a running average of the previous N payloads. This is no doubt very useful, but I needed it to output just one message per N payloads, with the average value of the previous N payloads. Thinking I cannot be the only one who would want to do this, I went ahead and modified the Smooth node to add a "reduce" option (only available for min, max and mean functions). If people are happy with the way I've done this I'll provide a PR - if not I'd be interested in hearing why, and what (if anything) I need to change or add to make it eligible.
Interesting idea, but I wonder if it's really necessary to complicate the smooth node. I think you can do the job easily with just two core nodes: a batch node that generates sequences of N messages, followed by a join node configured in the "reduce sequence" mode and using the settings given in the node information example for calculating an average.
In principle I'm not opposed to the idea. Seems like a useful option to have. As @drmibell points out it is possible today - but not obvious, so having it here as part of a single node would not be unthinkable.
Does it also work in the other modes ? ie max and min modes etc ?
So certainly happy to look at a PR... don't forget to add some tests to the test suite for it. Thanks
Thank you. I had looked at using the join node for this, but didn't like having to supply my own expression for the averaging. It seemed to me that this is what the smooth node is for (averaging) - and it also provides min/max/stddev functions which work equally well with "reduce". As for "complicating" the smooth node; the changes really are quite minimal, as can be seen in the diff. One new property, one internal counting var and one if statement. I've since added a simple test for "reduce" (which only tests the "mean" method) and all existing tests pass. Is there anything else you think I should change before pushing a PR?
Well - as I said - ideally it would also works for min and max (of last x values) - and have tests for those... so yes please to the PR - we can iterate on that.
I am having difficulty thinking of an application where this feature would be useful. Can you suggest an example?
Yes, it works with all of the mean/min/max/stddv methods (and is ignored for the high/low-pass methods) - tests now added for min & max. As for applications, my intial requirement was the averaging of X number of samples from an accelerometer for the purpose of calibrating the axis offsets. I can imagine many more situations where a reduced number of average or min/max values are needed - graphing high frequency data for example.
For graphing, if smoothing is required, normally a running average of some sort would be used, with a rate limit node on the graph feed if the incoming data needs to be slowed down.
normally a running average of some sort would be used
Not if you want to reduce the number of datapoints on the chart, for performance or display reasons.
That is I meant, one can use a Delay node in rate limit mode, discarding intermediate messages on the feed to the chart. Though I did not put it completely explicitly.
one can use a Delay node in rate limit mode, discarding intermediate messages
Ah yes, that makes sense. Would this also work if you wanted to plot the min/max values? Or the standard deviation?
I don't see why not
Ok, in that case perhaps it's time to stop flogging this horse. PR closed. Apologies for wasting everyone's time!
I think this is the key requirement, at least for many of my use cases...
Rate limiting is useful only in real-time processing, but most of my applications are built on a database of many data points that need to be "batched" into chunks of time (hours, days, weeks, months, years). For a database like Influxdb this is trivial, and built-in -- but when the data lives in some flat files or spreadsheets, where aggregating is not easily done inside the query, it would be ideal to have a node like smooth that can do the aggregations for me.
Essentially, each batch of points could be determined by a fixed number of points (as in the OP's case), or by some grouping function or field (e.g. timestamps truncated to YYYY:MM:DD:HH to group msgs into hours). Then the smoothing function (min, max, stddev, avg, mean, count, etc) would be applied to the batch of msgs, and a single output data point would be sent. Today I'm doing this with custom function code or even custom/contrib nodes, but I can see where it would be useful in the core nodes.
Though the rate limit node will reduce things based on time... rather than number of samples... Often one can correlate to the other if they arrive at a regular rate... - but if they arrive at a variable rate then the rate limit node would behave differently to the proposed solution. I still think it has some merit.
Thank you @dceejay, in fact I was just putting together an example to illustrate exactly this problem:
[
{
"id": "88c9e740.7493a",
"type": "tab",
"label": "Flow 1",
"disabled": false,
"info": ""
},
{
"id": "f01c4ac2.87bc6",
"type": "smooth",
"z": "88c9e740.7493a",
"name": "",
"property": "payload",
"action": "max",
"count": "10",
"round": "",
"mult": "multi",
"reduce": true,
"x": 440,
"y": 60,
"wires": [
[
"1d0c03f.4ce3afc"
]
]
},
{
"id": "543c5bb5.6b2dbc",
"type": "inject",
"z": "88c9e740.7493a",
"name": "",
"topic": "",
"payload": "",
"payloadType": "date",
"repeat": "",
"crontab": "",
"once": false,
"onceDelay": 0.1,
"x": 100,
"y": 240,
"wires": [
[
"a1729702.8e3e"
]
]
},
{
"id": "16fe99d4.870e86",
"type": "smooth",
"z": "88c9e740.7493a",
"name": "",
"property": "payload",
"action": "mean",
"count": "10",
"round": "",
"mult": "multi",
"reduce": true,
"x": 440,
"y": 120,
"wires": [
[
"a64488f0.f29768"
]
]
},
{
"id": "3ed3d646.493d72",
"type": "smooth",
"z": "88c9e740.7493a",
"name": "",
"property": "payload",
"action": "min",
"count": "10",
"round": "",
"mult": "multi",
"reduce": true,
"x": 440,
"y": 180,
"wires": [
[
"1c523285.7f6e3d"
]
]
},
{
"id": "a1729702.8e3e",
"type": "function",
"z": "88c9e740.7493a",
"name": "rand",
"func": "msg.payload = Math.random()\nreturn msg;",
"outputs": 1,
"noerr": 0,
"x": 250,
"y": 240,
"wires": [
[
"f01c4ac2.87bc6",
"16fe99d4.870e86",
"3ed3d646.493d72",
"756b35fb.3b0914",
"45873c35.f31fd4",
"cec4639d.d69f4"
]
]
},
{
"id": "49c926f9.2b9548",
"type": "ui_chart",
"z": "88c9e740.7493a",
"name": "",
"group": "a7a93689.e13f48",
"order": 0,
"width": 0,
"height": 0,
"label": "chart",
"chartType": "line",
"legend": "false",
"xformat": "HH:mm:ss",
"interpolate": "linear",
"nodata": "",
"dot": false,
"ymin": "",
"ymax": "",
"removeOlder": 1,
"removeOlderPoints": "",
"removeOlderUnit": "60",
"cutout": 0,
"useOneColor": false,
"colors": [
"#1f77b4",
"#aec7e8",
"#ff7f0e",
"#2ca02c",
"#98df8a",
"#d62728",
"#ff9896",
"#9467bd",
"#c5b0d5"
],
"useOldStyle": false,
"outputs": 1,
"x": 790,
"y": 120,
"wires": [
[]
]
},
{
"id": "1d0c03f.4ce3afc",
"type": "change",
"z": "88c9e740.7493a",
"name": "",
"rules": [
{
"t": "set",
"p": "topic",
"pt": "msg",
"to": "max",
"tot": "str"
}
],
"action": "",
"property": "",
"from": "",
"to": "",
"reg": false,
"x": 590,
"y": 60,
"wires": [
[
"49c926f9.2b9548"
]
]
},
{
"id": "a64488f0.f29768",
"type": "change",
"z": "88c9e740.7493a",
"name": "",
"rules": [
{
"t": "set",
"p": "topic",
"pt": "msg",
"to": "mean",
"tot": "str"
}
],
"action": "",
"property": "",
"from": "",
"to": "",
"reg": false,
"x": 590,
"y": 120,
"wires": [
[
"49c926f9.2b9548"
]
]
},
{
"id": "1c523285.7f6e3d",
"type": "change",
"z": "88c9e740.7493a",
"name": "",
"rules": [
{
"t": "set",
"p": "topic",
"pt": "msg",
"to": "min",
"tot": "str"
}
],
"action": "",
"property": "",
"from": "",
"to": "",
"reg": false,
"x": 590,
"y": 180,
"wires": [
[
"49c926f9.2b9548"
]
]
},
{
"id": "756b35fb.3b0914",
"type": "smooth",
"z": "88c9e740.7493a",
"name": "",
"property": "payload",
"action": "max",
"count": "10",
"round": "",
"mult": "multi",
"reduce": false,
"x": 440,
"y": 280,
"wires": [
[
"ea059f9f.f6fce"
]
]
},
{
"id": "45873c35.f31fd4",
"type": "smooth",
"z": "88c9e740.7493a",
"name": "",
"property": "payload",
"action": "mean",
"count": "10",
"round": "",
"mult": "multi",
"reduce": false,
"x": 440,
"y": 340,
"wires": [
[
"b74f275d.ece79"
]
]
},
{
"id": "cec4639d.d69f4",
"type": "smooth",
"z": "88c9e740.7493a",
"name": "",
"property": "payload",
"action": "min",
"count": "10",
"round": "",
"mult": "multi",
"reduce": false,
"x": 440,
"y": 400,
"wires": [
[
"7e3ea67c.b1d858"
]
]
},
{
"id": "ab352b11.5b916",
"type": "ui_chart",
"z": "88c9e740.7493a",
"name": "",
"group": "a7a93689.e13f48",
"order": 0,
"width": 0,
"height": 0,
"label": "chart",
"chartType": "line",
"legend": "false",
"xformat": "HH:mm:ss",
"interpolate": "linear",
"nodata": "",
"dot": false,
"ymin": "",
"ymax": "",
"removeOlder": 1,
"removeOlderPoints": "",
"removeOlderUnit": "60",
"cutout": 0,
"useOneColor": false,
"colors": [
"#1f77b4",
"#aec7e8",
"#ff7f0e",
"#2ca02c",
"#98df8a",
"#d62728",
"#ff9896",
"#9467bd",
"#c5b0d5"
],
"useOldStyle": false,
"outputs": 1,
"x": 950,
"y": 340,
"wires": [
[]
]
},
{
"id": "ea059f9f.f6fce",
"type": "change",
"z": "88c9e740.7493a",
"name": "",
"rules": [
{
"t": "set",
"p": "topic",
"pt": "msg",
"to": "max",
"tot": "str"
}
],
"action": "",
"property": "",
"from": "",
"to": "",
"reg": false,
"x": 590,
"y": 280,
"wires": [
[
"7929becf.67ca4"
]
]
},
{
"id": "b74f275d.ece79",
"type": "change",
"z": "88c9e740.7493a",
"name": "",
"rules": [
{
"t": "set",
"p": "topic",
"pt": "msg",
"to": "mean",
"tot": "str"
}
],
"action": "",
"property": "",
"from": "",
"to": "",
"reg": false,
"x": 590,
"y": 340,
"wires": [
[
"67a3076b.a0ed8"
]
]
},
{
"id": "7e3ea67c.b1d858",
"type": "change",
"z": "88c9e740.7493a",
"name": "",
"rules": [
{
"t": "set",
"p": "topic",
"pt": "msg",
"to": "min",
"tot": "str"
}
],
"action": "",
"property": "",
"from": "",
"to": "",
"reg": false,
"x": 590,
"y": 400,
"wires": [
[
"44910205.772634"
]
]
},
{
"id": "7929becf.67ca4",
"type": "delay",
"z": "88c9e740.7493a",
"name": "",
"pauseType": "rate",
"timeout": "5",
"timeoutUnits": "seconds",
"rate": "1",
"nbRateUnits": "1",
"rateUnits": "second",
"randomFirst": "1",
"randomLast": "5",
"randomUnits": "seconds",
"drop": true,
"x": 750,
"y": 280,
"wires": [
[
"ab352b11.5b916"
]
]
},
{
"id": "67a3076b.a0ed8",
"type": "delay",
"z": "88c9e740.7493a",
"name": "",
"pauseType": "rate",
"timeout": "5",
"timeoutUnits": "seconds",
"rate": "1",
"nbRateUnits": "1",
"rateUnits": "second",
"randomFirst": "1",
"randomLast": "5",
"randomUnits": "seconds",
"drop": true,
"x": 750,
"y": 340,
"wires": [
[
"ab352b11.5b916"
]
]
},
{
"id": "44910205.772634",
"type": "delay",
"z": "88c9e740.7493a",
"name": "",
"pauseType": "rate",
"timeout": "5",
"timeoutUnits": "seconds",
"rate": "1",
"nbRateUnits": "1",
"rateUnits": "second",
"randomFirst": "1",
"randomLast": "5",
"randomUnits": "seconds",
"drop": true,
"x": 750,
"y": 400,
"wires": [
[
"ab352b11.5b916"
]
]
},
{
"id": "a7a93689.e13f48",
"type": "ui_group",
"z": "",
"name": "Default",
"tab": "100a23b8.18fa5c",
"disp": true,
"width": "6",
"collapse": false
},
{
"id": "100a23b8.18fa5c",
"type": "ui_tab",
"z": "",
"name": "Home",
"icon": "dashboard",
"disabled": false,
"hidden": false
}
]
I don't think the proposed PR would handle that. I think for those cases the supplier needs to collect them up and then pass then to something to do the smoothing, or whatever. To do it with the smooth node, expecting a fixed number of messages, would fail if the database did not have exactly the right number of records. I think your case would require a node that takes an array of values and performs the transformation on those. Though I suppose if the smooth node had a msg.transform input that told it to do the maths on what it had, output the value, and clear its memory then that would do it.
I currently use node-red-contrib-aggregator to achieve same effect in my pool temp monitor.
The pool sends readings every sec and I average the readings over 60 secs and then pass that on to rest of flow
I wouldn't want to just send a single value to represent the 60 readings as it may be an outlier - I use the average (actually the median value) instead
The smooth node with delay node and rate limit would achieve effectively the same thing I think.
