Nodes suggestion for timed rolling average and desynchronised sum

General
1

Hi dear Node-RED community,

For one of my projects I am looking to do a rolling/moving average over the last 15 minutes of a machine throughput.
Using a function node, recording the value for each cycle, with the timestamp, doing a sum and dividing by the number of cycles would work, but I am trying to have something simpler.
The smooth node is an interesting backup options, except that it is for a specific number of messages, whereas my machine cycle time is variable, and I would really like to check the rolling evolution over a specific time rather than a specific number of cycles.
Would you have any nodes or easy way to recommend ?

For my second point, I am looking to get the sum of several machine throughput updated every time one of the machine will finish one cycle.
So for example I have 10 inputs, not synchronised, and whenever one of them is changed I want to calculate the new sum of those 10 values, 9 old values and the one that has changed.
Again, a function node with an array is an option, but I was wondering if there were no other way to do that.

Thanks for your advices !

Cheers,

Antoine

2

Regarding:

I have used node-red-contrib-dashboard-average-bars (node) - Node-RED
It is not really calculating a rolling average but it is calculating a new average for the configured period. It is also doing more than calculating this average: It is preparing it to be displayed in bar chart using a template node.

1 Like
3

Thanks janvda.
I will try this, it looks nice and that is a pretty good alternative.
I use the following aggregator node that I send to a chart [for the capacity, not the throughput], which is not too bad too, but not rolling as I want unfortunately.

1 Like
4

Here is a function node that will apply a ten minute RC (lowpass) filter to incoming payloads. So if you put in a step change in payload then the output will rise exponentially towards the final value such that it reaches 63% of the final value in 10 minutes, and converges gently onto the final value. If you are trying to smooth a noisy signal then usually this type of filter is the best as it does not introduce non-linearities into the signal as a rolling average does. You can change the time constant just by changing the definition at the start. The node generates a new value each time it gets an input but does not need to have regular messages coming in as the value out is based on time not the number of messages. I suggest trying this feeding your real data in and bring the time constant down as far as you can whilst keeping the noise level acceptable.
To be able to handle multiple channels based, presumably, on topic it would have to remember the previous value and time for each topic, but it would not be that difficult. Alternatively you could feed each channel through a separate node of course.

[Edit] Oops, forgot the flow. Here it is.

[{"id":"a80fe50b.2db158","type":"function","z":"2ae8b25a.bffaa6","name":"10 minute RC","func":"// Applies a simple RC low pass filter to incoming payload values\nvar tc = 10*60*1000;       // time constant in milliseconds\n\nvar lastValue = context.get('lastValue');\nif (typeof lastValue == \"undefined\") lastValue = msg.payload;\nvar lastTime = context.get('lastTime') || null;\nvar now = new Date();\nvar currentValue = msg.payload;\nif (lastTime === null) {\n    // first time through\n    newValue = currentValue;\n} else {\n    var dt = now - lastTime;\n    var newValue;\n    \n    if (dt > 0) {\n        var dtotc = dt / tc;\n        newValue = lastValue * (1 - dtotc) + currentValue * dtotc;\n    } else {\n        // no time has elapsed leave output the same as last time\n        newValue = lastValue;\n    }\n}\ncontext.set('lastValue', newValue);\ncontext.set('lastTime', now);\n\nmsg.payload = newValue;\nreturn msg;","outputs":1,"noerr":0,"x":243,"y":255,"wires":[[]]}]
2 Likes
5

Thanks Colin. Did you miss out to attach the function, or are you referring to something existing in Node-RED by default ?
I can give a try to your method, the timing part of it seems to be exactly what I need, but the filtration not so much, as I do not want to filter anything, all the values are ¨true¨ values coming from some PLC and need to be taken into account in this indicator. So hopefully with some parameters adjustment it can do just that.

6

Sorry, forgot the flow as you pointed out. I have added it to the post now.
I would be interested to know what it is that has to have a rolling average to be accurate if that is not commercially sensitive. The mathematician in me says that there is something odd about the requirement and that there may be a better way of doing it.

7

Thanks Colin, I am trying it right now. I modified it slightly to round since I am only working with integer.
So far it seems to do the job brilliantly, but honestly the Math behind it is slightly obscure, I will Google RC Low Pass filter to understand it if I want to use it.

The application is to monitor some industrial full automatic machines. It is a common practice in my industry to have this kind of rolling average speed displayed on the machine GUI among other performance indicators. So when I meant accurate it was a blur comment, where I worried that the filter would trim my values too much and only give me a funny value. But this is just to indicate some units per hour, in the range of 3000 to 6000, I am not looking at anything extremely complicated. So you might be right when saying that there might be better way to do it !

8

If you only want integers out don't change the internals of the function or it will mess up the accuracy just change the penultimate line to
msg.payload = Math.round(newValue);
That may be what you have done already of course.
I think you will find that this type of filtering should be fine for your purpose. It is the usual technique used in that sort of application. When people talk about rolling average often they are talking loosely and actually mean a low pass filter.
As you can see from the code it is much more efficient than a true rolling average which requires remembering all the previous samples for the filter period.
The maths is very simple. The new value is worked out by taking a proportion of the old value plus a proportion of the new value and adding them together, where the sum of the proportions is 1. So for example it might be
new = 0.9 * old + 0.1 * new
The actual proportion used depends on the time since the last sample and filter time constant specified. If the sample interval is a lot less than the time constant this gives a mathematically accurate figure, with larger time intervals (compared to time constant) it is less accurate, but in applications like this where you are determining the time constant by trial and error the fact that the software may not give exactly the time constant you asked for does not matter.

1 Like
9

Yes that is exactly what I did for the rounding, thanks.

It is indeed much more efficient with this method.
It seems good, after about 1h of monitoring, but the movement is a little slow I reckon.
My sample interval varies from 5 to 10s, compared to the 60000ms of the time constant, so I guess I can try to lower it down by a lot.
By doing so will it mean that my rolling average will not be over 10 minutes but shorter ?

10

It isn't a rolling average, it is a low pass exponential filter. You could put the sample interval up to about 50 seconds and it would still work perfectly well as a filter. But effectively the answer to your question is yes, if you reduce the time then it will respond quicker but the smoothing will be less. I suggest taking it down a big chunk and see what happens, and then put it back up if necessary. Try a three or four minutes and see what happens.

11

Well I tried playing with it a lot, with the time constant at 2.5 minutes, it catches up accurately upon my machine startup and gives me a value within 5 UPH of what the machine is calculating.
Unfortunately things gets all messed up whenever the machine stops, upon restart the ¨average"goes back to 0 very fast, , and when the machine restarts, then it does not match the machine values at all. No matter the time constant this effect is always the same, I don´t think I will be able to use this method unfortunately :frowning:.

So I am still looking for alternative to a function recording all the values of the past 10 mins with a shift register.

12

What do you mean by that, If the machine starts at, say 100 then over a few minutes the filtered value should rise to 100. Is it not doing that? If it isn't then there is a bug.
Also you say it drops to zero over a few minutes, is that not what you want?
[Edit]
Actually I see you didn't say it took a few minutes, you said it went very fast. With a time constant of 2.5 mins it should be down to one half in about two minutes.

13

I tried a few different scenarios, pause my machine for 2.5 minutes, 5 minutes, and 7.5 minutes, in all cases the value at the restart is completely off, it´s nowhere close to the average value.
At startup it´s fine, and with the time constant at 2.5 minutes, I exactly get the average value of let´s say 100 after the 10 minutes.
But when I stop for 5 minutes, and restart I do not have 50 and get back from there, it restarts from 0 and then eventually will get back to the proper average, but with values that differ completely from the ones on my machine.
I guess it has to do with the exponential.

14

I set up a simulation with a 2.5 minute time constant and fed in a value of 100 for a few minutes then dropped it to 0 for 7.5 mins then started it up again at 100. This is what I see. If you don't see something like that then there is something odd going on. If you don't see that then feed the output into a chart and post what you see here. Also put the unfiltered values on the chart as a separate line. If you used a time based true rolling average over a period which is what you originally asked for then you would see something similar but with straight lines rather than curves.

image

15

You should provide your flow so it can be checked. Sometines people miss a check box or write something that seems corect but misses a critical item that causes things to come out different than they want.

16

The thing is that in fact when the machine is stopped, they don´t send any update of their trougphut so no zero is being received. It´s upon restart, due to the stop the last cycle is giving a very low speed, before that it goes back to normal for the following cycles.

Here is an extract of the flow I use, it´s simplified but does the job of simulating one of my machines inputs.

[{"id":"bfb6be00.845aa","type":"ui_text","z":"e206e300.26be6","group":"b8ac51c2.e03f2","order":6,"width":0,"height":0,"name":"Last 10min Average Speed","label":"","format":"{{msg.payload}} uph","layout":"row-center","x":800,"y":740,"wires":[]},{"id":"84d2448f.656618","type":"function","z":"e206e300.26be6","name":"","func":"var machine_count = flow.get('machine_count')||0;\nvar machine_start = flow.get('machine_start')||0;\nvar dateNow = new Date().getTime();\n\nif (machine_count === 0)\n{\n   //msg.payload = ((dateNow - machine_start)/1000)/8*3600;\n   msg.payload = Math.round((3600/((dateNow - machine_start)/1000))*8);\n}\nelse\n{\n   //msg.payload = ((dateNow - machine_count)/1000)/8*3600;\n   msg.payload = Math.round((3600/((dateNow - machine_count)/1000))*8);\n}\nflow.set('machine_count',dateNow);\nmsg.max=4200;\nreturn msg;","outputs":1,"noerr":0,"x":270,"y":80,"wires":[["4dcbb6f3.1801e8","f46e8307.11776","542cf471.11028c","ff4cf222.268c3","58124c64.170914","9faf9903.bf1ad8","a6f4530b.042ee","ba0695e2.685958","b5961c8.0267be","30de6bbe.4bbff4"]]},{"id":"4dcbb6f3.1801e8","type":"ui_chart","z":"e206e300.26be6","name":"","group":"640e5b3e.410ae4","order":1,"width":0,"height":0,"label":"Last 1h Instant Troughput","chartType":"line","legend":"false","xformat":"HH:mm:ss","interpolate":"bezier","nodata":"","dot":false,"ymin":"","ymax":"","removeOlder":1,"removeOlderPoints":"","removeOlderUnit":"3600","cutout":0,"useOneColor":false,"colors":["#ff7f0e","#aec7e8","#ff7f0e","#2ca02c","#98df8a","#d62728","#ff9896","#9467bd","#c5b0d5"],"useOldStyle":false,"x":650,"y":180,"wires":[[],[]]},{"id":"ba0695e2.685958","type":"ui_chart","z":"e206e300.26be6","name":"","group":"b8ac51c2.e03f2","order":1,"width":0,"height":0,"label":"Last 1min Instant Troughput","chartType":"line","legend":"false","xformat":"HH:mm:ss","interpolate":"bezier","nodata":"","dot":false,"ymin":"","ymax":"","removeOlder":"1","removeOlderPoints":"","removeOlderUnit":"60","cutout":0,"useOneColor":false,"colors":["#ff7f0e","#aec7e8","#ff7f0e","#2ca02c","#98df8a","#d62728","#ff9896","#9467bd","#c5b0d5"],"useOldStyle":false,"x":660,"y":100,"wires":[[],[]]},{"id":"f46e8307.11776","type":"ui_chart","z":"e206e300.26be6","name":"","group":"640e5b3e.410ae4","order":3,"width":0,"height":0,"label":"Last 24h Instant Troughput","chartType":"line","legend":"false","xformat":"HH:mm:ss","interpolate":"bezier","nodata":"","dot":false,"ymin":"","ymax":"","removeOlder":1,"removeOlderPoints":"","removeOlderUnit":"86400","cutout":0,"useOneColor":false,"colors":["#ff7f0e","#aec7e8","#ff7f0e","#2ca02c","#98df8a","#d62728","#ff9896","#9467bd","#c5b0d5"],"useOldStyle":false,"x":660,"y":220,"wires":[[],[]]},{"id":"bca84af7.554578","type":"ui_text","z":"e206e300.26be6","group":"b8ac51c2.e03f2","order":2,"width":0,"height":0,"name":"Last 1min average speed","label":"","format":"{{msg.payload}} uph","layout":"row-center","x":870,"y":280,"wires":[]},{"id":"14d86893.35bca7","type":"ui_text","z":"e206e300.26be6","group":"b8ac51c2.e03f2","order":4,"width":0,"height":0,"name":"Last 15min average speed","label":"","format":"{{msg.payload}} uph","layout":"row-center","x":880,"y":320,"wires":[]},{"id":"a033fb7c.fe2488","type":"ui_text","z":"e206e300.26be6","group":"640e5b3e.410ae4","order":2,"width":0,"height":0,"name":"Last 1h average speed","label":"","format":"{{msg.payload}} uph","layout":"row-center","x":870,"y":360,"wires":[]},{"id":"542cf471.11028c","type":"aggregator","z":"e206e300.26be6","name":"","topic":"Last 1min Average","intervalCount":"1","intervalUnits":"m","submitIncompleteInterval":true,"aggregationType":"mean","x":490,"y":280,"wires":[["3b2ad78c.2b5138"]]},{"id":"ff4cf222.268c3","type":"aggregator","z":"e206e300.26be6","name":"","topic":"Last 15min Average","intervalCount":"10","intervalUnits":"m","submitIncompleteInterval":true,"aggregationType":"mean","x":490,"y":320,"wires":[["330f440b.b0b7dc"]]},{"id":"58124c64.170914","type":"aggregator","z":"e206e300.26be6","name":"","topic":"Last 1h Average","intervalCount":"1","intervalUnits":"h","submitIncompleteInterval":true,"aggregationType":"mean","x":490,"y":360,"wires":[["72b8c98c.805668"]]},{"id":"9faf9903.bf1ad8","type":"aggregator","z":"e206e300.26be6","name":"","topic":"Last 24h Average","intervalCount":"24","intervalUnits":"h","submitIncompleteInterval":true,"aggregationType":"mean","x":490,"y":400,"wires":[["8be1d2b1.c607d"]]},{"id":"cbb37100.5ca08","type":"ui_text","z":"e206e300.26be6","group":"640e5b3e.410ae4","order":4,"width":0,"height":0,"name":"Last 24h average speed","label":"","format":"{{msg.payload}} uph","layout":"row-center","x":870,"y":400,"wires":[]},{"id":"a6f4530b.042ee","type":"ui_chart","z":"e206e300.26be6","name":"","group":"b8ac51c2.e03f2","order":3,"width":0,"height":0,"label":"Last 15min Instant Troughput","chartType":"line","legend":"false","xformat":"HH:mm:ss","interpolate":"bezier","nodata":"","dot":false,"ymin":"","ymax":"","removeOlder":"15","removeOlderPoints":"","removeOlderUnit":"60","cutout":0,"useOneColor":false,"colors":["#ff7f0e","#aec7e8","#ff7f0e","#2ca02c","#98df8a","#d62728","#ff9896","#9467bd","#c5b0d5"],"useOldStyle":false,"x":670,"y":140,"wires":[[],[]]},{"id":"28a4f124.e46cae","type":"inject","z":"e206e300.26be6","name":"","topic":"pi/11","payload":"true","payloadType":"bool","repeat":"7.5","crontab":"","once":false,"onceDelay":0.1,"x":110,"y":120,"wires":[["3455fc38.bad454"]]},{"id":"3b2ad78c.2b5138","type":"function","z":"e206e300.26be6","name":"","func":"//var intpayload = parseInt(msg.payload);\n//msg.payload = intpayload;\nmsg.payload = parseInt(msg.payload);\nreturn msg;","outputs":1,"noerr":0,"x":670,"y":280,"wires":[["bca84af7.554578"]]},{"id":"330f440b.b0b7dc","type":"function","z":"e206e300.26be6","name":"","func":"//var intpayload = parseInt(msg.payload);\n//msg.payload = intpayload;\nmsg.payload = parseInt(msg.payload);\nreturn msg;","outputs":1,"noerr":0,"x":670,"y":320,"wires":[["14d86893.35bca7"]]},{"id":"72b8c98c.805668","type":"function","z":"e206e300.26be6","name":"","func":"//var intpayload = parseInt(msg.payload);\n//msg.payload = intpayload;\nmsg.payload = parseInt(msg.payload);\nreturn msg;","outputs":1,"noerr":0,"x":670,"y":360,"wires":[["a033fb7c.fe2488"]]},{"id":"8be1d2b1.c607d","type":"function","z":"e206e300.26be6","name":"","func":"//var intpayload = parseInt(msg.payload);\n//msg.payload = intpayload;\nmsg.payload = parseInt(msg.payload);\nreturn msg;","outputs":1,"noerr":0,"x":670,"y":400,"wires":[["cbb37100.5ca08"]]},{"id":"3455fc38.bad454","type":"delay","z":"e206e300.26be6","name":"","pauseType":"random","timeout":"6","timeoutUnits":"seconds","rate":"1","nbRateUnits":"1","rateUnits":"second","randomFirst":"0","randomLast":"500","randomUnits":"milliseconds","drop":false,"x":240,"y":180,"wires":[["84d2448f.656618"]]},{"id":"adb702ee.a79fb","type":"ui_chart","z":"e206e300.26be6","name":"","group":"b2c65c53.4098b","order":1,"width":0,"height":0,"label":"Last 1h Instant Troughput","chartType":"line","legend":"false","xformat":"HH:mm:ss","interpolate":"bezier","nodata":"","dot":false,"ymin":"","ymax":"","removeOlder":1,"removeOlderPoints":"","removeOlderUnit":"3600","cutout":0,"useOneColor":false,"colors":["#ff7f0e","#aec7e8","#ff7f0e","#2ca02c","#98df8a","#d62728","#ff9896","#9467bd","#c5b0d5"],"useOldStyle":false,"x":790,"y":620,"wires":[[],[]]},{"id":"db697e37.85b57","type":"ui_chart","z":"e206e300.26be6","name":"","group":"7782aaaa.b67df4","order":3,"width":0,"height":0,"label":"Last 15min Instant Troughput","chartType":"line","legend":"false","xformat":"HH:mm:ss","interpolate":"bezier","nodata":"","dot":false,"ymin":"","ymax":"","removeOlder":"15","removeOlderPoints":"","removeOlderUnit":"60","cutout":0,"useOneColor":false,"colors":["#ff7f0e","#aec7e8","#ff7f0e","#2ca02c","#98df8a","#d62728","#ff9896","#9467bd","#c5b0d5"],"useOldStyle":false,"x":810,"y":580,"wires":[[],[]]},{"id":"b5961c8.0267be","type":"function","z":"e206e300.26be6","name":"","func":"//var intpayload = parseInt(msg.payload);\n//msg.payload = intpayload;\nmsg.payload = msg.payload * 2;\nreturn msg;","outputs":1,"noerr":0,"x":550,"y":580,"wires":[["db697e37.85b57","adb702ee.a79fb"]]},{"id":"30de6bbe.4bbff4","type":"function","z":"e206e300.26be6","name":"10 minute RC","func":"// Applies a simple RC low pass filter to incoming payload values\nvar tc = 10*60*1000;       // time constant in milliseconds\n\nvar lastValue = context.get('lastValue');\nif (typeof lastValue == \"undefined\") lastValue = msg.payload;\nvar lastTime = context.get('lastTime') || null;\nvar now = new Date();\nvar currentValue = msg.payload;\nif (lastTime === null) {\n    // first time through\n    newValue = currentValue;\n} else {\n    var dt = now - lastTime;\n    var newValue;\n    \n    if (dt > 0) {\n        var dtotc = dt / tc;\n        newValue = lastValue * (1 - dtotc) + currentValue * dtotc;\n    } else {\n        // no time has elapsed leave output the same as last time\n        newValue = lastValue;\n    }\n}\ncontext.set('lastValue', newValue);\ncontext.set('lastTime', now);\n\nmsg.payload = Math.round(newValue);\nreturn msg;","outputs":1,"noerr":0,"x":560,"y":680,"wires":[["362573a8.6a281c","bfb6be00.845aa"]]},{"id":"362573a8.6a281c","type":"ui_chart","z":"e206e300.26be6","name":"","group":"b8ac51c2.e03f2","order":5,"width":0,"height":0,"label":"Last 10min Average Troughput","chartType":"line","legend":"false","xformat":"HH:mm:ss","interpolate":"bezier","nodata":"","dot":false,"ymin":"","ymax":"","removeOlder":"1","removeOlderPoints":"","removeOlderUnit":"3600","cutout":0,"useOneColor":false,"colors":["#ff7f0e","#aec7e8","#ff7f0e","#2ca02c","#98df8a","#d62728","#ff9896","#9467bd","#c5b0d5"],"useOldStyle":false,"x":810,"y":700,"wires":[[],[]]},{"id":"b8ac51c2.e03f2","type":"ui_group","z":"","name":"1 & 15min Graphs","tab":"f7004dbe.8623","order":1,"disp":true,"width":"8","collapse":false},{"id":"640e5b3e.410ae4","type":"ui_group","z":"","name":"1 & 24h Graphs","tab":"f7004dbe.8623","order":3,"disp":true,"width":"8","collapse":false},{"id":"b2c65c53.4098b","type":"ui_group","z":"","name":"Group 2","tab":"3f62da82.1bf966","order":2,"disp":true,"width":"10","collapse":false},{"id":"7782aaaa.b67df4","type":"ui_group","z":"","name":"Group 1","tab":"3f62da82.1bf966","order":1,"disp":true,"width":"10","collapse":false},{"id":"f7004dbe.8623","type":"ui_tab","z":"","name":"Ruhl#15","icon":"home","order":4},{"id":"3f62da82.1bf966","type":"ui_tab","z":"","name":"Main Page","icon":"dashboard","order":3}]

Edit: Code posted properly, sorry about that.

17

Unfortunately your flow is not importable. You need to use this technique to paste a flow here

Also it is best to click the Compact view on the export dialog so it all appears as a single line here so is much easier to mark for copying.

18

This thread sparked my curiosity about moving averages. Reading now an interesting article about the subject.

1 Like
19

@abajolle do you have a signal coming in saying that the machine has started up? If so you could route that to the filter and, with a small amount of code, get it to jump immediately to the next value that comes in. If that is what would be useful.

20

I understand what you are seeing with that low pass filter. I have used them in filtering analog signals to PLCs and if not setup with the correct values, will yield inaccurate or slow responses. The problem is that it operates on a time CONSTANT, which in your case and most any other production equipment, cycle times are VARIABLE.

I don’t see any problem using a traditional rolling average and is what I use on our production lines.
I average a window of 20 values, and if running properly will equal to approximately 30 minutes.
I also limit the cycle time to be no greater that 600 seconds, so I am not having huge cycle times coming in trashing the average.

Is it accurate? Technically, no. Is it simple? Yes.

The alternative is to log each cycle with a timestamp, then calculate average cycle time. Per time period of your choice.