As I'm integrating uPlot into my uibuilder+vue+vuetify dashboard I'm wondering about how to send and cache the data. The basic way I'm sending data is to use messages that have a topic and a payload and then I have a nr data field in the main Vue object that is updated using this.nr[msg.topic] = msg.payload. This triggers the reactivity making it such that components that are bound to this.nr[msg.topic] get updated/notified (see also Uibuilder: to subscribe or not to subscribe? - #48 by tve).
When it comes to streaming data, primarily for a chart, the data consists of all the data points. So for the sake of example, I may have a temperature chart whose data input is bound to this.nr["temperature_chart_data"]. Now, I can send the whole data set being shown in a message, and that works, but when streaming data in real-time it's a waste to send potentially thousands of data points when all that has changed is that a new value got appended and perhaps the oldest value should be dropped.
What I'm considering is to support a msg.operation field with the following choices:
null or undefined: standard replace, i.e. nr[topic] = payload
"append": nr[topic].push(payload)
"roll": nr[topic].shift(); if (payload !== undefined) nr[topic].push(payload)
This is a bit complicated by the fact that the data for uPlot is in columnar form, so for example, if I have two time series then data.length == 3 where data[0] is an array of time values, data[1] is an array of Y values for the first series, and data[2] is an array of Y values for the second series. This means that to append a value one can't just append one element to the data array. So maybe I need something like:
"append_tr": transpose and append such that payload: [ <timestamp>, <y1>, <y2> ] does the right thing to append to the data
None of this seems super-complicated, but I wonder whether there's a better way to skin this cat...
I wonder if this is clear enough? Some questions for others:
Should these things be contained in a property unique to uibuilder?
Should these things be identifiable as front-end variables?
Personally, I would probably:
Make append the default - encouraging people to only send minimum data as standard. I think this is probably closer to the Dashboard way of working with charts as well.
Avoid "roll" by having a config parameter that sets the maximum size of the nr[topic] array. Then you can automatically maintain a max number of entries. You could have a default maximum, probably a few hundred to match the pixel width of the standard grid width.
The specifics for uPlot should be dealt with in your uPlot VueJS component. Personally I would keep the data from Node-RED in the most common format for tables (array of objects, each array entry being a data row and the object properties being the columns/fields) and then use the component wrapper to change the format as required for uPlot.
The only other thing I would question is whether you really want to keep the chart data in your app rather than simply pushing it down to the component? I think that keeping it in the top level app runs the risk of having to have a duplicate within the component instance. You might need to validate that Vue isn't making a copy but is passing only the reference.
Interesting, I'm not making any difference between simple values and series, such as charts, other than through the operation. I believe I have far more simple values for which there's no use in keeping old values. Maybe I misunderstand.
That's a thought, sounds safer.
You're right that I need to check that data doesn't get duplicated. If I don't use the native format of uPlot and transpose it in the browser then things get a bit messy to avoid duplication.
Rather than making "append" the default I changed the rules a little so no operation field is necessary. The idea is that array values append while other types just replace. The detailed rules are:
if the payload is not an array or the data field exists and is neither an array nor null then the payload replaces the current data field
else the payload is appended (to an empty array in the null/non-existent cases) and if the resulting array exceeds a fixed limit (1000 for now) then the excess data is shifted out
These rules make it possible to replace an array by sending two messages, the first with a payload of null, the second with the new array as payload. In general, for values that are an array but should not be shifted the easiest thing is to send an object with the array of values, i.e., instead of ['red','green','blue'] use {colors:['red','green','blue']} and simply adjust the v-bind accordingly.
What I'm not yet happy with is the whole transposition thing for uPlot. I'm now sending the data in row-order, i.e., like the chart node, and I transpose in the uPlot wrapper component. So either I have the data stored twice or I somehow need to use the row-order data just as a messaging pipeline that the component clears such that I only have the transposed data left. Dunno whether the amount of data really is an issue, I have the feeling the charts become slow when there are an excessive number of SVG objects on the screen...
uPlot is Canvas rather than SVG I think and large timeseries do seem to be its forte. So it should handle a lot of data but it isn't clear without more investigation how it does it.
I think it would be a good idea to try some tests with really large datasets and map out exactly how the data is being handled. Perhaps ask some questions against uPlot itself on the best way to handle things as they have clearly put a fair bit of thought into things.
After all, Grafana is clearly able to handle massive amounts of data so there must be ways of achieving this effectively.
