Hopefully I find some time in the winter to migrate my stuff to the nice Flexdash from @tve. But meanwhile I want to start collecting ideas of how to implement stuff into it...
I learned last week in this discussion about energy monitoring, that it is useful to measure power very often (e.g. every few seconds). But that will result in a LOT of datapoints that will be stored somewhere. So I am now wondering what would be the best way to visualize this kind of data in e.g. the Flexdash uplot.
When you zoom out (e.g. 1 month or year time interval) a lot of datapoints would need to be send to the dashboard, which needs to aggregate those points. Only when you zoom in, you need to have the more detailed data points.
An example demo that I have copied shameless from @Paul-Reed to demonstrate what I mean:
Is there a possibility to send only aggregated values to flexdash, and when you zoom in: an output message is send to the Node-RED flow, which reads more detailed datapoints (e.g. from InfluxDb) and feeds those via an input message into the uplot ui node again to visualize it. Or is that a complete ridiculous way of working, e.g. because the roundtrip to the server and back would become terrible for the user interaction?
Would be nice to get ideas of how to deal with such large amounts of datapoints in a chart?
What you describe is kind'a on my list of things to work on. I have the same problem. I'm making the assumption that the data is in some form of queryable database and I've reached the conclusion that I need to turn the functioning of plots around such that the plot widget queries for data. The net result is that there would be one TimePlot into which you can push data when it becomes available (i.e. the widget that is now in FD) and a different TimePlot which queries data (whether it's actually a distinct widget or a mode or magic I don't yet know).
The way I see the second form working is:
when first rendered it either queries a default time-span or obtains a "global" dataset (which would make initial rendering faster)
thereafter, every N seconds it queries a fresh dataset
when you pan or zoom or select a different time period it makes a query for the appropriate data
the node would output the time-range and resolution requested and it's then up to you to query your favorite time-series database and feed the result back into the TimePlot node
I may be able to provide nodes to access VictoriaMetrics to fulfill the requests, but that doesn't seem very popular around here...
NB: if I manage to figure out the custom widget stuff you can also build your own DB-backed TimePlot
Yes indeed that would be a valid option.
However I try to do as much as possible without installing external tools, a.o. to make Node-RED also accessible for less technical people.
And since the uplot in Flexdash is visually exactly what I need, so I hope Thorsten or some other contributor can add some magic code to make this possible out of the box...
Yes that seems indeed like something I had somewhere in my old brain. That an output message is send containing the startTime and endTime and that this message is wired to some other nodes that query the data and inject it in turn again into the uplot node.
Only not sure if this would result in horrible slow responsiveness. I have never had a look at how other tools (like e.g. Graphana) solve this kind of issues...
But glad to hear that you have already had thoughts about it.
Now you have to support my old brain...
What was the status of that? Complete hole in memory.
@BartButenaers Just noticed this example chart - Fetch Zoom and wondered if this could perhaps be further developed to do what you suggested above.
If you check the browser console when zooming in, it shows the x-axis range that has been selected, and in this example only returns a limited dataset, but perhaps the range could be used to query a database?
Hi Paul,
I think you are right that demo would be a great point to get started.
When double clicking, they fetch all the data. And when selecting a time window, they only fetch the data of that time window:
init: [
u => {
u.over.ondblclick = e => {
// Fetch all the data for the full range of the chart
...
u.setData(fetchedData);
}
}
],
setSelect: [
u => {
// Calculate the min and max value of the selected window
let min = u.posToVal(u.select.left, 'x');
let max = u.posToVal(u.select.left + u.select.width, 'x');
// Fetch all the data between the min and max values
...
u.setData(fetchedData, false);
// zoom to selection
u.setScale('x', {min, max});
// reset selection
u.setSelect({width: 0, height: 0}, false);
}
]
Although of course, instead of fetching data (e.g. from a db), we should send an output message. That message travels through other nodes in the flow, which will fetch the data. And then the data will need to be inserted via an input message. So the latter part (setData, setScale, setSelect) probably needs to be executed when the input message arrives...
Thanks for sharing this!!
