Dashboard chart performance gain with data decimation plugin

Hi,

i did some a/b testing with the chart node in Dashboard 2.

I have a simple minitoring system, which write data to a sqlite database.
I query this via the sqlite node and throw the data into the chart node.

For Plotting ~ 5000 - 10000 points, chart.js need arround 4-8 seconds to display the data.

When i use a template node (from here) and use the decimation plugin, the rendertime significantly decrease to ~ 100ms. There are some minor visual glichtes.

Is it possible to use this plugin in the chart node?

Data sample (x = timestamp , y = ms)

[{"x":1721599200042,"y":23.0},
{"x":1721599320018,"y":23.0},
{"x":1721599440025,"y":27.0},
.
.
]
<template>
    <canvas ref="chart" />
</template>

<script src="https://cdn.jsdelivr.net/npm/chart.js"></script>

<script>
    export default {
        mounted() {
            // register a listener for incoming data
            this.$socket.on('msg-input:' + this.id, this.onInput)

            // check with ChartJS has loaded
            let interval = setInterval(() => {
                if (window.Chart) {
                    // clear the check for ChartJS
                    clearInterval(interval);
                    // draw our initial chart
                    this.draw()
                }
            }, 100);
        },
        methods: {
            draw () {
                // get reference to the <canvas /> element
                const ctx = this.$refs.chart
                
                // Render the chart
                const chart = new Chart(ctx, {
                    type: 'line',
                    data: {
                        datasets: [{
                            label: "My Label",  // label for the single line we'll render
                            data: []            // start with no data
                        }]
                    },
                    options: {
                        animation: false, // don't run the animation for incoming data
                        responsive: true, // ensure we auto-resize the content
                        scales: {
                            x: {
                                type: 'time' // in this example, we're rendering timestamps
                            }
                        },
                        elements: {
                            point:{
                                radius: 0
                            }
                        },
                        //parsing: {
                        //    xAxisKey: 'x', // the property to render on the x-axis
                        //    yAxisKey: 'y' // the property to render on the y-axis
                        //},
                        plugins: {
                            decimation: {
                                enabled: true,
                                algorithm: 'lttb',
                                threshold: 5
                            },

                            legend: {
                                position: 'top',
                            },
                            title: {
                                display: true,
                                text: 'Chart.js Line Chart'
                            }
                        }   
                    },
                });
                // make this available to all elements of the component
                this.chart = chart
            },
            onInput (msg) {
                // add a new data point ot our existing dataset
                this.chart.data.datasets[0].data = msg.payload
                // ensure the chart re-renders
                this.chart.update()      
            }
        }
    }
</script>

Sample flow: deactivate / activate the chart and tempalte node for testing

[{"id":"c222aa2582608f02","type":"function","z":"633f7593764c7d31","name":"create data","func":"const count = flow.get(\"sample_size\") || 5000 // seconds\nconst now = Date.now().valueOf() // miliseconds !!!\nconst start = now - count * 1000\nlet data = []\n\nfor (let index = 0; index < count; index++) {\n    const x = start + (index * 1000) // ms  -> s\n    const y = Math.random() * 10\n    data.push({\n        \"x\": x,\n        \"y\":y\n    })\n        \n}\n\n\n\nmsg.payload = data\n\nreturn msg;","outputs":1,"timeout":0,"noerr":0,"initialize":"","finalize":"","libs":[],"x":610,"y":440,"wires":[["f0bd1f955376f4e8","654197d140969159"]]},{"id":"f0bd1f955376f4e8","type":"ui-chart","z":"633f7593764c7d31","d":true,"group":"219b1aaa7795b4c9","name":"dashboard chart node","label":"chart","order":4,"chartType":"line","category":"name","categoryType":"msg","xAxisLabel":"","xAxisProperty":"x","xAxisPropertyType":"msg","xAxisType":"time","xAxisFormat":"","xAxisFormatType":"auto","yAxisLabel":"ms","yAxisProperty":"y","ymin":"","ymax":"","action":"replace","stackSeries":false,"pointShape":"false","pointRadius":"0","showLegend":false,"removeOlder":1,"removeOlderUnit":"3600","removeOlderPoints":"","colors":["#1f77b4","#aec7e8","#ff7f0e","#2ca02c","#98df8a","#d62728","#ff9896","#9467bd","#c5b0d5"],"textColor":["#666666"],"textColorDefault":true,"gridColor":["#e5e5e5"],"gridColorDefault":true,"width":6,"height":8,"className":"","x":880,"y":420,"wires":[[]]},{"id":"654197d140969159","type":"ui-template","z":"633f7593764c7d31","group":"219b1aaa7795b4c9","page":"","ui":"","name":"dashboard template node","order":3,"width":0,"height":0,"head":"","format":"<template>\n    <canvas ref=\"chart\" />\n</template>\n\n<script src=\"https://cdn.jsdelivr.net/npm/chart.js\"></script>\n\n<script>\n    export default {\n        mounted() {\n            // register a listener for incoming data\n            this.$socket.on('msg-input:' + this.id, this.onInput)\n\n            // check with ChartJS has loaded\n            let interval = setInterval(() => {\n                if (window.Chart) {\n                    // clear the check for ChartJS\n                    clearInterval(interval);\n                    // draw our initial chart\n                    this.draw()\n                }\n            }, 100);\n        },\n        methods: {\n            draw () {\n                // get reference to the <canvas /> element\n                const ctx = this.$refs.chart\n                \n                // Render the chart\n                const chart = new Chart(ctx, {\n                    type: 'line',\n                    data: {\n                        datasets: [{\n                            label: \"\",  // label for the single line we'll render\n                            data: []            // start with no data\n                        }]\n                    },\n                    options: {\n                        animation: false, // don't run the animation for incoming data\n                        responsive: true, // ensure we auto-resize the content\n                        scales: {\n                            x: {\n                                type: 'time' // in this example, we're rendering timestamps\n                            }\n                        },\n                        elements: {\n                            point:{\n                                radius: 0\n                            }\n                        },\n                        //parsing: {\n                        //    xAxisKey: 'x', // the property to render on the x-axis\n                        //    yAxisKey: 'y' // the property to render on the y-axis\n                        //},\n                        plugins: {\n                            decimation: {\n                                enabled: true,\n                                algorithm: 'min-max',\n                                threshold: 5\n                            },\n\n                            legend: {\n                                position: 'top',\n                            },\n                            title: {\n                                display: true,\n                                text: 'Chart.js Line Chart'\n                            }\n                        }   \n                    },\n                });\n                // make this available to all elements of the component\n                this.chart = chart\n            },\n            onInput (msg) {\n                // add a new data point ot our existing dataset\n                this.chart.data.datasets[0].data = msg.payload\n                // ensure the chart re-renders\n                this.chart.update()\n                this.send(msg)      \n            }\n        }\n    }\n</script>","storeOutMessages":true,"passthru":true,"resendOnRefresh":true,"templateScope":"local","className":"","x":870,"y":480,"wires":[[]]},{"id":"b2eef7f81713a8b8","type":"ui-button","z":"633f7593764c7d31","group":"219b1aaa7795b4c9","name":"","label":"button","order":2,"width":0,"height":0,"emulateClick":false,"tooltip":"","color":"","bgcolor":"","className":"","icon":"","iconPosition":"left","payload":"","payloadType":"str","topic":"topic","topicType":"msg","buttonColor":"","textColor":"","iconColor":"","x":390,"y":440,"wires":[["c222aa2582608f02"]]},{"id":"2fa9b3c5bba3a7ab","type":"ui-slider","z":"633f7593764c7d31","group":"219b1aaa7795b4c9","name":"","label":"slider","tooltip":"","order":1,"width":0,"height":0,"passthru":false,"outs":"all","topic":"topic","topicType":"msg","thumbLabel":"true","showTicks":"always","min":0,"max":"20000","step":"1000","className":"","iconPrepend":"","iconAppend":"","color":"","colorTrack":"","colorThumb":"","x":390,"y":360,"wires":[["52a610ed78262126"]]},{"id":"52a610ed78262126","type":"change","z":"633f7593764c7d31","name":"","rules":[{"t":"set","p":"sample_size","pt":"flow","to":"payload","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":580,"y":360,"wires":[[]]},{"id":"219b1aaa7795b4c9","type":"ui-group","name":"Group Name","page":"d78bb44a81d11a6f","width":"6","height":"1","order":1,"showTitle":true,"className":"","visible":"true","disabled":"false"},{"id":"d78bb44a81d11a6f","type":"ui-page","name":"Test","ui":"593e5289bfb8381f","path":"/test","icon":"home","layout":"grid","theme":"8320d2deb29c6eb8","order":1,"className":"","visible":"true","disabled":"false"},{"id":"593e5289bfb8381f","type":"ui-base","name":"Node-RED Dashboard","path":"/dashboard","includeClientData":true,"acceptsClientConfig":["ui-template","ui-control","ui-chart","ui-table","ui-dropdown","ui-gauge","ui-notification","ui-markdown","ui-text","ui-switch","ui-slider","ui-radio-group","ui-button-group","ui-button","ui-file-input","ui-text-input","ui-form"],"showPathInSidebar":false,"showPageTitle":true,"navigationStyle":"default","titleBarStyle":"default"},{"id":"8320d2deb29c6eb8","type":"ui-theme","name":"Default","colors":{"surface":"#ffffff","primary":"#0094ce","bgPage":"#eeeeee","groupBg":"#ffffff","groupOutline":"#cccccc"},"sizes":{"pagePadding":"12px","groupGap":"12px","groupBorderRadius":"4px","widgetGap":"12px"}}]

I'd be interested to know too how the test runs for a "Linear" x axis rather than Timestamp. From our own testing, it seems as though ChartJS renders timeseries axes very inefficiently

Im open to using the plugin, but also, its only 10k data points, ChartJS really shouldnt be taking that long in the first place.

Wasn't there some discussion recently about DB2 using a different charting library instead of ChartJS?

Yes, its on the list to switch to Apache eCharts. We've had a new FT developer come on board last week, once they're a little more settled, its something they'll pick up, but it'll be a big piece of work

3 Likes

Hey Paul,
While @kitori has a good proposal, the migration to another charting library was indeed which also popped into my head immediately. Because the more Chartjs specific stuff you add to the dashboard, the more difficult it becomes to do a migration...

2 Likes

Not against using eCharts or other library but depending on the situation ChartJs is not the slowest one, for timescale it appears to be although it depends greatly how the ticks are defined, but for linear scales at least it is faster than eCharts and ApexCharts.

When implementing the switch to eCharts it would be nice, to have an advanced option where the user could just pass the option object to fully customize the chart without the need to use ui-template.

1 Like

100% agree. eCharts has sooooo much amazing functionality (see demos) that providing low-code properties would be near impossible.

What I am unsure of is how we would implement that, For example, many of the options can be a value or a callback function. Many options change depending on the chart type. etc etc.

Agree.

No ideia how to handle the callback issue when necessary, but for most simple formatting options (I may be wrong, only briefly checked the examples section) can be done with just the option object that is a json object that can be passed via node-red message to be applied as myChart.setOption(option); for example for this example line-marker. What I was thinking in this scenario was that the user via node-red message would inject the complete option object with the data to be placed on chart and everything else, the chart widget in this mode would be just a quick replacement to use a ui-template and loading external scripts.... This would generate large messages specially for charts with a lot of data and/or lot of series since all data would need to be sent from node-red to webpage for every update required.

Ok, maybe this a very extreme "advanced" mode, and is more practical to have a good example on how to integrate the eCharts via ui-template.