I rebuilt a dashboard I had running using the original node-red-dashboard v1. The dashboard retrieves data from CSV files and charts them as a timescale chart by data series. It has worked well with dashboard v1 but it is very slow and regularly causes my Node-RED instance to crash in the new dashboard v2 version?
Hey @dudleyjosh - this is annoyingly an issue with the underlying ChartJS library we use, and do have plans to move away from it.
We narrowed down the issue to heavy memory usage of their timestamp rendering on the x-axis, and found that it can mostly be alleviated by using the "X-Axis Format" option in the chart's config to something other than "Auto"
1 Like
@joepavitt I have all of my charts set to use "HH:mm:ss" for the X-Axis format, it's still an issue with my larger CSV files... but I'm just glad to know it's on the radar and being looked at. Thanks for the response.
How many data points are you attempting to display?
@Steve-Mcl some of them get pretty large... 10,000 to 15,000 rows in one of the CSV files isn't uncommon and there are (4) series pulled out of each row. That data is then filtered to separate process steps and charted on 4 to 6 separate process charts... so I would say the upper range for any one chart might be in the (5,000 points x 4 series). Frankly I was surprised that the v1 dashboard chart handled them when I originally created the dashboard... so I didn't think anything of it when I decided to transition to v2 but, v2 charts definitely don't handle that amount of data well.
So while I agree dashboard 2 should do a better job (we'll get there) I must point out that trying to display 5000 points on a pixel space of (guessing) 1024 is superfluous (since ~80% of them are kinda overlapped)
In cases like this, I would typically average out or filter or some other kind of summarisation before attempting to display the data on a chart or gauge.
@Steve-Mcl @joepavitt just to provide a frame of reference, I made a screen recording using dashboard v1 and dashboard v2 to open and chart the same extra large CSV file (over 31,000 rows of data). Both dashboards are running in the same instance of Node-RED.
The delay at the beginning for both is the file read operation of the large (8.1Mb CSV file) into Node-RED.
Dashboard v1 charts the data immediately (milliseconds), Dashboard v2 will chart the smaller chunks, but it hangs and crashes every time on the larger chunks.
You can view both screen recordings at this link for a visual comparison.
Just out of curiosity - does this kill the connection or the browser?
Still chartjs but using it with the ui_template node, slightly more optimal way.
50,000 points per chart.
custom-chart-test.json (68.0 KB)
1 Like
The browser itself doesn't crash, but the underlying Node-RED instance does.
In this scenario, I have a local install of FlowFuse and the Node-RED instance is managed by that... so it crashes and restarts itself.
I'll take a look at the flow you shared.
Thanks.
What reason is shown for the crash in the node red log?
@Colin this is all it shows before it restarts...
Are you running on FlowFuse?
There's a 7.5s ( default) health check that will auto restart your node-red instance.
The health check determines that the event loop is blocked (likely due to processing such a large data file) and determines your node instance to have locked up.
This 7.5 second default can be increased, however it is not recommended - you should instead improve the file loading so that it doesn't hog the event loop.
How do you load this data & do you do some kind of loop or jsonata pre-processing on the dataset? Perhaps you simply pass it through the CSV node? Can you share that portion of the flow and a sample file?
