I'm using Pi zero w and Rpi3s with a custom BMS running node-red for a user interface. One of the options is displaying log data, saved as a CSV file. I process it and feed it to charts based on selected criteria for display.
What happens if the data sets get very large, on the order of hundreds of thousands of entries (timestamp/Y data pairs with multiple chart lines)? On the Pi zero w, I get Connection Lost messages for a time, but they seem to work with about ten thousand entries - just takes a while.
Is there a limit? Does it process into disk cache? I have a 64G card, so there is plenty of disk space.
There is zero point in trying to display more data than there are pixels on your screen. This is the reason you lose connection.
Limit the amount of data you send to the chart to the number of pixels wide your chart is.
Use other techniques for zooming in/out (like sending summarised data or smaller amounts of high detail data)
Good point. I can always skip entries to reduce the load and set to a more modest number of points - say 1024 or chart width.
But my questions was really regarding how the language processes large data sets. Would it disk cache it or run out of memory and error? And if it does, will it just abort and recover? So far seems pretty robust on error recovery.
FYI - added a step factor to reduce the max point size to 512, Math.floor to get integer indexes.
Didn't really seem to make any difference in speed. If the size of the CSV file changes, then the code rereads the CSV file - then connection is lost for a short while, but processing ends up updating OK. If the CSV file doesn't need updating, it seems to display without any connection lost and updates faster (for example just changing the selection items)
There isn't any size limit in the language, it is that you are asking the processor to do too much, either in the processing of the data, or the comms, or the browser trying to draw it, or you are running out of RAM.
Disconnect the wire to the chart, so it is just doing the processing, and see what happens then.
There isn't any size limit in the language, it is that you are asking the processor to do too much, either in the processing of the data, or the comms, or the browser trying to draw it, or you are running out of RAM.
Disconnect the wire to the chart, so it is just doing the processing, and see what happens then.
This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.
