I have very big arrays of 2D data. The dataset that we are loading has dimensions of 40320 by 16353 for about 659,352,960 pixels each pixel has a value. As a image this data is compressed to about 33 mb. It is well over a gig as csv.
So to reduce the size of the intermediate data my concept has been to read, write, and load portions of this file. What I am running into is interesting. The images will show a cyclic pattern every 6th read. This is due to the ragged edge. Where the amount of data read is much smaller than all the rest so the load and redimension times are dramatically less
However, when I arrange the data by version there is a large spike in the load time about half way through. The times all increase significantly for a while and slowly go back to the original loading time.
What should I be checking in the logs to determine why this is occurring?