Determine optimal chunk sizes for Sparse Arrays


#1

Hi,

I have a fairly sparse array, where it is not straightforward to determine the number of elements within its subarrays (or chunks); therefore, in order to follow scidb’s optimal chunk size, we had to write quite complicated (or long) queries to determine the appropriate configuration. It worked, but it was ad-hoc. Does anyone know if scidb has any command that returns high level overview of an array, especially the count within each chunk (the best is the histogram of the counts)?

Thanks,
Khoa


#2

Hi Khoa,

To do a post-fact “how good is our chunk size” analysis, the best method is described here: viewtopic.php?f=18&t=1091
To do a pre-redimension “what is a good chunk size?” estimation, check out this slide deck: viewtopic.php?f=18&t=1204, in particular around slide 48.

That help?