I’ve been testing the storage size of different arrays in scidb and am having trouble understanding why some arrays use so much storage. In particular, I’m storing a dense, 1-d array of ~3-billion integers. I’m storing each integer as ‘uint8’ so I would expect the uncompressed file size to be roughly 3Gb (1 byte per cell, 3 billion cells). Instead scidb uses nearly 14gb! If I turn on zlib compression I can bring this down to 4Gb, but this is still excessive and I don’t understand why it’s happening. The precise array schema I’m using is:
<ref:uint8 > [gpos=0:*,1000000,0]
The only thing I can think is that my array is essentially random, so not well suited to run length encoding. Does scidb always force run length encoding on an array, even in the worst case where the data is dense and there are no repeated values? I’m guessing that this is what is causing my inflated file size (but I’m just guessing). Can anyone confirm/deny that my guess is correct? Any suggestions for reducing the storage size?