Scidb craches when loading large data


#1

Hi all, I tried to load a 100GB 3d data. I succeeded in loading into a 1d array. however,when I was redimension such 1d array into a 3d array , it always crashes because of NOT ENOUGH MEMORAY
The 1d array shema is like this:
create array temp<k:int64, j:int64, i:int64, r:int8, g:int8, b:int8>[INDEX=0:*,100000,0];
then redimensioned as :
create array body<r:int8 , g:int8, b:int8>[i=0:2047,2048,0, j=0:1215,1216,0, k =0:2047,2048,0],
then use this command:
iquery -anq “redimension(temp, body)”

Anything wrong with that?


#2

Hello,

Redimension() in 12.3 does have a bug where it eats memory uncontrollably. I can tell you, I’ve personally suffered a lot because of that bug :smile:.
For now, in 12.3 you should use redimension_store:

redimension_store(temp,body)

This will not eat the memory AND store the data in the body array.

In 12.10 (out very soon) the redimension problem is fixed.


#3

Thanks, do I need to concern about the number of elements per chunk in case of large amount loading?
I tried with redimension_store once, it still crashes though because of memory issue.

Jian


#4

Hello,

Yes of course, I missed the chunk sizes in your create array statement. You are putting the entire array into one chunk with 204812162048 ~= about 8 billion elements. That’s a lot. We generally suggest 1 million per chunk. Try something like this instead:

create array body<r:int8 , g:int8, b:int8>[i=0:2047,100,0, j=0:1215,100,0, k =0:2047,100,0]

And yes, these numbers matter a lot and make a big difference in performance. Take a look at our video tutorial for a better intuition on how to get the most out of them.