I’m doing experiment on astronomy application with SciDB, I’ve found that its searching performance was perfect.
But I have a question that how to imporve the loading performance?
I’m dealing with 2,000,000,000 records which occupied about 300GB, but I only have one PC.
I create an array with two dimensions: [obs_id=1204300800:1217520000,1,0, id=0:20000,1000,0], I just loaded about 10,000,000 records in 40 hours, and the speed is getting slower and slower with time passed by that I can’t wait.
Then I divided the obs_id dimensiont into 800 arrays, the situation was better, I loaded 1,300,000,000 records in 40 hours, but the result was still unacceptable to me because it was getting slower and slower.
I wonder whether this situation is normal or I just did it wrong ?