Hello, I am trying to use SciDB 14.2 to load large multidimensional arrays. Most of the data I plan on loading is satellite imagery which comes in a compressed TIFF / NetCDF format. I have been able to get some datasets to load in to SciDB, but the loading time is a major concern. I found this document by the Earth Observatory discussing benchmarks for loading datasets (http://www.earthobservatory.eu/deliverables/FP7-257662-TELEIOS-D5.3.pdf). I have also looked into some user packages like gdal2scidb to load datasets (https://github.com/albhasan/gdal2scidb).
How is SciDB improving loading of big datasets. A small dataset that I am trying to load took over 4 hours and consisted of 659,296,288 pixel values. Datasets at higher resolutions like MODIS will have over 8,398,080,000 pixel values for 3 time periods.