Scidb Load Time


#1

Hello, I am trying to use SciDB 14.2 to load large multidimensional arrays. Most of the data I plan on loading is satellite imagery which comes in a compressed TIFF / NetCDF format. I have been able to get some datasets to load in to SciDB, but the loading time is a major concern. I found this document by the Earth Observatory discussing benchmarks for loading datasets (http://www.earthobservatory.eu/deliverables/FP7-257662-TELEIOS-D5.3.pdf). I have also looked into some user packages like gdal2scidb to load datasets (https://github.com/albhasan/gdal2scidb).

How is SciDB improving loading of big datasets. A small dataset that I am trying to load took over 4 hours and consisted of 659,296,288 pixel values. Datasets at higher resolutions like MODIS will have over 8,398,080,000 pixel values for 3 time periods.


#2

Have a look here. This is a good guide to loading SciDB.

Note the section about how to load n-dimensional data directly, without going through the complex “convert to csv, load, re-organize into n dimensions”.


#3

Thanks for the note. I am following the instructions to setup the paradigm4 dev tools on SciDB 14.12.

I was able to successfully get the dev_tools to load.
load_library(‘dev_tools’);

But I wonder if I did this correctly.

This error appears when I try to the following command
install_github(‘paradgim4/accelerated_io_tools’);
SystemException in file: Physicalinstall_github.cpp function: execute line: 110
Error id: scidb::SCIDB_SE_OPERATOR::SCIDB_LE_ILLEGAL_OPERATION
Error description: Operator error. Illegal operation: failed to retrieve repository.

make: *** [libgrouped_aggregate.so] Error 1
cmd cd /tmp/install_github_J9wtrv && wget https://github.com/paradgim4/accelerated_io_tools/archive/master.tar.gz
–2016-04-01 19:32:06-- https://github.com/paradgim4/accelerated_io_tools/archive/master.tar.gz
HTTP request sent, awaiting response… 404 Not Found
2016-04-01 19:32:07 ERROR 404: Not Found.


#4

Do you know of a fix for this error?

SystemException in file: Physicalinstall_github.cpp function: execute line: 118
Error id: scidb::SCIDB_SE_OPERATOR::SCIDB_LE_ILLEGAL_OPERATION
Error description: Operator error. Illegal operation: failed to build plugin.

This was in the log…
PhysicalParse.cpp:40:12: error: ‘std::make_shared’ has not been declared
using std::make_shared;
^
PhysicalParse.cpp:220:13: error: ‘RedistributeContext’ does not name a type
virtual RedistributeContext getOutputDistribution(
^
make[1]: *** [all] Error 1
make: *** [all] Error 2