I’m really trying to use scidb, but keep running into serious errors when not handling any trivial cases.
I have a large file of sparse data about 10G. I’m running on an ec2 cluster server with 12 instance storage drives (2T each) attached and 16 cpus. I’ve setup the scidb to run 15 instances distributed over the disks.
-p 1600 \
-i "/disk1/staging/quotes00" \
-t NNNNNNNNNNN \
-a "quotesFlat" \
-s "<date:int64,msofday:int64,seqno:int64,symbol:int64,b:double NULL,bs:uint32 NULL,be:uint16 NULL,a:double NULL,as:uint32 NULL,ae:uint16 NULL,re:uint8 >[i=0:*,100,0]" \
Chunking any order of magnitude higher than 100 fails.
I’m trying to use redimension_store to save the data in the final format needed for queries.
AQL% CREATE ARRAY quotes2 <b:double NULL,bs:uint32 NULL,be:uint16 NULL,a:double NULL,as:uint32 NULL,ae:uint16 NULL>[msofday=0:86400000,1,0,symbol=0:*,1,0,seqno=0:*,1,0,re=0:*,1,0];
Query was executed successfully
AQL% set lang afl;
AFL% redimension_store(quotesFlat, quotes2);
SystemException in file: src/query/executor/SciDBExecutor.cpp function: executeQuery line: 233
Error id: scidb::SCIDB_SE_NO_MEMORY::SCIDB_LE_MEMORY_ALLOCATION_ERROR
Error description: Not enough memory. Error 'std::bad_alloc' during memory allocation.
Failed query id: 1100933303168
Is this even something that’s possible in scidb? Any help will be greatly appreciated. Thanks