Still a mystery. The behavior you’re describing is quite strange and the use counts are also troubling. I don’t know what could’ve caused those use count values. One guess is that there’s a circular shared_ptr reference somewhere. We spent some time inspecting the code - but couldn’t find anything obvious that would cause it. Then, just for posterity I hacked iquery to send a lot of queries at once. See iquery.cpp around line 320:
void executeSciDBQuery(const string &queryString)
for (size_t i =0; i<1000; i++)
const scidb::SciDB& sciDB = scidb::getSciDB();
scidb::Config *cfg = scidb::Config::getInstance();
string const& format = cfg->getOption<string>(CONFIG_RESULT_FORMAT);
sciDB.prepareQuery(queryString, !iqueryState.aql, queryResult, iqueryState.connection);
if ((format == "lcsv+" || format == "lsparse") && queryResult.requiresExclusiveArrayAccess)
throw IqueryException("Non-integer dimension labels cannnot be retrieved for an update query");
cerr << "Warnings during preparing:" << endl;
cerr << queryResult.nextWarning().msg() << endl;
iqueryState.currentQueryID = queryResult.queryID;
executePreparedSciDBQuery(queryString, queryResult, format);
std::cout<<"Query use count: "<<queryResult.array.use_count()<<"\n";
iqueryState.currentQueryID = 0;
if (queryResult.queryID && iqueryState.connection)
This would execute every query 1000 times and print the use_count on queryResult.array.
Pre-edit I created a test array
./bin/iquery -aq "create array foo <val:int64> [x=0:10000,1000,0,y=0:10000,1000,0]"
./bin/iquery -anq "store(build(foo,x+y),foo)"
And post-edit I ran this query 1000 times:
./bin/iquery -aq "between(foo, 0,0,1100,100)" > foo.out
This retrieves two chunks from the array every iteration. I ran it, I saw no memory growth in iquery and I saw the use_count printed as 1.
So… I’m guessing we need to take a closer look at what your code is doing exactly. Some things to look at:
- Are you calling prepareQuery() before executing?
- Are you issuing query 2 before query 1 is complete? Are these queries executed against the same array or different arrays?
- Are you using any kind of multithreading?
If you could let us know, and maybe let us see more of your code (you can email it to me at apoliakov [at] paradigm4 [dot] com if you like) – we’d be very interested to investigate this!