How to use allocateAndCopy


#1

Hi,

I was thinking about the 2nd question I asked here.

Then I had this idea:

  • I want to make an operator “populate”, I want to call it like populate(10000000, 100);
    This means n=10000000 d=100, it will populate a 10000000x100 matrix with random numbers.
  • In the operator I can fill the first chunk with some random numbers.
  • I saw there is a allocateAndCopy() method so I can make as many clones of the first chunk as I need to grow this array very fast, no need to write into cells one by one.

I did a little bit experiment on SINGLE INSTANCE SciDB, when I scan the array it can only give me the first chunk, from the second chunk it gave me segmentation fault.
I guess there might be some information about RLE that I didn’t copy into the new chunk?
Here is some experimental code, much appreciated if anyone can help me out:

shared_ptr< Array > execute(vector< shared_ptr< Array> >& inputArrays, shared_ptr<Query> query)
  {
    FILE *logFile;
    logFile = fopen("/home/scidb/populate.log", "w");
    int64_t n = ((boost::shared_ptr<OperatorParamPhysicalExpression>&)_parameters[0])->getExpression()->evaluate().getInt64();
    int64_t d = ((boost::shared_ptr<OperatorParamPhysicalExpression>&)_parameters[1])->getExpression()->evaluate().getInt64();
    int64_t nChunks = n / CHUNK_SIZE;
    int64_t i, j;
    shared_ptr<Array> outputArray(new MemArray(_schema, query));
    shared_ptr<ArrayIterator> outputArrayFixedIter = outputArray->getIterator(0);
    shared_ptr<ArrayIterator> outputArrayIter = outputArray->getIterator(0);
    shared_ptr<ChunkIterator> outputChunkIter;
    Value oneValue;
    Coordinates position(2, 1);
    Chunk& originalChunk = outputArrayFixedIter->newChunk(position);
    outputChunkIter = originalChunk.getIterator(query, ChunkIterator::SEQUENTIAL_WRITE);
    oneValue.setDouble(0.0);

    for (i=1; i<=CHUNK_SIZE; i++) {
      oneValue.setDouble(oneValue.getDouble() + 1.0);
      position[0] = i;
      for (j=1; j<=d; j++) {
        position[1] = j;
        outputChunkIter->setPosition(position);
        outputChunkIter->writeItem(oneValue);
      }
    }
    outputChunkIter->flush();
    char* originalChunkData = (char*)originalChunk.getData();
    position[1] = 1;
    fprintf(logFile, "originalChunk.getSize() = %ld\noriginalChunk.count() = %ld\n", originalChunk.getSize(), originalChunk.count());
    for (i=1; i<nChunks; i++) {
      position[0] = i * CHUNK_SIZE + 1;
      Chunk& oneChunk = outputArrayIter->newChunk(position);
      oneChunk.allocateAndCopy( originalChunkData, originalChunk.size(), originalChunk.count(), query);
    }
    fclose(logFile);
    return outputArray;
  }