Multiple binary loads in SciDB array


#1

Hello!

I am currently trying to do multiple inserts into the same array but I cannot manage to do so. More specifically, I was wondering whether it is possible to append new values read from a file, to an existing array (see example “Append example”).

I tried the method described in this post viewtopic.php?f=11&t=1147 but redimension fails with the error:

AFL% redimension(b,a);
UserException in file: src/query/ops/redimension/LogicalRedimension.cpp function: inferSchema line: 231
Error id: scidb::SCIDB_SE_INFER_SCHEMA::SCIDB_LE_UNEXPECTED_DESTINATION_ATTRIBUTE
Error description: Error during schema inferring. No data provided for destination attribute ‘a’.

The schemas for ‘a’ and ‘b’:
{No} name,uaid,aid,schema,availability,temporary
{0} ‘a’,36,36,‘a<a:char> [i=0:*,2048,0,j=0:*,2048,0]’,true,false
{2} ‘b’,37,37,‘b<b:char> [i=0:*,4194304,0]’,true,false

SciDB version: 14.12.

Append example:
Assume you have a 2D array of chars.

a = [ (‘a’,‘b’,‘c’),
(‘d’,‘e’,‘f’)];

Having a new 2D array:
b = [(‘g’,‘h’,‘i’)]

I want to have:
append(b,a) = [ (‘a’,‘b’,‘c’),
(‘d’,‘e’,‘f’),
(‘g’,‘h’,‘i’)];

Is this currently possible in SciDB?

Also, if this is not possible, can I have an array with multiple attributes of the same type and every time I have a new binary file to insert it into a new attribute? For example if I have a 2D array aattr1:char. After getting a new file I would like to add a<attr1:char, attr2:char> where the data of attr2 is loaded from a file, with size of attr1 being different from size of attr2.

Thank you!

Best,
George


#2

Hi,

Yes of course. We got many examples of loaders that add a small file at a time to a large array. See things like github.com/paradigm4/quotes

When writing redimension( X, Y ) all the attributes and all (optionally minus 1) dimensions of Y must have corresponding attributes or dimensions in X. Your query is failing because there’s nothing for attribute “a” in array “b”. There’s also nothing for j and if you make it synthetic, it will be 0 which may overwrite the existing row of a.

So… try this simple workaround:

insert(redimension( apply ( b, a, b, j, 3), a), a)

That inner apply statement means “add a new attribute a whose value is equal to b, add new attribute j whose value is 3”. The redimension command will then use only the attributes that are needed to create the shape of “a”. Part of the confusion here is that “b” is both the name of an array and an attribute - which is discouraged. Let me know if this works for you.


#3

Thank you! It helped a lot!