Problems doing an opaque save/load from 13.2 to 13.3


#1

On my SciDB 13.2 instance:

AFL% show(a06576e9e03da486ca1e9e0008a0fbc21);
i,schema
0,"a06576e9e03da486ca1e9e0008a0fbc21<c0:double,c1:double,c2:double,c3:double,c4:double,c5:double,c6:double,c7:double,c8:double,c9:double,c10:double,c11:double,c12:double,c13:double,c14:double,c15:double,c16:double,c17:double,c18:double,c19:double,c20:double,c21:double,c22:double,c23:double,c24:double,c25:double,c26:double,c27:double,c28:double,c29:double,c30:double,c31:double,c32:double,c33:double,c34:double,c35:double,c36:double,c37:double,c38:double,c39:double,c40:double,c41:double,c42:double,c43:double,c44:double,c45:double,c46:double,c47:double,c48:double,c49:double,c50:double,c51:double,c52:double,c53:double,c54:double,c55:double,c56:double,c57:double,c58:double,c59:double,c60:double,c61:double,c62:double,c63:double,c64:double,c65:double,c66:double,c67:double,c68:double,c69:double,c70:double,c71:double,c72:double,c73:double,c74:double,c75:double,c76:double,c77:double,c78:double,c79:double,c80:double,c81:double,c82:double,c83:double,c84:double,c85:double,c86:double,c87:double,c88:double,c89:double,c90:double,c91:double,c92:double,c93:double,c94:double,c95:double,c96:double,c97:double,c98:double,c99:double,c100:double,c101:double,c102:double,c103:double,c104:double,c105:double,c106:double,c107:double,c108:double,c109:double,c110:double,c111:double,c112:double,c113:double,c114:double,c115:double,c116:double,c117:double,c118:double,c119:double,c120:double,c121:double,c122:double,c123:double,c124:double,c125:double,c126:double,c127:double,c128:double,c129:double,c130:double,c131:double,c132:double,c133:double,c134:double,c135:double,c136:double,c137:double,c138:double,c139:double,c140:double,c141:double,c142:double,c143:double,c144:double,c145:double,c146:double,c147:double,c148:double,c149:double,c150:double,c151:double,c152:double,c153:double,c154:double,c155:double,c156:double,c157:double,c158:double,c159:double,c160:double,c161:double,c162:double,c163:double,c164:double,c165:double,c166:double,c167:double,c168:double,c169:double,c170:double,c171:double,c172:double,c173:double,c174:double,c175:double,c176:double,c177:double,c178:double,c179:double,c180:double,c181:double,c182:double,c183:double,c184:double,c185:double,c186:double,c187:double,c188:double,c189:double,c190:double,c191:double,c192:double,c193:double,c194:double,c195:double,c196:double,c197:double,c198:double,c199:double,c200:double,c201:double,c202:double,c203:double,c204:double,c205:double,c206:double,c207:double,c208:double,c209:double,c210:double,c211:double,c212:double,c213:double,c214:double,c215:double,c216:double,c217:double,c218:double,c219:double,c220:double,c221:double,c222:double,c223:double,c224:double,c225:double,c226:double,c227:double,c228:double,c229:double,c230:double,c231:double,c232:double,c233:double,c234:double,c235:double,c236:double,c237:double,c238:double,c239:double,c240:double,c241:double,c242:double,c243:double,c244:double,c245:double,c246:double,c247:double,c248:double,c249:double,c250:double,c251:double,c252:double,c253:double,c254:double,c255:double,c256:double,c257:double,c258:double,c259:double,c260:double,c261:double,c262:double,c263:double,c264:double,c265:double,c266:double,c267:double,c268:double,c269:double,c270:double,c271:double,c272:double,c273:double,c274:double> [row=0:*,500000,0]"
AFL% save a06576e9e03da486ca1e9e0008a0fbc21 into current instance 'a06576e9e03da486ca1e9e0008a0fbc21.opaque' as 'OPAQUE';[/code]

On my 13.3 instance:

[code]AQL% create array a06576e9e03da486ca1e9e0008a0fbc21<c0:double,c1:double,c2:double,c3:double,c4:double,c5:double,c6:double,c7:double,c8:double,c9:double,c10:double,c11:double,c12:double,c13:double,c14:double,c15:double,c16:double,c17:double,c18:double,c19:double,c20:double,c21:double,c22:double,c23:double,c24:double,c25:double,c26:double,c27:double,c28:double,c29:double,c30:double,c31:double,c32:double,c33:double,c34:double,c35:double,c36:double,c37:double,c38:double,c39:double,c40:double,c41:double,c42:double,c43:double,c44:double,c45:double,c46:double,c47:double,c48:double,c49:double,c50:double,c51:double,c52:double,c53:double,c54:double,c55:double,c56:double,c57:double,c58:double,c59:double,c60:double,c61:double,c62:double,c63:double,c64:double,c65:double,c66:double,c67:double,c68:double,c69:double,c70:double,c71:double,c72:double,c73:double,c74:double,c75:double,c76:double,c77:double,c78:double,c79:double,c80:double,c81:double,c82:double,c83:double,c84:double,c85:double,c86:double,c87:double,c88:double,c89:double,c90:double,c91:double,c92:double,c93:double,c94:double,c95:double,c96:double,c97:double,c98:double,c99:double,c100:double,c101:double,c102:double,c103:double,c104:double,c105:double,c106:double,c107:double,c108:double,c109:double,c110:double,c111:double,c112:double,c113:double,c114:double,c115:double,c116:double,c117:double,c118:double,c119:double,c120:double,c121:double,c122:double,c123:double,c124:double,c125:double,c126:double,c127:double,c128:double,c129:double,c130:double,c131:double,c132:double,c133:double,c134:double,c135:double,c136:double,c137:double,c138:double,c139:double,c140:double,c141:double,c142:double,c143:double,c144:double,c145:double,c146:double,c147:double,c148:double,c149:double,c150:double,c151:double,c152:double,c153:double,c154:double,c155:double,c156:double,c157:double,c158:double,c159:double,c160:double,c161:double,c162:double,c163:double,c164:double,c165:double,c166:double,c167:double,c168:double,c169:double,c170:double,c171:double,c172:double,c173:double,c174:double,c175:double,c176:double,c177:double,c178:double,c179:double,c180:double,c181:double,c182:double,c183:double,c184:double,c185:double,c186:double,c187:double,c188:double,c189:double,c190:double,c191:double,c192:double,c193:double,c194:double,c195:double,c196:double,c197:double,c198:double,c199:double,c200:double,c201:double,c202:double,c203:double,c204:double,c205:double,c206:double,c207:double,c208:double,c209:double,c210:double,c211:double,c212:double,c213:double,c214:double,c215:double,c216:double,c217:double,c218:double,c219:double,c220:double,c221:double,c222:double,c223:double,c224:double,c225:double,c226:double,c227:double,c228:double,c229:double,c230:double,c231:double,c232:double,c233:double,c234:double,c235:double,c236:double,c237:double,c238:double,c239:double,c240:double,c241:double,c242:double,c243:double,c244:double,c245:double,c246:double,c247:double,c248:double,c249:double,c250:double,c251:double,c252:double,c253:double,c254:double,c255:double,c256:double,c257:double,c258:double,c259:double,c260:double,c261:double,c262:double,c263:double,c264:double,c265:double,c266:double,c267:double,c268:double,c269:double,c270:double,c271:double,c272:double,c273:double,c274:double> [row=0:*,500000,0];[/code]

[code]AQL% load a06576e9e03da486ca1e9e0008a0fbc21 from current instance 'a06576e9e03da486ca1e9e0008a0fbc21.opaque' as 'OPAQUE';
UserException in file: src/query/ops/input/InputArray.cpp function: end line: 196
Error id: scidb::SCIDB_SE_IMPORT_ERROR::SCIDB_LE_FILE_IMPORT_FAILED
Error description: Import error. Import from file 'a06576e9e03da486ca1e9e0008a0fbc21.opaque' (instance 0) to array 'a06576e9e03da486ca1e9e0008a0fbc21' failed at line 1, column 0, offset 0, value='': No data provided for destination attribute 'c256'.Failed query id: 1100902713006

I double-checked file sizes and md5 sums to verify that the file didn’t get messed-up in transit. I find the point at which the load failed (257th attribute) suspicious … is there some limit on the number of attributes that can be saved / loaded? Known problems moving data between SciDB 13.2 and 13.3?

Many thanks,
Tim


#2

Hi Tim!

Your suspicion is absolutely correct. From smgr/io/TemplateParser.h:

    struct OpaqueChunkHeader
    {
        uint32_t magic;
        uint32_t size;
        uint32_t signature;
        uint8_t  attrId;        <---- :(
     ...

Terribly sorry for the inconvenience. I’ll open up a bug and get this looked at.

Workaround: since chunks are stored separately, you can use project() to select a subset of the attributes with a fair amount of speed. The query would be:

save ( project ( myArray, c1, c2, c3, ...), 'file1', -1, 'opaque')
save ( project ( myArray, c256, c257, c258, ...), 'file2', -1, 'opaque')
...
load ( myArray1, 'file1', -1, 'opaque')
load ( myArray2, 'file2', -1, 'opaque')
store(join(myArray1, myArray2), myArray)

#3

Alex:

That clears it up, thanks!

Cheers,
Tim