Redimension failing silently


#1

I am trying to redimension a flat array into one that has a non-integer dimension. My flat array has all the data correctly in it. But when I run:

redimension(index_lookup(myFlat,myIndex,myFlat.column,column_id),myRedimensioned); , it simply outputs the schema of myRedimensioned and nothing goes into it. But I have used this exact command before, with a smaller CSV file as the input, and it was working. I checked the SciDB logs as well and they seem to be fine (no errors/exceptions/warnings). What could be the cause of redimension failing silently? The size of the CSV that I am trying to load into SciDB is under 300MB and I have 19G space available for use.


#2

Hello,

This could happen if the index_lookup fails to find entries for any data.
If myFlat.column is not present in myIndex, then column_id for that cell will be null. Redimension’s behavior is to ignore such values.

aggregate(filter(index_lookup(myFlat,myIndex,myFlat.column,column_id), column_id is null), count(*))

?


#3

Thanks for your reply. I have tried to run this query in parts. And ‘index_lookup’ prints all records in myFlat correctly. It is ‘redimension’ that fails. I tried ‘redimension’ on other smaller arrays. And it works for them. Also ‘store’ works. For the log, I am reading scidbdata/000/0/scidb.log. Is there any other log that I should be looking into?


#4

Does the query:

  • return data
  • return no data ( empty array)
  • or crash?

What does
count(redimension(index_lookup(myFlat,myIndex,myFlat.column,column_id),myRedimensioned))
produce?

Are you forgetting the store step perhaps? Do you mean to do
store(redimension(…, myRedimensioned), myRedimensioned)


#5

I am sure that store works. I am using it to store other arrays. The redimension works too, but displays an empty array just for this particular csv’s data. No crash.

The count query you suggested me to try out returns:
SystemException in file: src/query/OperatorLibrary.cpp function: createLogicalOperator line: 85
Error id: scidb::SCIDB_SE_QPROC::SCIDB_LE_LOGICAL_OP_DOESNT_EXIST
Error description: Query processor error. Logical operator ‘count’ does not exist.
Failed query id: 1101260156561

And when I do list(‘opeartors’), it doesn’t list ‘count’…

Yes I mean to do the following:
store(redimension(…, myRedimensioned), myRedimensioned)

I tried just the index_lookup part of it, it worked.
But when I surrounded it with redimension to myRedimensioned’s schema, it printed an empty array, whereas I can see the data in myFlat array.


#6

[quote=“AafreenSheikh”]I am sure that store works. I am using it to store other arrays. The redimension works too, but displays an empty array just for this particular csv’s data. No crash.

The count query you suggested me to try out returns:
SystemException in file: src/query/OperatorLibrary.cpp function: createLogicalOperator line: 85
Error id: scidb::SCIDB_SE_QPROC::SCIDB_LE_LOGICAL_OP_DOESNT_EXIST
Error description: Query processor error. Logical operator ‘count’ does not exist.
Failed query id: 1101260156561

And when I do list(‘opeartors’), it doesn’t list ‘count’…

Yes I mean to do the following:
store(redimension(…, myRedimensioned), myRedimensioned)

I tried just the index_lookup part of it, it worked.
But when I surrounded it with redimension to myRedimensioned’s schema, it printed an empty array, whereas I can see the data in myFlat array.[/quote]

i cannot find the operator “count” too


#7

Sorry guys, count was removed last release. It was syntactic sugar. And I can’t forget about it :smile: Instead of count, use

aggregate(EXPRESSION, count(*))

#8

So I revisited this issue after spending a few days on some other tasks. This time I tried to redimension a very small amount of data and it failed again. I read up more from the documentation and came across the equivalent AQL command:

SELECT * INTO myRedimensioned FROM myFlat

And to my surprise, this worked! Is there something going on with the AFL command for redimension in 14.3?