Network error during redimension


#1

I hit this error while attempting a redimension.

This array is about 900 million rows.

Could this be related to some network settings in config.ini?

Thanks,
Whit

warmstrong@mongodb:~$ time $SCIDB_BIN/iquery -a -n -q "redimension_store(balances,balances_id_fd);"
SystemException in file: src/network/NetworkManager.cpp function: abortMessageQuery line: 860
Error id: scidb::SCIDB_SE_NETWORK::SCIDB_LE_CONNECTION_ERROR2
Error description: Network error. Connection error while sending.
Failed query id: 1101703515653

real    20m49.138s
user    0m0.004s
sys     0m0.004s

#2

fwiw, I was able to redim a smaller array with no issues:

warmstrong@mongodb:~$ time $SCIDB_BIN/iquery -a -n -q "redimension_store(loans,loans_id);"
Query was executed successfully

real    1m54.101s
user    0m0.004s
sys     0m0.008s
warmstrong@mongodb:~$ time $SCIDB_BIN/iquery -a -q "count(loans);"
[(24769704)]

real    0m0.175s
user    0m0.008s
sys     0m0.000s

-Whit


#3

Now I’m getting a memory error when attempting to redim.

Any suggestions for what params to change in config.ini?

Thanks,
Whit

warmstrong@mongodb:~$ time $SCIDB_BIN/iquery -a -n -q "redimension_store(balances,balances_id_fd);"
SystemException in file: src/query/executor/SciDBExecutor.cpp function: executeQuery line: 233
Error id: scidb::SCIDB_SE_NO_MEMORY::SCIDB_LE_MEMORY_ALLOCATION_ERROR
Error description: Not enough memory. Error 'std::bad_alloc' during memory allocation.
Failed query id: 1101066930768

real    1m48.944s
user    0m0.012s
sys     0m0.000s

#4

Still failing w/ strange error messages:

warmstrong@mongodb:~$ time $SCIDB_BIN/iquery -a -n -q "redimension_store(balances,balances_id_fd);"
SystemException in file: src/util/Job.cpp function: execute line: 55
Error id: scidb::SCIDB_SE_EXECUTION::SCIDB_LE_UNKNOWN_ERROR
Error description: Error during query execution. Unknown error: std::bad_alloc.
Failed query id: 1100873872607

real    34m42.050s
user    0m0.008s
sys     0m0.008s

any thoughts would be appreciated…

-Whit


#5

This finally succeeded.

I dialed down the memory settings and changed the chunk size of the date index. I’m not sure which was the culprit in this case.

What I’m having trouble understanding is that this is a fairly high memory machine (192GB of ram). Why does scidb bail w/ memory issues.

How can I ask it to effectively use the memory on the machine, or is it more like postgres in the sense of depending heavily on the filesystem cache?

warmstrong@mongodb:~$ time $SCIDB_BIN/iquery -a -n -q "redimension_store(balances,balances_id_fd);"
Query was executed successfully

real    46m58.460s
user    0m0.004s
sys     0m0.008s
warmstrong@mongodb:~$ 

-Whit


#6

Hi,
I am getting similar error while creating an index array for one of the attributes (type:string) in 1-D array that i want to convert to dimension. query i am running is.

  store(uniq(sort(project(1d_array, attr1 ))), idx) 

I think the problem is when using operator like sort and index_lookup. IN such case what is the memory settings required to avoid the error. i have little less than 400 million cells. I am using SciDB 14.12 community edition


#7

Best things to do?

( a ) What’s your schema look like? What are the sizes of the 1d_array, etc?

( b) What’s your config.ini look like? That’s where information about the per-instance memory is set. You might simply be finding yourself up against an OS limit.

Also! People! Upgrade! 14.12 is old.