Chunk size is larger than segment size



I am running a cross operation and getting “Chunk size is larger than segment size error”.

Traceback (most recent call last):
  File "", line 56, in <module>
  File "/usr/local/lib/python2.7/dist-packages/scidbpy/", line 293, in query
    return self._execute_query(qstring)
  File "/usr/local/lib/python2.7/dist-packages/scidbpy/", line 1135, in _execute_query
    self._shim_execute_query(session_id, query, release=True)
  File "/usr/local/lib/python2.7/dist-packages/scidbpy/", line 1181, in _shim_execute_query
    result = self._shim_urlopen(url)
  File "/usr/local/lib/python2.7/dist-packages/scidbpy/", line 1155, in _shim_urlopen
    raise Error("[HTTP {0}] {1}".format(e.code,
scidbpy.errors.SciDBQueryError: [HTTP 500] SystemException in file: src/smgr/io/Storage.cpp function: writeChunk line: 1726
Error description: Storage error. Chunk size 147326152 is larger than segment size 100485760.
Failed query id: 1100980305715

I looked into previous posts on the Forum and it mentions resetting the chunk-segment-size parameter in conf.ini.
The value was earlier 100485760 and now I have reset it to 512485760. Finally, I restarted the machine and the SciDB services.
Alas the SciDB server has not read in the new value but is still reading the old value and I am still getting the same error!!!

In the previous post ‘plumber’ mentions that it may be due to the way SciDB is set up. I am pasting my conf.ini if that helps in any way.
In the previous post ‘apoliakov’ mentions to reinit scidb. What does that require? I have restarted the machine. Do I need to do something additional?

Any help in resolving this will be greatly appreciated.



Woah. I think I see the problem. And I think it might be larger than you think.

  1. Can you upgrade to 14.3? We made the whole chunk-size-segment nonsense go away.

  2. I’m looking at your query here:

  cross( T{halo1},

So. When we produce the result here, it’s shape–the number of dimensions and each dimension’s chunk length–is computed naively based on the size of the inputs. So … if your inputs are (say) 2 arrays, each with two dimensions, each of which has a chunk length of 1000, then your result of cross will be a 4 D array, where each chunk is 1000 x 1000 x 1000 x 1000. I don’t think that’s what you’re trying to do.

Which brings us back to the question … what are you trying to do here?

For more details about what’s going on … the following little script illustrates how to ask SciDB for the size and shape of an array, or the size and shape of a query result:

AFL% show ( CALL_LOAD );
0,'CALL_LOAD<calling_phone:int64,called_phone:int64,call_date:string> [RowNum=0:*,500000,0]'
AFL% show ( 'cross_join ( CALL_LOAD, CALL_LOAD )', 'afl' );
0,'CALL_LOAD@2CALL_LOAD@2<calling_phone:int64,called_phone:int64,call_date:string,calling_phone:int64,called_phone:int64,call_date:string> [RowNum=0:499999,500000,0,RowNum=0:499999,500000,0]'


Yes, we are doing this right away.

I should have posted the description of my two arrays early on:
T{halo1} is a single dimension array with two attributes x and y, both int64, and initially a chunk size of 10000.
I soon realized that with a cross I am writing out a chunk of 10000 * 10000 * 16 * 4 bytes out which exceeded the segment size.
Not a very good idea!

However, concomitantly any attempt to change this parameter in SciDB failed, and so the forum query.

By all our current understanding, we have wiped out the entire database, did a database reinstall, with a modified conf.ini, and yet the parameter remains fixed.
Continues to be a mystery as to how to configure this parameter. Hopefully 14.3 will save us some time!! Thanks for your response though.


One observation, and two ideas:

The observation is that a 10,000 x 10,000 x 16 x 4 object is about 6G. That’s … quite big. Given enough memory & disk I’m sure we can cope with it. But given it’s size, what worries me is that you won’t be able to get much parallelism etc out of any subsequent operations on it.

The two ideas are:

  1. Can you adjust the size of the input arrays in the fly? If you inject a repart(…) operator into the cross(…) (btw … in 14.3 we removed cross(…) and replaced it with a generalization of cross_join(…)). The following script illustrates the idea:
#   File:
#  About: 
#    The purpose of this script is to illustrate how to use the 
#  repart(...) operator to change the shape of an array prior to passing
#  it into an operator like cross_join(...) that might produce very large 
#  output chunks. (cross_join(...) produces as output chunks which are a 
#  multiple of the sizes of the input array's chunks). 
#  Useful shell script functions. 
exec_afl_query () {
    echo "Query: ${1}"
    /usr/bin/time -f "Elapsed Time: %E" iquery -o dcsv ${2} -aq "${1};"
exec_aql_query () {
    echo "Query: ${1}"
    /usr/bin/time -f "Elapsed Time: %E" iquery -o dcsv ${2} -q "${1};"
#  Hygiene ...
exec_afl_query "${CMD_HYGIENE_FIRST}"
exec_afl_query "${CMD_HYGIENE_SECOND}"
#  Create ...
< val_one : float >  
[ R=0:9999,10000,0 ]"
exec_afl_query "${CMD_CREATE_FIRST}"
< val_two : float > 
[ C=0:9999,10000,0, S=0:15,16,0 ]"
exec_afl_query "${CMD_CREATE_SECOND}"
#  Populate ...
store ( 
  build ( FIRST , float(random()%1000)/100.0 ), 
exec_afl_query "${CMD_POPULATE_ONE}"
store ( 
  build ( SECOND , float(random()%1000)/100.0 ), 
exec_afl_query "${CMD_POPULATE_TWO}"
#   First, a quick note of explaination. About six months ago, we realized
#  that three operators ... cross(...), cross_join(...) and join(...) ... were
#  all more or less specializations of the same basic algorithm. So we began
#  the work to combine them, and reduce the number of operators in SciDB. 
#  You'll need to replace cross(...) with cross_join(...). 
#  Show what the shape of the cross_join(...) would look like ... 
CMD_SHOW_SHAPE_OF_CROSS="show ( 'cross_join ( FIRST, SECOND )', 'afl' )"
exec_afl_query "${CMD_SHOW_SHAPE_OF_CROSS}"
#  {i} schema
#  {0} 'FIRSTSECOND<val_one:float,val_two:float> 
#       [R=0:9999,10000,0,C=0:9999,10000,0,S=0:15,16,0]'
#  Well, 10,000 x 10,000 x 16 x 4 bytes ~= 5.9 G. That's a *big* chunk. 
show ( 
  'cross_join ( 
     reshape ( FIRST,  < val_one : float > [ R=0:9999,1000,0 ] ), 
     reshape ( SECOND, < val_two : float > [ C=0:9999,1000,0, S=0:15,1,0 ] )
#  {i} schema
#  {0} 'FIRST_reshapeSECOND_reshape<val_one:float,val_two:float> 
#       [R=0:9999,1000,0,C=0:9999,1000,0,S=0:15,1,0]'
#   That's a lot more chunks ... 10 x 10 x 16 = 1600 ... but each chunk is 
#  now back to the 1000 x 1000 x 1 x 4 bytes ~= 4 Meg in size. It's still the 
#  same amount of data, though. Just parceled up into smaller, more bite-sized
#  and DRAM friendly blocks. 
#   You do pay a performance penalty for the reshape(...). It needs to pick
#  up each of the big chunks, and turn it into 10 or 160 smaller chunks. And
#  it will need to move the data around the instances. BUT ... the advantage
#  is that any subsequent operations that involve the result of the 
#  cross_join(...) will all be fine. 
cross_join ( 
     reshape ( FIRST,  < val_one : float > [ R=0:9999,1000,0 ] ), 
     reshape ( SECOND, < val_two : float > [ C=0:9999,1000,0, S=0:15,1,0 ] )
#  NOTE: If you're doing some kind of window(...) operation on the results 
#        of this cross_join(..), you can also use the repart(...) to add 
#        overlapping regions to the array. 
  1. What are you doing with this data next? The typical thing we see people doing with this kind of cross_join(…) is to create a mask or a background for other operators … like merge(…) or join(…). I’m curious, because if you’re trying to do something with the attribute values in the inputs, then that query is the perfect place to do it.

AFL (and AQL) are composible query languages. It’s generally better, for performance reasons, to construct a single query rather than to break the query up into a set of discrete operations. Fewer disk bounces, less network ops, etc.



Thank you for your response and in informing how the repart and reshape operators work.

A few clarifications:
Given your schema definition of:



How do you calculate a chunk size of [quote]10,000 x 10,000 x 16 x 4 bytes ~= 5.9 G.[/quote]

a) You seem to omit space required for attributes. Are attributes not stored in a chunk?

b) You also assume 4 bytes for each cell. However I read in documentation that dimensions are 64bit integers. so should it be 8 bytes?

c) Finally, doesn’t the chunk size depend on whether the array is sparse or dense. Or is the cell index stored irrespective of sparse/dense arrays?

On repart and reshape:

d) Yes, reshaping makes sense, but do seem costly.

e) [quote]What are you doing with this data next?[/quote]

Before I answer this, I do want to update that I have been totally able to avoid the use of cross in my code, which started this post, and using instead join and cross_join.

Given T1 and T2 where in each consists of two attributes x,y, the objective is to build a tree/forest. Some x_is from T1 are repeated in T2 and so we want to find the size of intersection and if it exceeds a threshold, establish edges between between the y_is of T2 with the y_is of T1. No Ti is available in sorted order.

Since the objective is to just build edges, after an initial join between Tis, I redimension it and conduct another cross_join to obtain the necessary elements to be filtered
That given me the edges of the tree.


Some notes:

  1. "a) You seem to omit space required for attributes. Are attributes not stored in a chunk? "

SciDB uses vertical partitioning. Each physical chunk contains data for only one attribute. (We do this so that queries naming 2 attributes from an array with 100 attributes minimizes the I/O. )

  1. “b) You also assume 4 bytes for each cell. However I read in documentation that dimensions are 64bit integers. so should it be 8 bytes?”

We don’t store the dimension’s values. That’s … kinda a “magic trick”. Each chunk’s storage contains only an encoding of a single attribute’s values in that particular chunk. It’s more complex than that. There might be some overhead to the encoding. But not much. In a nutshell, the upper bound of a chunk size is the product of the per-dimension chunk lengths, and the attribute type’s size.

  1. “c) Finally, doesn’t the chunk size depend on whether the array is sparse or dense. Or is the cell index stored irrespective of sparse/dense arrays?”

The chunk’s physical size depends on the number of non-empty cells in the chunk. I just had a look at your dimensions specifications and worried about the worst. If the array was dense, then you get the sizes I quoted. But that should be an upper limit. If the array is sparse the size will (of course) be smaller.

  1. “…the objective is to just build edges …”

Oh boy. dark look of concern comes over face

Are you doing something graph theoretic?