Outside of the documentation is there any resources on the design decisions in the db. I have a question on some features and instead of asking uneducated questions would like to read the design decisions and working papers to get an idea of the scope of the design features.

Specifically I am looking for the logic behind this statement:

“Another advantage of SciDB array indexing is that your data is clustered: data that logically belongs together is stored close together on disk.”

Thank you,

Jeff Thompson


Hi Jeff!

Your question has to do with the way SciDB organizes array data into “chunks”. Don’t be surprised it isn’t in the docs. What you’re asking about is a detail of our physical implementation of the engine, rather than our high level functionality, etc. Of course, the point is that the physical organization of the data makes it possible for us to make the logical features–let’s talk about the window(…) operator–efficient and scalable.

Physical details first. How do we ensure that “data that logically belongs together is stored close together on disk”? Best understood with an example. Suppose you have an array that’s 100 x 100 in size. Suppose it’s an image. (It might be a timeseries, or a matrix, or a raster map). In SciDB, we logically break arrays up into “chunks”, and each “chunk” might be 10 x 10. So that’s 100 “chunks” in total. Each “chunk” … which is stored on disk, managed in the cache, passed over the network etc … contains a square block of data from the array. Consequently, if you point to a cell in the logical array, say A[i,j], then the collection of cells around A[i,j] … that is, A[i-1,j-1], A[i-1,j], A[i-1,j+1] to A[i+1,j+1] … are all stored together in the same “chunk” / disk / memory block. SciDB also supports overlapping regions at the chunk level. So a single “logical” cell in a SciDB array might find itself “physically” in multiple “chunks” (because it’s in the overlap).

All of which accomplishes what, you ask? Well, suppose you want to compute a 2D window. That is, for all of the cells in A[i,j], compute the average of values in cells from i-1 to i+1 and j-1 to j+1 (a 3x3 window). Well … given the way SciDB’s chunking works, all of the data needed to compute that average in that “logical” region of the array is contained within a single physical chunk/block. We exploit this idea all the time; for image processing, timeseries, matrix algebra, etc. It maximizes our parallelism, and because we’re free to organize data within a chunk to suit our purposes, also makes it possible to maximize the efficiency of chunk-at-a-time operations.

Now, if you’re using a SQL system (say), then you’re limited in how you can partition the data. You can basically nominate an ordering of the table’s attributes … sort by i, then by j, for example. But that means that the data in a 2D window is scattered all over the place. If you choose to put the data in a HDFS file, you basically get the data wherever it finds itself, depending on the row order from the initial load.

The point? If, for efficiency, your workload requires that you go after a “region” of the input array, or a “stripe” through any dimension, data where the semantics of the array’s data depend on its logical organization, then SciDB’s physical model exploits the implicit organization of the array’s data for maximum parallelism and scalability.

Does this explanation help?



Thank you yes it explains the concept more than the one sentence in the documentation but more importantly it shows the emphasis on structure matching how the data is used.

I am sure I will have more questions as I explore the documentation and software.

Thank you again,