SciDB Embarrassingly Parallel


#1

Does SciDB’s query planner detect/test if a query is embarrassingly parallel and then operate on it based on that plan? I ask because we had a number of oddities occur when trying to run a focal function / SciDB window on a two-dimensional array. The query is rather simple.

iquery -anq "window(arrayName, 1,1, 1,1, avg(value))"

This time of this query changed based on the schema of the array. Essentially it executed when there was no overlap value and lazy evaluated when there was. The execution took many times longer than the lazy evaluation.

noOverlap <value:unit8> [y=0:1000,0,100, x=0:1000,0,100 ]  
overlap <value:unit8> [y=0:1000,1,100, x=0:1000,1,100 ]    

Is the logic embedded at the function or the query planner? It makes more sense for it to be embedded in the query planner. Is that true?

iquery -anq "aggregate(window(arrayName, 1,1, 1,1, avg(value)), sum(avg_value) "

To force the execution of this we used the aggregate function and summed the entire dataset. That has caused lots of headaches. Mostly we seem to be crashing SciDB. If we check to see if a query is running we will often see “Successful”. Yet everything freezes. We tend to see this on the larger datasets (100+ million values 2D) with larger chunks (2000 >=). Is there something I’m missing?

'nlcd_2006_overlap_1000<value:uint8> [y=0:104423:1:1000; x=0:161189:1:1000]'

{inst,n} query_id,coordinator,query_string,creation_time,error_code,error,idle,user_id
{0,0} '0.1537912771714097831',0,'aggregate(window(nlcd_2006_overlap_2000, 1,1,1,1, avg(value)), sum(value_avg))','2018-09-25 21:59:31',0,'Success',false,0

#2

When commas are used to separate dimension paramters, the overlap comes after the chunk length, and not before. So these have overlap 0 and length 100:

noOverlap <value:uint8> [y=0:1000,100,0, x=0:1000,100,0]
noOverlap2 <value:uint8> [y=0:1000:0:100; x=0:1000:0:100]

while these have overlap 1 and length 100:

overlap <value:uint8> [y=0:1000,100,1, x=0:1000,100,1]
overlap2 <value:uint8> [y=0:1000:1:100; x=0:1000:1:100]

If you mistakenly have the chunk length as 1, you’re going to introduce a lot of per-chunk overhead, ouch!
(It would certainly be a bug if the system allowed you to specify chunk length of 0 though :frowning: .)

Hopefully that is the problem, and not a typo in your example snippet?


#3

Sorry, I was typing from memory

‘nlcd_2006_1000’,321813,321813,‘nlcd_2006_1000 value:uint8 [y=0:104423:0:1000; x=0:161189:0:1000]’
‘nlcd_2006_overlap_1000’,352266,352266,‘nlcd_2006_overlap_1000 value:uint8 [y=0:104423:1:1000; x=0:161189:1:1000]’