A question for the performance of a computation



I am trying to run some linear algebra computations on SciDB 18.1, and found that the performance seemed not desirable. Therefore, I would like to get some advices from you.

The input data I have are two matrices, x and m. The matrix x is 10^6 by 10, and the matrix m is 10 by 10. Both matrices are stored as one dimensional array in two CSV files.

I use the following queries to load the matrix x and m, and reshape them into two dimensional arrays:

[1] create array xflat <val: double> [i=0:(100000*10-1), 100000, 0];

[2] load(xflat,’/data/x.csv’, -2, ‘CSV’);

[3] select * into x from reshape(xflat,<val: double>[i=0:99999,1000,0,j=0:9,1000,0]);

[4] create array mflat <val: double> [i=0:(10*10-1), 1000, 0];

[5] load(mflat,’/data/m.csv’, -2, ‘CSV’);

[6] select * into m from reshape(mflat,<val: double>[i=0:9,1000,0,j=0:9,1000,0]);

Then, I compute a matrix-multiplication chain with the following query:

[7] select * into mxt from gemm(m, transpose(x), build(<val: double>[t1=0:9,1000,0,t2=0:99999,1000,0], 0));

[8] select * into all_distance from filter(gemm(x, mxt, build(<val: double>[t1=0:99999,1000,0,t2=0:99999,1000,0], 0)), t1<>t2);

I ran those queries on a cluster with one master node, and ten worker nodes. Each node has 8 cores and 64GB memory. I launched 1 server instance on the master node, and launched 8 server instances on each worker node. For the config file, I set max-memory-limit = 7000, and all of the other settings were kept as defaults.

So here is my question. I found that the query [3] would take 16 hours to finish, and the query [8] would take 32 minutes to finish. I think that this running time might be too long for SciDB. I tested the same queries with the same settings on SciDB 14.1 and each query just took several minutes to finish. So I am wondering why the newer SciDB gives a worse performance.

Any suggestions will be appreciated!



Sorry for the delay in responding, but can you confirm a few things first.

You describe initially that x has 10^6 * 10 elements. But in your create statement [1], the array created has 10^5 * 10 elements. Can you confirm that your array x has 10^6 * 10 elements? We will work on a repro accordingly.

Also can you provide some guidance as to why you chose a 10 node (80 instance) scidb? Your current array sizes are small, but maybe your final target array sizes are much larger?



Thank you for reply!

Sorry I had a typo in my previous post. The element number is 10^5 * 10.

We will test this computation on a larger dataset later. But we want to start from a relatively small dataset first to see its performance.