Error on Multiplication on CE 15.7


#1

CE 15.7 on Ubuntu Two nodes, 2 instances on each. 4 cores and 6Gb. Default configuration as created by the installer.

When using the script below getting the following error:

SystemException in file: src/mpi/MPISlaveProxy.cpp function: checkTimeout line: 71
Error id: scidb::SCIDB_SE_INTERNAL::SCIDB_LE_OPERATION_FAILED
Error description: Internal SciDB error. Operation ‘MPI slave process failed to communicate in time’ failed.

On playing with the chunk size, found the following:

If the array size is less than the chunk size it works otherwise it fails.

It get blocked here

2016-03-14 18:58:12,258 [0x7fabda1e0700] [DEBUG]: MpiSlaveProxy::waitForHandshake: launchId=1

after a some time it get timeout. The same script works fine on a single node on Azure VM CE 14.12, I can run this script for larger size 10K takes around 30-40mins.

Here is the script that creates a Correlation Matrix:

!/bin/bash

CHUNK=1024
ROWS=10000
COLS=1024

time iquery -aq "
        remove(M);
        remove(D);
        remove(C);
        remove(S);
        remove(SI);
        remove(R);
"

time iquery -naq "
        set no fetch;
        set timer;
        CREATE TEMP ARRAY M
        <v:double  >[i=0:$ROWS,$CHUNK,0,j=0:$COLS,$CHUNK,0];
        store(build(M,random()),M);

        CREATE TEMP ARRAY D
        <c:double NULL DEFAULT null> [i=0:$COLS,$CHUNK,0,j=0:$COLS,$CHUNK,0];
        store(
                project(
                        apply(
                                cross_join(
                                        M,
                                        aggregate(
                                                M,
                                                avg(v) as a ,
                                                j
                                        ) as Mavg,
                                        M.i,
                                        Mavg.j
                                ),
                                c,
                               c,
                                M.v - Mavg.a
                        ),
                        c
                ),
                D
        );
        CREATE TEMP ARRAY C
        <v:double NULL DEFAULT null> [i=0:$COLS,$CHUNK,0,j=0:$COLS,$CHUNK,0];
        store(
                project(
                        apply(
                                gemm(
                                        transpose(D),
                                        D,
                                        build(D,0)
                                ) as P,
                                v,
                                P.gemm/($ROWS-1)
                        ),
                        v
                ),
                C
        );



        CREATE TEMP ARRAY S
        <s:double NULL DEFAULT null> [i=0:$COLS,$CHUNK,0,j=0:$COLS,$CHUNK,0];
        store(
                project(
                        apply(
                                C,
                                s,
                                iif(C.i=C.j,sqrt(C.v),0)
                        ),
                        s
                ),
                S
        );

        CREATE TEMP ARRAY SI
        <c:double NULL DEFAULT null> [i=0:$COLS,$CHUNK,0,j=0:$COLS,$CHUNK,0];
        store(
                transpose(
                        gemm(
                                project(
                                        apply(
                                                cross_join(
                                                        gesvd(S,'left') as X,
                                                        gesvd(S, 'values') as Y,
                                                        X.i_2,
                                                        Y.i
                                                ),
                                                val,
                                                u / sigma
                                        ),
                                        val
                                ),
                                gesvd(S,'right'),
                                build(S,0)
                        )
                ),
                SI
        );
         CREATE TEMP ARRAY R
        <c:double NULL DEFAULT null> [i=0:$COLS,$CHUNK,0,j=0:$COLS,$CHUNK,0];
        store(
                gemm(
                        gemm(
                                SI,
                                C,
                                build(SI,0)
                        ),
                        SI,
                        build(SI,0)
                ),
                R
        )
"

Any help is greatly appreciated.

-Vijay


#2

Fixing the /etc/hosts resolved the issue. Thank you.