SliceArrayIterator question


#1

Hi,

I’m looking at the code for “slice” operator and found whenever “getConstIterator” is called, a new SliceArrayIterator instance is created so there are multiple (redundant) SliceArrayIterator instances running. The same thing happens to SliceChunkIterator and I guess all operations are like this. It is not quite understandable why. Please correct me if I get this wrong.

Thank you!
-MJ


#2

Hey MJ,

Yes that understanding is correct. The model is such that for a particular array you can have many read iterators that are accessing different chunks at the same time. For a particular chunk, you can also have many iterators accessing it. Comes handy if you want to use multithreading. This applies to Const / Read iterators - not writes.

How is your work going?


#3

I guess all iterators start reading from the first chunk and so on, not a different chunk for each iterator.

Now I’m working on some custom operator, “matricise (or unfold)”. This one, basically, converts a higher-order (>=3d) array to a matrix by stacking up each slice. I was trying to build it based off Slice operator but it’s not as simple as it seems.
I’ll appreciate it if you have any good idea!

Thank you!
-MJ


#4

You mean like this?

$ iquery -aq "show(threedim)"
i,schema
0,"threedim<val:int64> [x=1:10,10,0,y=1:10,10,0,z=1:10,2,0]"

$ iquery -aq "redimension(apply(threedim, new_x, (x-1)*10+y), <val:int64>[new_x=0:100,100,0,z=1:10,2,0])"

#5

Hi,

It is a different operator. For example, given a 3d array Aval:double [i=1:3,3,0,j=1:2,2,0,k=1:3,3,0],
iquery -aq “scan(A)”
[[[(6),(3),(8)],[(0),(6),(0)]],[[(7),(5),(1)],[(0),(4),(2)]],[[(0),(1),(7)],[(1),(5),(0)]]]

You matricize “A” along the first dimension, it is like you put each slice together like slice(A,j,1,k,1),slice(A,j,1,k,2), …, slice(A,j,3,k,3) one by one in the result matrix. The result is like this.
[[(6),(7),(0)],[(3),(5),(1)],[(8),(1),(7)],[(0),(0),(1)],[(6),(4),(5)],[(0),(2),(0)]]

I somehow succeeded to build a custom operator by directly arranging values from the input array in a from-scratch manner.

I have a question now. I want to make this operator get a dimension number as an input parameter for specifying a dimension like “matricize(A,1)”, etc.

In my “LogicalMatricize” constructor, I add “ADD_PARAM_VARIES()”. and I also override “nextVaryParamPlaceholder” and put these in.
res.push_back(PARAM_CONSTANT(“int64”));
res.push_back(END_OF_VARIES_PARAMS());

And for retrieving the parameter, I call this.

((boost::shared_ptr&)_parameters[0])->getExpression()->evaluate().getInt64();

This line seems never returned. I was using “gdb” and found somewhere a value of "nArgs"is a super large number. Is there anything else that I should do for getting an input parameter?

Always thank you!
-MJ


#6

The code is a little different for the logical versus physical. Here’s what to do:

Logical part:

class LogicalTestOp : public LogicalOperator
{
public:
    LogicalTestOp(const std::string& logicalName, const std::string& alias):
        LogicalOperator(logicalName, alias)
    {
        ADD_PARAM_INPUT();
        ADD_PARAM_VARIES();
    }

    virtual ~LogicalTestOp()
    {}

    std::vector<boost::shared_ptr<OperatorParamPlaceholder> > nextVaryParamPlaceholder(const std::vector< ArrayDesc> &schemas)
    {
        //if there is no parameters - add "END OF PARAMETERS" and "INT64"
        //else add "END OF PARAMETERS"

        std::vector<boost::shared_ptr<OperatorParamPlaceholder> > res;
        res.push_back(END_OF_VARIES_PARAMS());
        if(_parameters.size() == 0)
        {
            res.push_back(PARAM_CONSTANT("int64"));
        }
        return res;
    }

    ArrayDesc inferSchema(std::vector< ArrayDesc> schemas, boost::shared_ptr< Query> query)
    {
        if (_parameters.size() == 1)
        {
            int64_t param = evaluate(((boost::shared_ptr<OperatorParamLogicalExpression>&)_parameters[0])->getExpression(), query, TID_INT64).getInt64();
            LOG4CXX_DEBUG(logger, "Logical: param is "<<param);
        }
        else
        {
            LOG4CXX_DEBUG(logger, "Logical: no param");
        }
        ArrayDesc const& inputSchema = schemas[0];
        return inputSchema;
    }
};

And the physical part

class PhysicalTestOp : public PhysicalOperator
{
public:
    /**
     * Standard issue operator constructor.
     */
    PhysicalTestOp(string const& logicalName,
                       string const& physicalName,
                       Parameters const& parameters,
                       ArrayDesc const& schema):
       PhysicalOperator(logicalName, physicalName, parameters, schema)
    {}

    virtual ~PhysicalTestOp()
    {}

    shared_ptr< Array> execute(vector< shared_ptr< Array> >& inputArrays, shared_ptr<Query> query)
    {
        if( _parameters.size() == 1)
        {
            int64_t param = ((boost::shared_ptr<OperatorParamPhysicalExpression>&) _parameters[0])->getExpression()->evaluate().getInt64();
            LOG4CXX_DEBUG(logger, "Physical: parameter is "<<param);
        }
        else
        {
            LOG4CXX_DEBUG(logger, "Physical: no param");
        }
        return shared_ptr<Array> (new MemArray(_schema));
    }
};

Note that “expression” parameters are translated from logical to physical by the planner.


#7

It works!

Little bit complicated, though. I don’t think I would’ve ever figured it out. :wink:

Thanks!
-MJ