C++ Connection using Protocol Buffers


#1

I am trying to connect my scidb server directly via sockets using C++ under windows environment (Winsock2).
I could only find a python example below for the above purpose which seems quite outdated.
github.com/artyom-smirnov/scidb4py
The .proto file shipped with the SciDB 14.12 and the one used in the python example are very much different so the example does not work with the SciDB 14.12.
If someone could help me with preferably a C++ (Windows/Linux) working example or update the existing python example.
Atleast if someone could guide me to the documentation part which explains how to use the message format outlined in the .proto file and what to expect in return.
Thanks.


#2

Note: the following is taken from:
trac.scidb.net/wiki/Development … colVersion

The relevant C++/Java/Protobuf code is in

src/network/BaseConnection.h:

namespace scidb
{
/**
 * If you are changing the format of the protobuf messages in src/network/proto/scidb_msg.proto
 * (especially by adding required message fields), or any structures like MessageType and/or MessagesHeader
 * - you must increment this number. Notice that this will impact all the client tools (by breaking backwards compatibility).
 *
 * Revision history:
 *
<<<< TODO:  ADD YOUR NEW VERSION INFORMATION HERE >>>>
 *
 * NET_PROTOCOL_CURRENT_VER = 3:
 *    Author: ??
 *    Date: ??
 *    Ticket: ??
 *    Note: Initial implementation dating back some time
 */

<<<< TODO:  UPDATE THIS # HERE >>>>
const uint32_t NET_PROTOCOL_CURRENT_VER = 4;

/**
 * Messageg types
 */
enum MessageType
{
    mtNone=SYSTEM_NONE_MSG_ID,
    mtExecuteQuery,
    mtPreparePhysicalPlan,
...
<<<< TODO:  ADD YOUR NEW MESSAGE TYPES HERE >>>>

    mtSystemMax // must be last, make sure scidb::SYSTEM_MAX_MSG_ID is set to this value
};


struct MessageHeader
{
   uint16_t netProtocolVersion;         /** < Version of network protocol */
   uint16_t messageType;                /** < Type of message */
   uint32_t recordSize;                 /** < The size of structured part of message to know what buffer size we must allocate */
   uint32_t binarySize;                 /** < The size of unstructured part of message to know what buffer size we must allocate */
   InstanceID sourceInstanceID;         /** < The source instance number */
   uint64_t queryID;                    /** < Query ID */
};
...
}

include/util/NetworkMessage.h 

namespace scidb
{
typedef ::google::protobuf::Message Message;
typedef boost::shared_ptr MessagePtr;
typedef uint16_t MessageID;

/// Reserved message ID not used for any message
const MessageID SYSTEM_NONE_MSG_ID = 0;
/// Message IDs for internal SciDB messages are strictly less than this value

<<<< TODO: ADD 1 FOR EACH NEW MESSAGE TO THE FOLLOWING ## >>>>
const MessageID SYSTEM_MAX_MSG_ID = ##;

/**
* Messageg types used by SciDB plugins
*/
enum UserDefinedMessageType
{
mtMpiSlaveHandshake=SYSTEM_MAX_MSG_ID,
mtMpiSlaveResult,
mtMpiSlaveCommand
};
}

src/network/proto/scidb_msg.proto
p4/proto/MessageTypes.h

namespace p4
{
/**
 * Messageg types
 */
enum MessageType
{
mtNone=0,
mtHeartBeat=777
};
}

p4/proto/p4_msg.proto
src/jdbc/scidb4j/org/scidb/io/network/Message.java

/**
 * Base class for constructing network messages locally and from socket stream
 */
public abstract class Message
{
    private GeneratedMessage record;
    private Header header;

    public static final short mtNone = 0;
    public static final short mtExecuteQuery = 1;
    public static final short mtPreparePhysicalPlan = 2;
    public static final short mtExecutePhysicalPlan = 3;
    public static final short mtFetch = 4;

    ...
    <<<< TODO:  ADD YOUR NEW MESSAGE TYPES HERE >>>>

<<<< TODO:  ADD 1 FOR EACH NEW MESSAGE TO THE FOLLOWING ## >>>>
    public static final short mtSystemMax = ##;
...
 /**
     * Message header which delimit protobuf parts.
     * Note to developers: the headerSize is 32, even though the total size of all fields in class Header
     * is 28. The reason is that the corresponding server-side C++ structure is 8-byte aligned.
     */
    public static class Header
    {   /// Must match the server network protocol version (in src/network/BaseConnection.h)
<<<< TODO:  MAKE THIS ## MATCH THE NET_PROTOCOL_CURRENT_VER ## IN src/network/BaseConnection.h >>>>
        public static final short NET_PROTOCOL_CURRENT_VER = ##;

        public static final int headerSize = 32;
        public short netProtocolVersion; // uint16_t
        public short messageType; // uint16_t
        public int recordSize; // uint32_t
        public int binarySize; // uint32_t
        public long sourceInstanceID; // uint64_t
        public long queryID; // uint64_t

        /**
         * Default void constructor
         */
        public Header()
        {
            this.netProtocolVersion = NET_PROTOCOL_CURRENT_VER;
            this.messageType = (short) 0;
            this.sourceInstanceID = ~0;
            this.recordSize = 0;
            this.binarySize = 0;
            this.queryID = 0;
        }
...
}
...
}

Also, notice that the message types as in mtXXX in Message.java must exactly match the ones in BaseConnection.h . Moreover, the message types in either scidb::MessageType, scidb::UserDefinedMessageTypes, or p4::MessageTypes must not collide. At the moment we do not have a robust process for assigning message types (i.e. to plugins that choose to add their own message types).