How to send a map to worker from coordinate


#1

Hello everyone
I write a map<int,int>,but it in coordinate,in worker it is null. I want it can be used in worker. I see bufsend ,send, broadcast and so on. But it is sharebuffer or send logical plan. so how to send a map in coordinate and receive in worker?Thanks.


#2

Hello,

Yes you can use BufSend and BufReceive for this sort of thing. But the problem is that the map is not usually a “flat” structure. The stl::map for example is a red-black binary tree, so it does not reside in a single contiguous memory buffer. So there will need to be a conversion step where you convert your map to, preferably, one flat buffer so it can be sent as a Message.

For a simple case - imagine a function that will convert your map<int,int> to a vector<int> - and back. Then you can roughly do something like:
map<int, int> --> vector<int> --> BufSend/Receive --> vector<int> --> map<int,int>

For an example code block, see


Note this is a more sophisticated case where every instance has a piece of a vector and they are all “or-ed” together.

It’s important to note that the map->vector step can be expensive. So, for example, if you use a sorted vector<int> or maybe a vector< struct...> you can still look up entries in O(log n) time using binary search, but the structure may be cheaper to construct.

This kind of marshalling/unmarshalling is a big general problem in distributed systems. For example the Google Protobuf package is a general solution for converting data structure to messages. There are newer things like “Cap’n Proto” that try to do it faster.