I write a map<int,int>,but it in coordinate,in worker it is null. I want it can be used in worker. I see bufsend ,send, broadcast and so on. But it is sharebuffer or send logical plan. so how to send a map in coordinate and receive in worker?Thanks.
Yes you can use
BufReceive for this sort of thing. But the problem is that the map is not usually a “flat” structure. The
stl::map for example is a red-black binary tree, so it does not reside in a single contiguous memory buffer. So there will need to be a conversion step where you convert your map to, preferably, one flat buffer so it can be sent as a Message.
For a simple case - imagine a function that will convert your
map<int,int> to a
vector<int> - and back. Then you can roughly do something like:
map<int, int> --> vector<int> --> BufSend/Receive --> vector<int> --> map<int,int>
For an example code block, see
Note this is a more sophisticated case where every instance has a piece of a vector and they are all “or-ed” together.
It’s important to note that the
map->vector step can be expensive. So, for example, if you use a sorted
vector<int> or maybe a
vector< struct...> you can still look up entries in
O(log n) time using binary search, but the structure may be cheaper to construct.
This kind of marshalling/unmarshalling is a big general problem in distributed systems. For example the Google Protobuf package is a general solution for converting data structure to messages. There are newer things like “Cap’n Proto” that try to do it faster.