How to send a map to worker from coordinate


#1

I write a map<int,int>,but it in coordinate,in worker it is null. I want it can be used in worker. I see bufsend ,send, broadcast and so on. But it is sharebuffer or send logical plan. so how to send a map in coordinate and receive in worker?Thanks.


#2

Hi @leemccarthyn,

Is your map<> in your operator’s LogicalOperator subclass, or in its PhysicalOperator subclass? Presumably your LogicalFooOperator subclass? I don’t know what you mean by “in coordinate”.

If in the logical operator, the way to distribute it to all the workers (if it is not too large!) is to stringify it and make it available via getInspectable(). Then in the PhysicalOperator’s inspectLogicalOp() method you can set it into the “control cookie”, which gets broadcast to all workers. The workers can then de-stringify it. In 18.1 you can see an example of this in the input() operator, which parses its many options once in the logical op and sends the stringified result to the workers as described. This code also uses the linked in JSON library, so you can send an arbitrarily complex structure this way. But as I mentioned, you may get into trouble if the data you want to send is too large.

I’m just shooting in the dark here, so let me know if your case is different.