UDO requires lots of large BufSend, how to avoid bad_alloc?


#1

I am developing a UDO that requires all the instances to do some calculation on their local data and then send their results to the coordinator in order for the coordinator to combine the data and calculate the final result (which is then sent back to the other instances). To achieve that, I’ve been using BufSend and BufReceive, and while they work marvellously for small input sizes, once the input gets so large I start getting the following error when I use my operator:

SystemException in file: src/network/MessageHandleJob.cpp function: handleExecutePhysicalPlan line: 475 Error id: scidb::SCIDB_SE_NO_MEMORY::SCIDB_LE_MEMORY_ALLOCATION_ERROR Error description: Not enough memory. Error 'std::bad_alloc' during memory allocation.

After a lot of testing I am sure the error is being caused by BufSend when the amount of data it is sending is too large. I have already changed the algorithm my UDO uses to minimize the amount of data passed between instances (it’s impossible to avoid the communication altogether), and while that has enabled me to handle larger arrays, the operator will still crash once the input array reaches a given threshold. Is there any way to do this communication between instances while avoiding crashes regardless of the size of the input array (perhaps with some other functions besides BufSend() and BufReceive() that I may have missed?), or are SciDB UDOs unable to handle inter-instance data passing whose size scales with the size of the array?


#2

To be clear, I ideally want this operator to work for any size array even if that means it slows down considerably after the input array becomes so large; a slow operator could be made faster, but a crashing operator can’t really be optimized! My current plan of attack is to somehow utilize mutexes/CVs to prevent an instance from sending its data if too much data is already being sent (and has not yet been received), but before I delve too deeply into that I’d like to know if there is already similar functionality built into SciDB somewhere. If I do need to implement this myself, another thing I’ll need to know is how to determine what the maximum amount of data can be sent before SciDB crashes, and to be frank I don’t have any clue how to do that, haha!


#3

So if you read the function redistribute() in Operator.cpp you see it attempts to address these very issues:

  1. The data to send comes from an array, it’s the client’s responsibility to ensure that array is properly spilled to disk
  2. The data to receive is also put into an array. That’s a MemArray. Despite the name, this object will write chunks to disk when they are not used, thus not keeping all data in memory
  3. There are barrier and sync calls inserted, in a (not too advanced) attempt to limit the size of outstanding network traffic. AKA “throttle control”. You will see it’s related to the network-buffer setting. At this moment, one of our developers is prototyping a way to make this step more sophisticated and efficient.

So those three things make life somewhat more manageable. And so, one recommendation for what you’re doing is to write the data into an array (of some shape) and simply call redistribute( psLocalInstance, 0) to send everything to coordinator. That might work. But – when writing data to the arrays you have to be careful about coordinates. If redistribute sees two cells at the same position (i.e. {10, 51}) it will absolutely “merge” them by picking an arbitrary value and ignoring the other. One potential solution is to use “instance_id” as a dimension in your array. And, of course you become limited to the scidb array datatypes, but you can definitely shove whatever you like into strings…

Make sense?


#4

how to send a map to other instances? thanks.