What is the role of Sync() during redistribute()?


Dear all,

First of all, Thank you for the last issue about redimension failure.

I have some question about redistribute().

What is the role of Sync() during redistribute()?
redistribute() calls sync() after chunk is sent to any instance if totalBytesSent is bigger than totalBytesSynced + networkBufferLimit.
I think this means to call Sync() if chunk which doesn’t be synchronized is sent over size of networkbufferlimit.
If so, What is to synchronize?

And, I wonder why sync() and barrier() are needed.

Thank you.


That code is fairly old, primitive logic that aims to avoid overwhelming someone. What it does is:

The sync gives the receiver a chance to respond with a “I am ready for more”. The sender waits for the sync response before sending more. Otherwise the receiver may become hosed.

It’s notable that 14.12 adds a new kind of “streaming redistribute” that uses a more sophisticated flow control. The old redistribute is still used in a few places but the plan is to phase it out over time.