[SPARK-32917][SHUFFLE][CORE] Adds support for executors to push shuffle blocks after successful map task completion#30312

Closed

otterc wants to merge 35 commits intoapache:masterfrom

linkedin:SPARK-32917

Commits on Dec 4, 2020

LIHADOOP-48527 Enable Shuffle writer to push blocks to remote shuffle services.

Min Shen
authored and
otterc
committed
LIHADOOP-53496 Not logging all block push exceptions on the client
otterc
committed
LIHADOOP-54374 Separate the configurations for connection creation timeout and connection idle timeout

zhouyejoe
authored and
otterc
committed
LIHADOOP-54370 Not to retry on certain exceptions when pushing blocks
otterc
committed
Made the code compile, added more tests, introduced ShuffleBlockPushId
otterc
committed
Renamed isPushBasedShuffleEnabled to isPushShuffleEnabled and fixed the comment
otterc
committed
Added Since tags all the newly introduced classes
otterc
committed
Explicitly converting ArrayBuffer to Seq so that build with SBT succeeds
otterc
committed
Update core/src/main/scala/org/apache/spark/shuffle/ShuffleWriter.scala

otterc
and
Ngone51
committed
Addressed the typos and other comments
otterc
committed
Moved push of data out of ShuffleWriter and addressed other minor comments
otterc
committed
LIHADOOP-48527 Enable Shuffle writer to push blocks to remote shuffle services.

Min Shen
authored and
otterc
committed
LIHADOOP-54370 Not to retry on certain exceptions when pushing blocks
otterc
committed
Made the code compile, added more tests, introduced ShuffleBlockPushId
otterc
committed
Added Since tags all the newly introduced classes
otterc
committed
Moved push of data out of ShuffleWriter and addressed other minor comments
otterc
committed
Renamed PushShuffleSupport to PushShufleComponent and changed initiate push to be triggered explictly
otterc
committed
Renamed PushShuffleComponent to PushShuffleWriterComponent
otterc
committed
Not changing the default value of connectionCreationTimeout
otterc
committed
Renamed PushShuffleWriterComponent to ShuffleBlockPusher and addressed other comments
otterc
committed
Addressed review comments
otterc
committed
Fixed the documentation for numPushThreads
otterc
committed
Changed the initiateBlockPush method, added tests, and stop pushing completely when too late
otterc
committed
Update core/src/main/scala/org/apache/spark/shuffle/ShuffleBlockPusher.scala

otterc
and
Ngone51
committed
Addressed review comments
otterc
committed
Addressed review comments
otterc
committed
Changed the default of maxBlockSizeToPush to 1M
otterc
committed
Removed the stopPushing flag which is not needed
otterc
committed
Addressed review comments
otterc
committed

Commits on Dec 17, 2020

Commits on Dec 19, 2020