[SPARK-32916][SHUFFLE] Implementation of shuffle service that leverages push-based shuffle in YARN deployment mode #30062

Closed

otterc wants to merge 35 commits intoapache:masterfrom

linkedin:SPARK-32916

Commits on Nov 6, 2020

LIHADOOP-48527 Magnet shuffle service block transfer netty protocol

Victsm
authored and
otterc
committed
LIHADOOP-53438 Using different appId for the tests in RemoteBlockPushResolverSuite
otterc
committed
LIHADOOP-53496 Not logging all block push exceptions on the client
otterc
committed
LIHADOOP-53700 Separate configuration for caching the merged index files in NM

zhouyejoe
authored and
otterc
committed
LIHADOOP-53940 Logging the data file and index file path when shuffle service is unable to create them
otterc
committed
LIHADOOP-54059 LIHADOOP-53496 Handle the inconsistencies between local dirs provided to executor and the shuffle service and not log all exceptions at error/warning level
otterc
committed
LIHADOOP-54379 Sorting the disks both on shuffle service and executors
otterc
committed
LIHADOOP-52494 Magnet fallback to origin shuffle blocks when fetch of a shuffle chunk fails
otterc
committed
LIHADOOP-55372 reduced the default for minChunkSizeInMergedShuffleFile
otterc
committed
LIHADOOP-55315 Avoid network when fetching merged shuffle file in local host with a consistent view of app local dirs among different executors

zhouyejoe
authored and
otterc
committed
LIHADOOP-55654 Duplicate application init calls trigger NPE and wrong local dirs update in shuffle service. Also fixing a memory leak.

zhouyejoe
authored and
otterc
committed
Further prune changes that should go into a later PR.

Victsm
authored and
otterc
committed
LIHADOOP-54379 Sorting the disks both on shuffle service and executors
otterc
committed
LIHADOOP-55022 Disable the merged shuffle file cleanup in stopApplication event in YarnShuffleService as NM is able to cleanup the app local dir in Hadoop 2.10

zhouyejoe
authored and
otterc
committed
Tests and cleanup
otterc
committed
LIHADOOP-55948 Failure in the push stream should not change the currentMap if it's not the active stream
otterc
committed
Minor style corrections
otterc
committed
Fixed style issues
otterc
committed
Renamed variables, methods, fixed indentation, addressed other review comments
otterc
committed
Addressing review comments
otterc
committed
Changed the partitions map and addressed other review comments
otterc
committed
Added support for subdirs under merge_manager dirs and removed the yarn dir pattern
otterc
committed
Addressed test failure and other review comments in RemoteBlockPushResolver
otterc
committed
Minor change in finalization
otterc
committed
Removing the partition from inner map after the files are closed
otterc
committed
Server side configuration to specify the implementation of MergedShuffleFileManager
otterc
committed
Change the Push block stream to not encode shuffle Id, map index, and reduceId in a string
otterc
committed
Fixed typos, address review comments, made NoOp the default impl, and addressed the issue of race conditions
otterc
committed
Addressed review comments
otterc
committed
Fix IndexOutOfBoundsException and avoid instantiating AppsPathInfo multiple times
otterc
committed
Removed duplicate declaration of shuffle push prefix
otterc
committed
Added UT for collision with 2 streams
otterc
committed
Removed unnecessary TODOs, marked MergedShuffleFileManager evolving, added since tag to all the new classes introduced
otterc
committed
Changed the serialization on chunktracker and removed serializedSizeInBytes from the bitmap.encode
otterc
committed

Commits on Nov 8, 2020

Use RandomAccessFile
otterc
committed