[SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown #28708

Closed

holdenk wants to merge 51 commits intoapache:masterfrom

holdenk:SPARK-20629-copy-shuffle-data-when-nodes-are-being-shutdown-cleaned-up

Commits on Jun 17, 2020

Add an option to migrate shuffle blocks as well as the current cache blocks during decommissioning
holdenk
committed
Update core/src/main/scala/org/apache/spark/storage/BlockManager.scala

holdenk
and
attilapiros
committed
Update core/src/main/scala/org/apache/spark/shuffle/IndexShuffleBlockResolver.scala

holdenk
and
attilapiros
committed
First pass at the feedback from @attilapiros, mostly minor re-arrangement
holdenk
committed
Saw a test failure which could come from us not having a reasonable timeout here
holdenk
committed
Improve error logging

holdenk
and
attilapiros
committed
cleanup

holdenk
and
attilapiros
committed
cleanup

holdenk
and
attilapiros
committed
Add more info to debugging

holdenk
and
attilapiros
committed
logging string interpolation

holdenk
and
attilapiros
committed
logging string interpolation

holdenk
and
attilapiros
committed
logging string interpolation

holdenk
and
attilapiros
committed
Generalize the decom put to check put as stream and shuffle blocks as well (help avoid cascading block migration)
holdenk
committed
spacing
holdenk
committed
Fix long line, make our shuffle block threads stop so we don't leak threads during testing.
holdenk
committed
Remove un-needed shuffleStatus.invalidateSerializedMapOutputStatusCache and log the scheduler when asked to decom and we can't
holdenk
committed
Always transfer shuffle blocks as put, take out the spark.network.maxRemoteBlockSizeFetchToMem test that we don't need anymore, add back in submitting the thread I accidently took out in applying s…
holdenk
committed
Logging and thread name cleanup
holdenk
committed
Try and fix up the K8s integration test a bit
holdenk
committed
add shuffle migration test to BlockManagerSuite

Attila Zsolt Piros
authored and
holdenk
committed
Use StorageLevel.DISK_ONLY instead of manually making our own, and store the execution service being used to run the threads so we can stop them explicitly. Still uses a graceful stop for other exe…
holdenk
committed
Cleanup thread leaks
holdenk
committed
Remove excess logging
holdenk
committed
Increase cohesion by moving methods and members into BlockManagerDecommissionManager

attilapiros
authored and
holdenk
committed
Remove ref to peerProvider we didn't port over
holdenk
committed
extract BlockManagerDecommissionManager into a toplevel class

attilapiros
authored and
holdenk
committed
Cleanup to just use the BM not the other interfaces
holdenk
committed
Some reordering and adding BlockManagerDecommissionManagerSuite

attilapiros
authored and
holdenk
committed
Rename the two different test suites for decommissioning so the difference is clear from the class name (integration-style/unit-style)
holdenk
committed
Various small cleanups including renaming BlockManagerDecommissionManager to BlockManagerDecommissioner
holdenk
committed
Stop the block decom manager at the end of the block decom unit test
holdenk
committed
Add an explanation about why we use producer/consumer here.
holdenk
committed
Add a case class for the shuffle block id + map id so we're more clear on what were passing around than just rabndom tuples
holdenk
committed
Fix the unit test
holdenk
committed
Code review feedback: add test for rejecting blocks with bad shuffle resolver, fix string interopolation, remove un-needed retry logic in decommissioning, etc.
holdenk
committed
Update warning message to use key from the config package
holdenk
committed
Used a (configurable) thread pool for shuffle migrations to allow users to limit the number of concurrent migrations. We still limit to one runnable per peer executor so we don't overwhelm any part…
holdenk
committed
Fix the K8s decom test to use the correct config params
holdenk
committed
Re-enable rest of k8s tests
holdenk
committed

Commits on Jun 29, 2020

Commits on Jul 14, 2020

Commits on Jul 15, 2020

Commits on Jul 16, 2020

Code review feedback, remove regex we don't need, reduce some log levels, add a Since annotaiton.
holdenk
committed

Commits on Jul 17, 2020