[SPARK-31197][CORE] Shutdown executor once we are done decommissioning#29211

Closed

holdenk wants to merge 20 commits intoapache:masterfrom

holdenk:SPARK-31197-exit-execs-redone

Commits on Aug 5, 2020

Shutdown executor once we are done decommissioning
holdenk
committed
Whitespace cleanup
holdenk
committed
Code review feedback, various cleanups
holdenk
committed
Just use new thread and set daemon since runInNewThread does a join we don't want.
holdenk
committed
Update the text since 10 seconds is quick but not as fast as we can
holdenk
committed
rename WorkerDecommission message to DecommissionWorker. (CR feedback)
holdenk
committed
Revert "rename WorkerDecommission message to DecommissionWorker. (CR feedback)"
holdenk
committed
Fix it so we don't block the decom message
holdenk
committed
Was thinking a little bit, if we move the sleep up a bit we can unify it. The only cost is we _might_ wait an extra one second in the event that storage decom is not enabled and there is no running…
holdenk
committed
Fix
holdenk
committed
Intentionally don't start the tread to verify the executor is being shut down as planned
holdenk
committed
Revert "Intentionally don't start the tread to verify the executor is being shut down as planned"
holdenk
committed
fix thread start
holdenk
committed
Use tryAcquire so we don't block forever (CR feedback)
holdenk
committed
Since we shut down & then re-launch the executors now as decommissioning (unless we specify not to relaunch) we no longer expect the task to fail.
holdenk
committed
The executor can also have exited completely and then there is no entry
holdenk
committed
Try and write some comments
holdenk
committed
Add unit tests for how we handle timestamp tracking of migration times and add some more explicit cases where the migration being done remains valid and has the max value timestamp (e.g. the migrat…
holdenk
committed
Code review feedback, mostly cleanup the unit tests
holdenk
committed
CR feedback, replace some magic values with option in the tests to improve readability.
holdenk
committed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[SPARK-31197][CORE] Shutdown executor once we are done decommissioning#29211

[SPARK-31197][CORE] Shutdown executor once we are done decommissioning#29211
holdenk wants to merge 20 commits intoapache:masterfrom
holdenk:SPARK-31197-exit-execs-redone

Commits on Aug 5, 2020

Shutdown executor once we are done decommissioning

Whitespace cleanup

Code review feedback, various cleanups

Just use new thread and set daemon since runInNewThread does a join we don't want.

Update the text since 10 seconds is quick but not as fast as we can

rename WorkerDecommission message to DecommissionWorker. (CR feedback)

Revert "rename WorkerDecommission message to DecommissionWorker. (CR feedback)"

Fix it so we don't block the decom message

Was thinking a little bit, if we move the sleep up a bit we can unify it. The only cost is we _might_ wait an extra one second in the event that storage decom is not enabled and there is no running…

Fix

Intentionally don't start the tread to verify the executor is being shut down as planned

Revert "Intentionally don't start the tread to verify the executor is being shut down as planned"

fix thread start

Use tryAcquire so we don't block forever (CR feedback)

Since we shut down & then re-launch the executors now as decommissioning (unless we specify not to relaunch) we no longer expect the task to fail.

The executor can also have exited completely and then there is no entry

Try and write some comments

Add unit tests for how we handle timestamp tracking of migration times and add some more explicit cases where the migration being done remains valid and has the max value timestamp (e.g. the migrat…

Code review feedback, mostly cleanup the unit tests

CR feedback, replace some magic values with option in the tests to improve readability.

[SPARK-31197][CORE] Shutdown executor once we are done decommissioning#29211holdenk wants to merge 20 commits intoapache:masterapache/spark:masterfrom holdenk:SPARK-31197-exit-execs-redoneholdenk/spark:SPARK-31197-exit-execs-redoneCopy head branch name to clipboard

Commits

Commits on Aug 5, 2020

[SPARK-31197][CORE] Shutdown executor once we are done decommissioning#29211
holdenk wants to merge 20 commits intoapache:masterfrom
holdenk:SPARK-31197-exit-execs-redone