[SPARK-4879] Use driver to coordinate Hadoop output committing for speculative tasks#4066

Closed

JoshRosen wants to merge 38 commits intoapache:masterfrom

JoshRosen:SPARK-4879-sparkhadoopwriter-fix

Commits on Jan 21, 2015

[SPARK-4879] Use the Spark driver to authorize Hadoop commits.
mccheah
committed

Commits on Jan 26, 2015

Using more concurrency to process OutputCommitCoordinator requests.
mccheah
committed

Commits on Jan 27, 2015

Commits on Jan 29, 2015

Commits on Feb 3, 2015

Only coordinate when speculation is enabled; add configuration option to bypass new coordination.
JoshRosen
committed
(Temporarily) re-enable “always coordinate” for testing purposes.
JoshRosen
committed
Bug fix: use task ID instead of StageID to index into authorizedCommitters.
JoshRosen
committed
Merge remote-tracking branch 'origin/master' into SPARK-4879-sparkhadoopwriter-fix
JoshRosen
committed
Some misc. code style + doc changes:
JoshRosen
committed
Move CommitDeniedException to executors package; remove `@DeveloperAPI` annotation.
JoshRosen
committed
Roll back TaskSetManager changes that broke other tests.
JoshRosen
committed
Roll back unnecessary DAGSchedulerSingleThreadedProcessLoop refactoring:
JoshRosen
committed
Create fake TaskInfo using dummy fields instead of Mockito.
JoshRosen
committed
Some cleanup in OutputCommitCoordinatorSuite
JoshRosen
committed

Commits on Feb 4, 2015

Commits on Feb 5, 2015

Roll back copiesRunning change in TaskSetManager
JoshRosen
committed

Commits on Feb 10, 2015

Commits on Feb 11, 2015