Skip to content

[SPARK-3099] Staging Directory is never deleted when we run job with YARN Client Mode #2017

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
sarutak wants to merge 2 commits into from
Closed

[SPARK-3099] Staging Directory is never deleted when we run job with YARN Client Mode #2017

sarutak wants to merge 2 commits into from

Conversation

@sarutak
Copy link
Member

@sarutak sarutak commented Aug 18, 2014

No description provided.

@SparkQA
Copy link

SparkQA commented Aug 18, 2014

QA tests have started for PR 2017 at commit 31aac69.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Aug 18, 2014

QA tests have finished for PR 2017 at commit 31aac69.

  • This patch fails unit tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • class AppMasterShutdownHook(appMaster: ExecutorLauncher) extends Runnable
    • class AppMasterShutdownHook(appMaster: ExecutorLauncher) extends Runnable

@sarutak
Copy link
Member Author

sarutak commented Aug 18, 2014

Jenkins, retest this please

@SparkQA
Copy link

SparkQA commented Aug 18, 2014

QA tests have started for PR 2017 at commit 31aac69.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Aug 18, 2014

QA tests have finished for PR 2017 at commit 31aac69.

  • This patch fails unit tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • class AppMasterShutdownHook(appMaster: ExecutorLauncher) extends Runnable
    • class AppMasterShutdownHook(appMaster: ExecutorLauncher) extends Runnable

@sarutak
Copy link
Member Author

sarutak commented Aug 19, 2014

Jenkins, retest this please.

@SparkQA
Copy link

SparkQA commented Aug 19, 2014

QA tests have started for PR 2017 at commit 31aac69.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Aug 19, 2014

QA tests have finished for PR 2017 at commit 31aac69.

  • This patch passes unit tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • class AppMasterShutdownHook(appMaster: ExecutorLauncher) extends Runnable
    • class AppMasterShutdownHook(appMaster: ExecutorLauncher) extends Runnable

@sarutak sarutak changed the title [SPARK-3099] Add a shutdown hook for cleanup staging directory to ExecutorLauncher [SPARK-3099] Staging Directory is never deleted when we run job with YARN Client Mode Aug 19, 2014
@sarutak
Copy link
Member Author

sarutak commented Aug 19, 2014

#2022 changed ExecutorLauncher so rebased this PR.

@SparkQA
Copy link

SparkQA commented Aug 19, 2014

QA tests have started for PR 2017 at commit 3c0ae2f.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Aug 19, 2014

QA tests have finished for PR 2017 at commit 3c0ae2f.

  • This patch passes unit tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • class AppMasterShutdownHook(appMaster: ExecutorLauncher) extends Runnable
    • class AppMasterShutdownHook(appMaster: ExecutorLauncher) extends Runnable

@vanzin
Copy link
Contributor

vanzin commented Aug 19, 2014

Hi @sarutak,

I refactored a lot of this code as part of #2020, I'm pretty sure I fixed this issue in the process. Could you take a look? Thanks!

@sarutak
Copy link
Member Author

sarutak commented Aug 20, 2014

Hi @vanzin ,

Thanks, I'll test it.

@sarutak
Copy link
Member Author

sarutak commented Aug 29, 2014

Closing this PR because #2020 resolved this issue.

@sarutak sarutak closed this Aug 29, 2014
@sarutak sarutak deleted the SPARK-3099 branch April 11, 2015 05:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@sarutak @SparkQA @vanzin