Skip to content

[SPARK-4642] Add description about spark.yarn.queue to running-on-YARN document. #3500

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
tsudukim wants to merge 3 commits into from
Closed

[SPARK-4642] Add description about spark.yarn.queue to running-on-YARN document. #3500

tsudukim wants to merge 3 commits into from

Conversation

@tsudukim
Copy link
Contributor

@tsudukim tsudukim commented Nov 28, 2014

Added descriptions about these parameters.

  • spark.yarn.queue

Modified description about the defalut value of this parameter.

  • spark.yarn.submit.file.replication

@tsudukim
[SPARK-4642] Documents about running-on-YARN needs update
88cac9b 
Added descriptions about these parameters.
- spark.yarn.report.interval
- spark.yarn.queue
- spark.yarn.user.classpath.first
- spark.yarn.scheduler.reporterThread.maxFailures

Modified description about the defalut value of this parameter.
- spark.yarn.submit.file.replication
@SparkQA
Copy link

SparkQA commented Nov 28, 2014

Test build #23927 has started for PR 3500 at commit 88cac9b.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Nov 28, 2014

Test build #23927 has finished for PR 3500 at commit 88cac9b.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

AmplabJenkins commented Nov 28, 2014

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23927/
Test PASSed.

@sryza
Copy link
Contributor

sryza commented Dec 1, 2014

Hi @tsudukim , spark.yarn.queue would be useful to document, but I think the others were left intentionally undocumented. Because the Spark-on-YARN internals still need some hardening, we'd like to avoid exposing some properties so that we can change them later.

@tgravescs

@tgravescs
Copy link
Contributor

tgravescs commented Dec 1, 2014

@sryza is correct. Most of those were intentionally left undocumented. If you have reason they need to be changed then we can revisit them sooner to make sure they are what we want and get them documented. Note there is a different pr up to fix up spark.yarn.user.classpath.first (#3233)

@SparkQA
Copy link

SparkQA commented Dec 2, 2014

Test build #24013 has started for PR 3500 at commit 21cf624.

  • This patch merges cleanly.

@tsudukim
Copy link
Contributor Author

tsudukim commented Dec 2, 2014

@sryza and @tgravescs Thank you for your review. I removed them. Only spark.yarn.queue is added.

@SparkQA
Copy link

SparkQA commented Dec 2, 2014

Test build #24013 has finished for PR 3500 at commit 21cf624.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

AmplabJenkins commented Dec 2, 2014

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24013/
Test PASSed.

sryza
sryza reviewed Dec 3, 2014
View reviewed changes
docs/running-on-yarn.md Outdated
Copy link
Contributor

@sryza sryza Dec 3, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A little better gramatically: "The name of the YARN queue to which the application is submitted."

@sryza
Copy link
Contributor

sryza commented Dec 3, 2014

Had one more nit. Otherwise this looks good to me.

@tsudukim
Copy link
Contributor Author

tsudukim commented Dec 3, 2014

@sryza Thank you for your helpful advice! Modified it.

@SparkQA
Copy link

SparkQA commented Dec 3, 2014

Test build #24098 has started for PR 3500 at commit ce99655.

  • This patch merges cleanly.

@andrewor14
Copy link
Contributor

andrewor14 commented Dec 3, 2014

Ok merging this into master and 1.2 thank you @tsudukim and reviewers

@andrewor14
Copy link
Contributor

andrewor14 commented Dec 3, 2014

Actually @tsudukim could you update the PR description and the title? Now this only documents spark.yarn.queue, and it would be good if these reflect the actual changes here. I will merge this once you make the modifications.

@tsudukim tsudukim changed the title [SPARK-4642] Documents about running-on-YARN needs update [SPARK-4642] Add description about spark.yarn.queue to running-on-YARN document. Dec 3, 2014
@tsudukim
Copy link
Contributor Author

tsudukim commented Dec 3, 2014

@andrewor14 Thank you!
I modified description and title of this tickets from this web ui, but I'm not sure if it's right where I modified. Is this OK?

@andrewor14
Copy link
Contributor

andrewor14 commented Dec 3, 2014

Alright I'm merging this into master 1.2 and 1.1 thanks

@SparkQA
Copy link

SparkQA commented Dec 3, 2014

Test build #24098 has finished for PR 3500 at commit ce99655.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

AmplabJenkins commented Dec 3, 2014

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24098/
Test PASSed.

asfgit pushed a commit that referenced this pull request Dec 3, 2014
@tsudukim
[SPARK-4642] Add description about spark.yarn.queue to running-on-YAR…
4a71e08 
…N document.

Added descriptions about these parameters.
- spark.yarn.queue

Modified description about the defalut value of this parameter.
- spark.yarn.submit.file.replication

Author: Masayoshi TSUZUKI <tsudukim@oss.nttdata.co.jp>

Closes #3500 from tsudukim/feature/SPARK-4642 and squashes the following commits:

ce99655 [Masayoshi TSUZUKI] better gramatically.
21cf624 [Masayoshi TSUZUKI] Removed intentionally undocumented properties.
88cac9b [Masayoshi TSUZUKI] [SPARK-4642] Documents about running-on-YARN needs update

(cherry picked from commit 692f493)
Signed-off-by: Andrew Or <andrew@databricks.com>
asfgit pushed a commit that referenced this pull request Dec 3, 2014
@tsudukim
[SPARK-4642] Add description about spark.yarn.queue to running-on-YAR…
3e3cd5a 
…N document.

Added descriptions about these parameters.
- spark.yarn.queue

Modified description about the defalut value of this parameter.
- spark.yarn.submit.file.replication

Author: Masayoshi TSUZUKI <tsudukim@oss.nttdata.co.jp>

Closes #3500 from tsudukim/feature/SPARK-4642 and squashes the following commits:

ce99655 [Masayoshi TSUZUKI] better gramatically.
21cf624 [Masayoshi TSUZUKI] Removed intentionally undocumented properties.
88cac9b [Masayoshi TSUZUKI] [SPARK-4642] Documents about running-on-YARN needs update
@asfgit asfgit closed this in 692f493 Dec 3, 2014
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@tsudukim @SparkQA @AmplabJenkins @sryza @tgravescs @andrewor14