[SPARK-2165] spark on yarn: add support for setting maxAppAttempts in the ApplicationSubmissionContext

[knusbaum]

No description provided.

[AmplabJenkins]

Can one of the admins verify this patch?

[tgravescs]

We haven't been adding specific routines to set the configs. The user can just set it using the existing SparkConf.set routines so I think we should remove this.

[tgravescs]

lets make the config name yarn specific. spark.yarn.maxappattempts.

Also can you update the yarn documentation to include the new config and description. lookin docs/running-on-yarn.md

[tgravescs]

Jenkins, test this please.

[SparkQA]

QA tests have started for PR 1279 at commit f848797.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 1279 at commit f848797.

• This patch fails unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class OutputFaker(output: Seq[Attribute], child: SparkPlan) extends SparkPlan
  • implicit class LogicalPlanHacks(s: SchemaRDD)
  • implicit class PhysicalPlanHacks(originalPlan: SparkPlan)
  • class FakeParquetSerDe extends SerDe

[tgravescs]

Something must be wrong with the QA box as this patch doesn't add any classes. The test failure is unrelated to this patch also.

[tgravescs]

we should change the debug statement to say something like spark.yarn.maxappattempts is not set, using cluster default...

[tgravescs]

Jenkins, test this please.

[SparkQA]

QA tests have started for PR 1279 at commit f848797.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 1279 at commit f848797.

• This patch passes unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[tgravescs]

So one thing I just realized we need to update is in the ApplicationMaster we are using the maxAppAttempts to determine if its the last AM retry. Currently its just grabbing the cluster maximum. We need to update that to handle this config.

[SparkQA]

Can one of the admins verify this patch?

[andrewor14]

I would prefer this to be called spark.yarn.applicationSubmissionMaxAttempts or something to be more specific

[tgravescs]

its not the number of application submission attempts though, this is how many times the resource manager will retry the application for you. to me application submission is done by the client, not a retry from the RM. what isn't clear about the current one so we can come up with something more clear? We could do something like ApplicationMasterMaxAttempts, am-maxattempts, rmappattempts

[andrewor14]

I see, our client only submits it once, but it's up to the RM how many attempts it makes to set up state for the application or launch the AM container etc. (The docs at https://hadoop.apache.org/docs/current/api/org/apache/hadoop/yarn/api/records/ApplicationSubmissionContext.html#setMaxAppAttempts(int) are pretty misleading...)

Then maybe it makes sense to call this spark.yarn.application.rmMaxAttempts or something? Or spark.yarn.amMaxAttempts. Or actually even just spark.yarn.applicationMaxAttempts sound alright now after considering the more verbose alternatives.

[tgravescs]

Ah and I see where you got the submissions from. I'm fine with any of those also. We'll see if Kyle responds otherwise I might take this over. We should just make sure to document it we'll so user understands.

[andrewor14]

This has merge conflicts, @knusbaum can you rebase on master?

[tgravescs]

Also note Kyle was intern at yahoo but has went back to school not sure if he will have time to continue this. We can wait to see if he responds

[pwendell]

Okay let's close this issue for now and he can reopen it if he has time.

[WangTaoTheTonic]

Looks like Kyle @knusbaum is busy, I will take this over if everyone is ok with it.
#3878