[MESOS][SPARK-8798] Allow additional uris to be fetched with mesos

[tnachen]

Some users like to download additional files in their sandbox that they can refer to from their spark program, or even later mount these files to another directory.

[tnachen]

@andrewor14 :)

[SparkQA]

Test build #36418 has finished for PR 7195 at commit 50aa382.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class Heartbeat(workerId: String, worker: RpcEndpointRef) extends DeployMessage
  • case class RegisteredWorker(master: RpcEndpointRef, masterWebUiUrl: String) extends DeployMessage
  • case class RegisterApplication(appDescription: ApplicationDescription, driver: RpcEndpointRef)
  • case class RegisteredApplication(appId: String, master: RpcEndpointRef) extends DeployMessage
  • case class SubmitDriverResponse(
  • case class KillDriverResponse(
  • case class MasterChanged(master: RpcEndpointRef, masterWebUiUrl: String)
  • class DCT(override val uid: String)
  • class MinMaxScaler(override val uid: String)
  • class PCA (override val uid: String) extends Estimator[PCAModel] with PCAParams
  • class StreamingLinearAlgorithm(object):
  • class StreamingLinearRegressionWithSGD(StreamingLinearAlgorithm):
  • class AnalysisException(Exception):
  • class FlumeUtils(object):
  • case class Cast(child: Expression, dataType: DataType) extends UnaryExpression with Logging
  • trait ExpectsInputTypes
  • trait AutoCastInputTypes
  • abstract class BinaryExpression extends Expression with trees.BinaryNode[Expression]
  • abstract class BinaryOperator extends BinaryExpression
  • trait ExtractValue
  • abstract class ExtractValueWithStruct extends UnaryExpression with ExtractValue
  • abstract class ExtractValueWithOrdinal extends BinaryExpression with ExtractValue
  • abstract class BinaryArithmetic extends BinaryOperator
  • class SpecificOrdering extends $
  • class SpecificProjection extends $
  • final class SpecificRow extends $
  • case class CreateNamedStruct(children: Seq[Expression]) extends Expression
  • case class Hex(child: Expression) extends UnaryExpression with Serializable
  • case class ShiftLeft(left: Expression, right: Expression) extends BinaryExpression
  • case class ShiftRight(left: Expression, right: Expression) extends BinaryExpression
  • case class UnHex(child: Expression) extends UnaryExpression with Serializable
  • case class Sha1(child: Expression) extends UnaryExpression with AutoCastInputTypes
  • case class Crc32(child: Expression)
  • case class Not(child: Expression) extends UnaryExpression with Predicate with AutoCastInputTypes
  • abstract class BinaryComparison extends BinaryOperator with Predicate
  • trait StringRegexExpression extends AutoCastInputTypes
  • trait CaseConversionExpression extends AutoCastInputTypes
  • trait StringComparison extends AutoCastInputTypes
  • case class StringLength(child: Expression) extends UnaryExpression with AutoCastInputTypes
  • // compiled class file for the closure here will conflict with the one in callUDF (upper case).

[tnachen]

jenkins retest please

[andrewor14]

retest this please

[SparkQA]

Test build #36430 has finished for PR 7195 at commit 50aa382.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class CreateNamedStruct(children: Seq[Expression]) extends Expression
  • case class ShiftLeft(left: Expression, right: Expression) extends BinaryExpression
  • case class ShiftRight(left: Expression, right: Expression) extends BinaryExpression

[andrewor14]

this is duplicated 3 times. Do you think the refactoring your other patch (roles principals) will help? Should we merge that one first?

[dragos]

There's a common superclass of all these three classes called MesosSchedulerUtils. I think this code could go there.

[SparkQA]

Test build #37757 has finished for PR 7195 at commit 42e2ee2.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #39941 has finished for PR 7195 at commit 741548f.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[andrewor14]

Merging into master 1.5