[SPARK-15994] [MESOS] Allow enabling Mesos fetch cache in coarse executor backend #13713

drcrallen · 2016-06-16T18:11:26Z

Mesos 0.23.0 introduces a Fetch Cache feature http://mesos.apache.org/documentation/latest/fetcher/ which allows caching of resources specified in command URIs.

This patch:

Updates the Mesos shaded protobuf dependency to 0.23.0
Allows setting spark.mesos.fetcherCache.enable to enable the fetch cache for all specified URIs. (URIs must be specified for the setting to have any affect)
Updates documentation for Mesos configuration with the new setting.

This patch does NOT:

Allow for per-URI caching configuration. The cache setting is global to ALL URIs for the command.

Mesos 0.23.0 introduces a Fetch Cache feature http://mesos.apache.org/documentation/latest/fetcher/ which allows caching of resources specified in command URIs. This patch: * Updates the Mesos shaded protobuf dependency to 0.23.0 * Allows setting `spark.mesos.fetchCache.enable` to enable the fetch cache for all specified URIs. (URIs must be specified for the setting to have any affect) * Updates documentation for Mesos configuration with the new setting. This patch does NOT: * Allow for per-URI caching configuration. The cache setting is global to ALL URIs for the command.

tnachen · 2016-06-22T03:10:58Z

ok to test

SparkQA · 2016-06-22T03:16:11Z

Test build #60998 has finished for PR 13713 at commit 1ebfaf6.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds no public classes.

drcrallen · 2016-06-29T20:42:08Z

@tnachen any comments?

SparkQA · 2016-07-27T17:45:57Z

Test build #62931 has finished for PR 13713 at commit a9b735c.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds no public classes.

tnachen · 2016-07-27T20:57:36Z

We also fetch URIs for running drivers in cluster mode (MesosClusterScheduler.scala). I'm thinking we should also allow this configuration to effect that too.

SparkQA · 2016-07-30T04:16:30Z

Test build #63032 has finished for PR 13713 at commit 974245f.

This patch fails build dependency tests.
This patch does not merge cleanly.
This patch adds no public classes.

drcrallen · 2016-07-30T15:53:07Z

OoooOOO master updated to 1.0.0

Fixed merge conflicts

drcrallen · 2016-07-30T15:56:58Z

@tnachen this test is a little larger than I originally anticipated. Let me see if I can add some unit tests

SparkQA · 2016-07-30T17:55:38Z

Test build #63046 has finished for PR 13713 at commit bad9d69.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class SparkConf(loadDefaults: Boolean) extends Cloneable with Logging with Serializable

tnachen · 2016-10-08T23:51:36Z

@drcrallen Are you still planning to update this? It's quite a useful feature, so hoping this can get in. Also since Fine grain mode is depcreated I don't think we need to update it too.

drcrallen · 2016-10-10T14:43:55Z

I can update this today.

drcrallen · 2016-10-10T22:11:40Z

I'm still adding a basic test for this.

SparkQA · 2016-10-10T22:17:39Z

Test build #66688 has finished for PR 13713 at commit 456d28f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

drcrallen · 2016-10-10T22:31:56Z

Tested with mvn test -Pmesos -pl mesos -Dtest=MesosCoarseGrainedSchedulerBackendSuite

drcrallen · 2016-10-10T22:32:27Z

@tnachen let me know if there's any outstanding issues here.

tnachen · 2016-10-10T22:36:08Z

...scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackendSuite.scala

    assert(launchedTasks.head.getCommand.getUrisList.asScala(0).getValue == url)
  }

+  test("mesos supports setting fetcher") {


s/supports setting fetcher/supports setting fetcher cache/g

tnachen · 2016-10-10T22:37:37Z

docs/running-on-mesos.md

+  <td><code>spark.mesos.fetchCache.enable</code></td>
+  <td><code>false</code></td>
+  <td>
+    If set to `true`, all URIs in `spark.mesos.uris` will be eligible for caching by the [Mesos fetch cache](http://mesos.apache.org/documentation/latest/fetcher/)


From the implementation you actually set all downloadable URIs (like spark.executor.uri, jarUrl, etc) to be fetcher cachable. I think we need to be more explicit here that it's more than just spark.mesos.uris

Added more info.

tnachen · 2016-10-10T22:37:59Z

Other than the 2 comments, the changes LGTM. @mgummelt @srowen

SparkQA · 2016-10-10T22:54:13Z

Test build #66690 has finished for PR 13713 at commit 97c0d30.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-10-11T02:45:11Z

Test build #66703 has finished for PR 13713 at commit 54ae86b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

mgummelt · 2016-10-11T17:31:24Z

docs/running-on-mesos.md

+  <td><code>spark.mesos.fetchCache.enable</code></td>
+  <td><code>false</code></td>
+  <td>
+    If set to `true`, all URIs (example: `spark.executor.uri`, `spark.mesos.uris`) will be eligible for caching by the [Mesos fetch cache](http://mesos.apache.org/documentation/latest/fetcher/)


s/fetch/fetcher

s/eligible for caching/cached (they're not just eligible, they are in fact cached)

mgummelt · 2016-10-11T17:32:10Z

docs/running-on-mesos.md

-
-
+<tr>
+  <td><code>spark.mesos.fetchCache.enable</code></td>


s/fetchCache/fetcherCache (et al)

mgummelt · 2016-10-11T17:32:45Z

mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala

  private val queuedCapacity = conf.getInt("spark.mesos.maxDrivers", 200)
  private val retainedDrivers = conf.getInt("spark.mesos.retainedDrivers", 200)
  private val maxRetryWaitTime = conf.getInt("spark.mesos.cluster.retry.wait.max", 60) // 1 minute
+  private val useFetchCache = conf.getBoolean("spark.mesos.fetchCache.enable", false)


We really need to factor out all these conf vars like YARN does at some point

Good idea but outside scope of this PR

mgummelt · 2016-10-11T17:34:11Z

...scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackendSuite.scala

+    ), false)
+    val offers = List(Resources(backend.executorMemory(sc), 1))
+    offerResources(offers)
+    // Don't crash!


it originally didn't do much except check that it didn't fail. I have since added proper checking of the task info for if the fetcher cache was enabled. Will remove

mgummelt · 2016-10-11T17:34:45Z

...c/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosFineGrainedSchedulerBackend.scala


  private[mesos] val mesosExecutorCores = sc.conf.getDouble("spark.mesos.mesosExecutor.cores", 1)

+  private val useFetchCache = sc.conf.getBoolean("spark.mesos.fetchCache.enable", false)


Please remove changes to the fine grained scheduler. It's deprecated, and we shouldn't be adding any new features to it.

SparkQA · 2016-10-12T23:02:36Z

Test build #66850 has finished for PR 13713 at commit cc3901e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-10-12T23:03:06Z

Test build #66852 has finished for PR 13713 at commit fc0802c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

mgummelt · 2016-10-13T17:42:36Z

mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerUtils.scala

+                useFetchCache: Boolean = false): Unit = {
    uris.split(",").foreach { uri =>
-      builder.addUris(CommandInfo.URI.newBuilder().setValue(uri.trim()))
+      builder.addUris(CommandInfo.URI.newBuilder().setValue(uri.trim()).setCache(useFetchCache))


s/fetch/fetcher

mgummelt · 2016-10-13T17:43:10Z

LGTM @srowen

SparkQA · 2016-10-13T20:17:22Z

Test build #66909 has finished for PR 13713 at commit 7195c6e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

drcrallen · 2016-10-25T17:42:20Z

@mgummelt do you know what still needs to be done to get this in?

mgummelt · 2016-10-31T17:35:40Z

We need to get @srowen or one of the other committers to merge it.

srowen · 2016-11-01T13:14:26Z

Merged to master

…tor backend Mesos 0.23.0 introduces a Fetch Cache feature http://mesos.apache.org/documentation/latest/fetcher/ which allows caching of resources specified in command URIs. This patch: - Updates the Mesos shaded protobuf dependency to 0.23.0 - Allows setting `spark.mesos.fetcherCache.enable` to enable the fetch cache for all specified URIs. (URIs must be specified for the setting to have any affect) - Updates documentation for Mesos configuration with the new setting. This patch does NOT: - Allow for per-URI caching configuration. The cache setting is global to ALL URIs for the command. Author: Charles Allen <charles@allen-net.com> Closes apache#13713 from drcrallen/SPARK15994.

Merge remote-tracking branch 'apache/master' into SPARK15994

a9b735c

Add fetch cache to more places

974245f

Merge remote-tracking branch 'apache/master' into SPARK15994

bad9d69

Merge remote-tracking branch 'apache/master' into SPARK15994

456d28f

Add simple URI test

97c0d30

tnachen reviewed Oct 10, 2016

View reviewed changes

Update docs per review comments

54ae86b

mgummelt suggested changes Oct 11, 2016

View reviewed changes

drcrallen added 2 commits October 12, 2016 15:31

Cleanup as per review comments

cc3901e

More concrete language in docs on fetcher cache

fc0802c

mgummelt reviewed Oct 13, 2016

View reviewed changes

Fix missing fetch -> fetcher rename

7195c6e

asfgit closed this in e34b4e1 Nov 1, 2016

drcrallen deleted the SPARK15994 branch November 1, 2016 18:03


		private[mesos] val mesosExecutorCores = sc.conf.getDouble("spark.mesos.mesosExecutor.cores", 1)

		private val useFetchCache = sc.conf.getBoolean("spark.mesos.fetchCache.enable", false)



		<tr>
		<td><code>spark.mesos.fetchCache.enable</code></td>

[SPARK-15994] [MESOS] Allow enabling Mesos fetch cache in coarse executor backend #13713

[SPARK-15994] [MESOS] Allow enabling Mesos fetch cache in coarse executor backend #13713

Uh oh!

Conversation

drcrallen commented Jun 16, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tnachen commented Jun 22, 2016

Uh oh!

SparkQA commented Jun 22, 2016

Uh oh!

drcrallen commented Jun 29, 2016

Uh oh!

SparkQA commented Jul 27, 2016

Uh oh!

tnachen commented Jul 27, 2016

Uh oh!

SparkQA commented Jul 30, 2016

Uh oh!

drcrallen commented Jul 30, 2016

Uh oh!

drcrallen commented Jul 30, 2016

Uh oh!

SparkQA commented Jul 30, 2016

Uh oh!

tnachen commented Oct 8, 2016

Uh oh!

drcrallen commented Oct 10, 2016

Uh oh!

drcrallen commented Oct 10, 2016

Uh oh!

SparkQA commented Oct 10, 2016

Uh oh!

drcrallen commented Oct 10, 2016

Uh oh!

drcrallen commented Oct 10, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tnachen commented Oct 10, 2016

Uh oh!

SparkQA commented Oct 10, 2016

Uh oh!

SparkQA commented Oct 11, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mgummelt Oct 11, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Oct 12, 2016

Uh oh!

SparkQA commented Oct 12, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

drcrallen commented Jun 16, 2016 •

edited

Loading

mgummelt Oct 11, 2016 •

edited

Loading