[SPARK-23146][K8S] Support client mode. #21748

mccheah · 2018-07-11T18:02:55Z

What changes were proposed in this pull request?

Support client mode for the Kubernetes scheduler.

Client mode works more or less identically to cluster mode. However, in client mode, the Spark Context needs to be manually bootstrapped with certain properties which would have otherwise been set up by spark-submit in cluster mode. Specifically:

If the user doesn't provide a driver pod name, we don't add an owner reference. This is for usage when the driver is not running in a pod in the cluster. In such a case, the driver can only provide a best effort to clean up the executors when the driver exits, but cleaning up the resources is not guaranteed. The executor JVMs should exit if the driver JVM exits, but the pods will still remain in the cluster in a COMPLETED or FAILED state.
The user must provide a host (spark.driver.host) and port (spark.driver.port) that the executors can connect to. When using spark-submit in cluster mode, spark-submit generates the headless service automatically; in client mode, the user is responsible for setting up their own connectivity.

We also change the authentication configuration prefixes for client mode.

How was this patch tested?

Adding an integration test to exercise client mode support.

Client mode works more or less identically to cluster mode. However, in client mode, the Spark Context needs to be manually bootstrapped with certain properties which would have otherwise been set up by spark-submit in cluster mode. Specifically: - The user must provide a pod name for the driver. This implies that all drivers in client mode must be running inside a pod. This pod is primarily used to create the owner reference graph so that executors are not orphaned if the driver pod is deleted. - The user must provide a host (spark.driver.host) and port (spark.driver.port) that the executors can connect to. When using spark-submit in cluster mode, spark-submit generates the headless service automatically; in client mode, the user is responsible for setting up their own connectivity.

mccheah · 2018-07-11T18:05:03Z

TODO - finish verifying the integration test, and docs.

mccheah · 2018-07-11T18:06:58Z

test this please

SparkQA · 2018-07-11T18:22:59Z

Test build #92869 has finished for PR 21748 at commit 19618aa.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

mccheah · 2018-07-11T18:40:19Z

retest this please

SparkQA · 2018-07-11T18:52:26Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/861/

SparkQA · 2018-07-11T18:59:13Z

Test build #92873 has finished for PR 21748 at commit 19618aa.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-07-11T19:02:51Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/861/

SparkQA · 2018-07-11T19:03:45Z

Test build #92875 has finished for PR 21748 at commit 94ed1cc.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-07-11T19:13:40Z

Test build #92878 has finished for PR 21748 at commit 560993e.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-07-11T19:18:39Z

Test build #92879 has finished for PR 21748 at commit e961fd3.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-07-11T19:24:06Z

Test build #92880 has finished for PR 21748 at commit a00561f.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-07-11T19:33:51Z

Test build #92881 has finished for PR 21748 at commit a2609b0.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-07-11T19:38:27Z

Test build #92882 has finished for PR 21748 at commit 97f1284.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

mccheah · 2018-07-11T19:41:25Z

retest this please

SparkQA · 2018-07-11T19:53:55Z

Test build #92884 has finished for PR 21748 at commit 97f1284.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-07-11T20:03:17Z

Test build #92887 has finished for PR 21748 at commit 2205220.

This patch fails to build.
This patch merges cleanly.
This patch adds the following public classes (experimental):
public class JavaSummarizerExample
trait ComplexTypeMergingExpression extends Expression
sealed trait MultipleWatermarkPolicy
case class WatermarkTracker(policy: MultipleWatermarkPolicy) extends Logging

shaneknapp · 2018-07-11T20:55:58Z

i'm going to kill the ubuntu build and reboot the worker. i'll retrigger when it's back.

liyinan926 · 2018-07-20T18:50:59Z

docs/running-on-kubernetes.md

-and your spark driver's port to `spark.driver.port`.
+driver pod to be routable from the executors by a stable hostname. When deploying your headless service, ensure that
+the service's label selector will only match the driver pod and no other pods; it is recommended to assign your driver
+pod a sufficiently unique label and to use that label in the node selector of the headless service. Specify the driver's


s/node selector/label selector/.

liyinan926 · 2018-07-20T18:53:00Z

docs/running-on-kubernetes.md

-server fails for any reason, these pods will remain in the cluster. The executor processes should exit when they cannot
-reach the driver, so the executor pods should not consume resources in the cluster after your application exits.
+The driver will look for a pod with the given name in the namespace specified by `spark.kubernetes.namespace`, and
+all executor pods will have their owner reference field set to point to that pod. Be careful to avoid setting the


s/all executor pods will have their owner reference field set/a OwnerReference point to that pod will be added to each of the executor pods..

liyinan926 · 2018-07-20T18:53:27Z

docs/running-on-kubernetes.md

+The driver will look for a pod with the given name in the namespace specified by `spark.kubernetes.namespace`, and
+all executor pods will have their owner reference field set to point to that pod. Be careful to avoid setting the
+owner reference to a pod that is not actually that driver pod, or else the executors may be terminated prematurely when
+the wrong pod is terminated.


s/terminated/deleted/.

liyinan926 · 2018-07-20T18:55:18Z

docs/running-on-kubernetes.md

+actually running in a pod, keep in mind that the executor pods may not be deleted from the cluster when the application
+exits. The Spark scheduler attempts to delete these pods, but if the network request to the API server fails for any
+reason, these pods will remain in the cluster. The executor processes should exit when they cannot reach the driver, so
+the executor pods should not consume resources in the cluster after your application exits.


s/should not consume resources/should not consume compute resources (cpus and memory)/.

liyinan926 · 2018-07-20T18:56:31Z

docs/running-on-kubernetes.md

-reach the driver, so the executor pods should not consume resources in the cluster after your application exits.
+The driver will look for a pod with the given name in the namespace specified by `spark.kubernetes.namespace`, and
+all executor pods will have their owner reference field set to point to that pod. Be careful to avoid setting the
+owner reference to a pod that is not actually that driver pod, or else the executors may be terminated prematurely when


s/owner reference/OwnerReference/ for consistency.

liyinan926 · 2018-07-20T18:59:16Z

docs/running-on-kubernetes.md

+If your application is not running inside a pod, or if `spark.driver.pod.name` is not set when your application is
+actually running in a pod, keep in mind that the executor pods may not be deleted from the cluster when the application
+exits. The Spark scheduler attempts to delete these pods, but if the network request to the API server fails for any
+reason, these pods will remain in the cluster. The executor processes should exit when they cannot reach the driver, so


s/these pods will remain in the cluster/these pods may not get deleted properly/. There's a pod-specific GC that deletes terminated pods based on a cluster-wide capacity (by default 12500 pods). It sorts those pods by creation timestamp before deleting them. But this is unpredictable.

SparkQA · 2018-07-20T18:59:53Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/1180/

SparkQA · 2018-07-20T19:08:10Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/1180/

mccheah · 2018-07-20T19:09:05Z

@liyinan926 did some of my own edits on top of your suggestions for docs wording on the latest patch.

SparkQA · 2018-07-20T19:19:28Z

Test build #93358 has finished for PR 21748 at commit d90f753.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

mccheah · 2018-07-20T19:19:37Z

test this please

SparkQA · 2018-07-20T19:25:17Z

Test build #93359 has finished for PR 21748 at commit 001a525.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

mccheah · 2018-07-20T19:38:07Z

Anyone know what's happening with this:

[error] /home/jenkins/workspace/testing-k8s-prb-make-spark-distribution-unified/external/avro/src/test/scala/org/apache/spark/sql/avro/SerializableSchemaSuite.scala:39: Symbol 'term org.scalacheck' is missing from the classpath.
[error] This symbol is required by 'method org.scalatest.prop.Configuration.getParams'.
[error] Make sure that term scalacheck is in your classpath and check for conflicting dependencies with `-Ylog-classpath`.
[error] A full rebuild may help if 'Configuration.class' was compiled against an incompatible version of org.
[error]     serializer.deserialize[Any](serialized) match {
[error]                                ^
[error] one error found
[error] Compile failed at Jul 20, 2018 12:33:14 PM [1.370s]

@shaneknapp

SparkQA · 2018-07-20T19:48:54Z

Test build #93361 has finished for PR 21748 at commit 72c96e0.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

mccheah · 2018-07-20T19:51:11Z

Never mind, think it's recovering now.

liyinan926 · 2018-07-20T20:11:17Z

LGTM for the docs updates.

SparkQA · 2018-07-20T20:25:29Z

Test build #93360 has finished for PR 21748 at commit 72c96e0.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-07-20T23:41:12Z

Test build #93357 has finished for PR 21748 at commit 086747e.

This patch passes all tests.
This patch does not merge cleanly.
This patch adds no public classes.

mccheah · 2018-07-23T20:15:47Z

Merging in a few hours if no additional comments are raised.

felixcheung

LGTM! minor goodness for documentation/example IMO would be great to have later

felixcheung · 2018-07-24T06:23:42Z

docs/running-on-kubernetes.md

+driver pod to be routable from the executors by a stable hostname. When deploying your headless service, ensure that
+the service's label selector will only match the driver pod and no other pods; it is recommended to assign your driver
+pod a sufficiently unique label and to use that label in the label selector of the headless service. Specify the driver's
+hostname via `spark.driver.host` and your spark driver's port to `spark.driver.port`.


@mccheah as for your comment #21748 (comment) so this manual setup is ok, right?

there are some level of complexity here - perhaps a quick follow up of some sample template/kubectl commands would be helpful

Yeah manual setup is fine for now. Think additional docs around how to do all this can be a separate PR.

felixcheung · 2018-07-24T06:24:53Z

docs/running-on-kubernetes.md

+actually running in a pod, keep in mind that the executor pods may not be properly deleted from the cluster when the
+application exits. The Spark scheduler attempts to delete these pods, but if the network request to the API server fails
+for any reason, these pods will remain in the cluster. The executor processes should exit when they cannot reach the
+driver, so the executor pods should not consume compute resources (cpu and memory) in the cluster after your application


executor processes should exit when they cannot reach the driver
what's the time out value? is it configurable?

Unclear, it triggers in the onDisconnected event so I think there's a persistent socket connection that's dropped that causes the exit. So, it should more or less be instantaneous.

felixcheung · 2018-07-24T06:26:22Z

...es/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterManager.scala

+        Some(new File(Config.KUBERNETES_SERVICE_ACCOUNT_CA_CRT_PATH)))
+    } else {
+      (KUBERNETES_AUTH_CLIENT_MODE_PREFIX,
+        masterURL.substring("k8s://".length()),


I thought there's some function for parsing the k8s master url?

We can make such a helper function, currently this logic is done here and in KubernetesClientApplication

mccheah · 2018-07-25T16:37:57Z

Ok after the next build passes I'm going to merge immediately. Thanks for the review.

SparkQA · 2018-07-25T16:52:12Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/1317/

SparkQA · 2018-07-25T16:59:40Z

Test build #93551 has finished for PR 21748 at commit ded1ff6.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-07-25T16:59:47Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/1317/

ifilonenko · 2018-07-25T18:14:51Z

@mccheah the integration tests did not include the ClientModeTestsSuite. Can you add with ClientModeTestsSuite to the KuberneteSuite else, the PRB doesn't actually test the client mode support accurately.

skonto · 2018-07-27T09:45:15Z

...-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/ClientModeTestsSuite.scala

+          .withLabels(labels.asJava)
+          .endMetadata()
+        .withNewSpec()
+          .withServiceAccountName("default")


@mccheah if people use spark-rbac.yaml this will fail. It fails for me. Shouldnt be hardcoded.
Error: "User "system:serviceaccount:spark:default" cannot get pods in the namespace "spark"."

Yup we can fix this

is there a JIRA?

I created one: https://issues.apache.org/jira/browse/SPARK-24963

mccheah changed the title ~~[SPARK-23146] Support client mode.~~ [SPARK-23146][K8S] Support client mode. Jul 11, 2018

mccheah added 2 commits July 11, 2018 11:46

Allow driver pod name to be optional.

4bab48b

Add driver pod name check for cluster mode

94ed1cc

Fix build

560993e

mccheah mentioned this pull request Jul 11, 2018

[SPARK-23146][WIP] Support client mode for Kubernetes in Out-Cluster mode #20451

Closed

mccheah added 2 commits July 11, 2018 12:04

Remove unused imports

e961fd3

Fix build

a00561f

More compilation fixes.

a2609b0

Fix build again

97f1284

mccheah added 2 commits July 11, 2018 12:49

Small change to force build to re-run

9a17830

Merge remote-tracking branch 'apache/master' into k8s-client-mode

2205220

Fix build

64cc39b

mccheah added 3 commits July 20, 2018 11:43

Clarify some docs

086747e

Merge remote-tracking branch 'apache/master' into k8s-client-mode

7d0bc2a

Fix typo

d90f753

liyinan926 reviewed Jul 20, 2018

View reviewed changes

mccheah added 2 commits July 20, 2018 12:03

Fix wording

001a525

More feedback

72c96e0

liyinan926 mentioned this pull request Jul 20, 2018

[SPARK-23078] [CORE] [K8s] allow Spark Thrift Server to run in Kubernetes Cluster mode #20272

Closed

felixcheung approved these changes Jul 24, 2018

View reviewed changes

Helper method for master url parsing

ded1ff6

asfgit closed this in 571a6f0 Jul 25, 2018

skonto reviewed Jul 27, 2018

View reviewed changes

mccheah deleted the k8s-client-mode branch July 27, 2018 18:02

[SPARK-23146][K8S] Support client mode. #21748

[SPARK-23146][K8S] Support client mode. #21748

Uh oh!

Conversation

mccheah commented Jul 11, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

mccheah commented Jul 11, 2018

Uh oh!

mccheah commented Jul 11, 2018

Uh oh!

SparkQA commented Jul 11, 2018

Uh oh!

mccheah commented Jul 11, 2018

Uh oh!

SparkQA commented Jul 11, 2018

Uh oh!

SparkQA commented Jul 11, 2018

Uh oh!

SparkQA commented Jul 11, 2018

Uh oh!

SparkQA commented Jul 11, 2018

Uh oh!

SparkQA commented Jul 11, 2018

Uh oh!

SparkQA commented Jul 11, 2018

Uh oh!

SparkQA commented Jul 11, 2018

Uh oh!

SparkQA commented Jul 11, 2018

Uh oh!

SparkQA commented Jul 11, 2018

Uh oh!

mccheah commented Jul 11, 2018

Uh oh!

SparkQA commented Jul 11, 2018

Uh oh!

SparkQA commented Jul 11, 2018

Uh oh!

shaneknapp commented Jul 11, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jul 20, 2018

Uh oh!

SparkQA commented Jul 20, 2018

Uh oh!

mccheah commented Jul 20, 2018

Uh oh!

SparkQA commented Jul 20, 2018

Uh oh!

mccheah commented Jul 20, 2018

Uh oh!

SparkQA commented Jul 20, 2018

Uh oh!

mccheah commented Jul 20, 2018

Uh oh!

SparkQA commented Jul 20, 2018

Uh oh!

mccheah commented Jul 20, 2018

Uh oh!

liyinan926 commented Jul 20, 2018

Uh oh!

SparkQA commented Jul 20, 2018

Uh oh!

SparkQA commented Jul 20, 2018

Uh oh!

mccheah commented Jul 23, 2018

mccheah commented Jul 11, 2018 •

edited

Loading

felixcheung Jul 24, 2018 •

edited

Loading

felixcheung Jul 24, 2018 •

edited

Loading

ifilonenko commented Jul 25, 2018 •

edited

Loading

skonto Jul 27, 2018 •

edited

Loading

skonto Jul 29, 2018 •

edited

Loading