Kubernetes ThirdPartyResource for tracking Spark Jobs

[foxish]

Issues:

• We need a way for users to find, monitor status, and kill their spark jobs.
• Other cluster managers provide an interface to view all of a user's spark-jobs in a cluster, along with their state (QUEUED, SUBMITTED, RUNNING, FINISHED, FAILED, KILLED).

Shortcomings of the current method:

• It is not possible to get sufficient detail about every spark job just by looking at the pods and their states.
• As the spark framework is currently performing the entire function of a kubernetes controller, it gives us no way to find the state of various running Spark jobs than to implement spark-submit --status, or spark-submit --kill which would have to encode special logic to find the drivers in a particular namespace and find their status. This would be bad for cluster administrators who deal with kubernetes abstractions, and not application specifics.
• Accessing arbitrary data such as progress from the driver pod, or the names of its associated executors is difficult.

Proposed Solution:

A ThirdPartyResource to keep track of the execution state (pending/running/failed) of each SparkJob, failure reasons if any, the identity of the driver and executor pods associated, as well as configuration metadata associated with that job (number of executors, memory per executor, etc).

metadata:
  name: spark-job.kubernetes.io
  labels:
    resource: spark-job
    object: spark
apiVersion: extensions/v1beta1
kind: ThirdPartyResource
description: "A resource that manages a spark job"
versions:
  - name: v1

The cluster administrator is responsible for creating this resource, which makes a new API endpoint available in Kubernetes. This TPR would enable us to create objects of the kind SparkJob and store JSON within them. Each such object would be associated with a single spark job, and would store all the status and metadata associated with it. The driver pod is responsible for the life-cycle of the SparkJob object, from creation till deletion.

A sample object of the above kind looks like the following:

{
    "apiVersion": "kubernetes.io/v1",
    "image": "driver-image",
    "kind": "SparkJob",
    "metadata": {
        "name": "spark-driver-1924",
        "namespace": "default",
        "selfLink": "/apis/kubernetes.io/v1/namespaces/default/sparkjobs/spark-driver-1924",
        "uid": "91022bc2-a71d-11e6-a4be-42010af00002",
        "resourceVersion": "765519",
        "creationTimestamp": "2016-11-10T08:13:31Z"
    },
    "num-executors": 10,
    "state": "completed",
    "driver-pod": "driver-2ds9f"
    ...
    ...
}

The driver pod has complete visibility into progress of the job, and can set the status of its SparkJob object. The driver can also watch this resource for configuration changes which may be triggered by the user/cluster administrator. Killing a spark-job can be performed by destroying the associated SparkJob object, which will cause the driver pod to terminate its executors and clean up gracefully.

• It makes the state of each SparkJob available outside Spark itself and provides some visibility to the cluster administrator regarding various running spark jobs in the system.
• This SparkJob object can be consumed by spark-submit --status, or a dashboard to display various details about spark jobs in the system.

Further thought:

• What if the driver pods do not exit cleanly (or are force-killed)? Who is responsible for cleaning up the SparkJob object?

[foxish]

cc @tnachen @erikerlandson @erictune

[tnachen]

It sounds like a great way to provide visiblity and operate Spark.
Need to think through a bit more, but some questions for now:

• Are we going to have a TPR per namespace, so not all Spark jobs across all namespaces is visible and managable?
• This TPR is going to have a UI and persistent storage somewhere to actually store all the state of each job? Then this also becomes where a driver can also queue jobs up as well?

[erikerlandson]

@foxish TPR looks like a cool kube feature! Regarding lifecycle visibility and mgmt, I'm increasingly interested in adding some new published metrics to Spark. Particularly, having Executors report some new metrics that reflect the size of their queued job backlog. My main interest in that is so it could be harvested for use by an HPA for executor pods. In other words, it would allow external scaling logic to use the same flavor of information that Spark's dynamic-executor logic uses, but since the logic could be external, it opens up a large solution space of scaling logic without requiring that logic to live inside Spark (let Spark do what it does well, and not try to be in the business of advanced scaling and sharing features)(*)

But you could take this idea further, and publish other metrics that provide increased visibility into the job progress. Some of this may already be available from the rest API that already exists in Spark? Spark's web UI gets its job progress information from somewhere.

(*) This would also require the ability to externally signal a Spark executor into graceful shutdown, but that is outside the scope of this topic

[tnachen]

Job progress information is all available through the Spark Listener interface that gets reported from the job itself. I think most of the dynamic allocation information is published through jmx already.

Anyhow, seems like an interesting idea but I don't know how this fits with this proposed TPR?

[iyanuobidele]

It sure sounds like a great way to monitor the state of spark jobs!

Are we going to have a TPR per namespace.....

I believe k8s doesn't currently support non-namespaced object, so for this to work we need to create this TPR to exist in the namespace as the spark job about to be deployed. So this might answer some part of @tnachen's first question. Since we already depend on the namespace specified at runtime or in the spark conf file, I agree with @foxish's initial idea of creating the TPR ahead of time in this same namespace.

so not all Spark jobs across all namespaces is visible and managable?

By this, if you mean the new kind just created by TPR, I believe you should be able to do kubectl get sparkjob --all-namespaces provided that the TPR's in all namespaces all have the same resource type name. (I haven't really tried this out)

[erikerlandson]

@tnachen I was proposing some possible alternatives to increasing visibility into driver+executor(s) state (from the k8s layer). (alternatives that might be aligned with some of my other interests)

But TPR is idiomatic w.r.t. a kube environment. I might turn it around, and consider publishing TPR objects to represent the loading state of each executor.

Can HPA using custom metrics be aimed at one of these ThirdPartyResource endpoints?

[iyanuobidele]

Some questions:

The driver pod has complete visibility into progress of the job, and can set the status of its SparkJob object.

How do you see implementing this ? Maybe explain a little further what the relationship between the pod and the sparkjob resource type is. Is this like a pod and job kind of relationship, and if so are there any additional reasons why we can't use the existing job kind instead of creating a new resource type ?

[erikerlandson]

@iyanuobidele, IIUC the idea is to have the scheduler (and/or scheduler backend) publish these objects into the namespace the app is running on, presumably using the fabric8 client, with information on the current state of the application

[foxish]

@iyanuobidele
The TPR object is meant to store state about the running SparkJob and act as a data model. There is benefit in looking at individual entities as "SparkJobs" in the kubernetes system, as opposed to simply driver/executor pods IMO. Exposing the state of the spark job to kubernetes also allows us to write custom controllers which can make cluster-level decisions as @erikerlandson mentioned.

Is this like a pod and job kind of relationship, and if so are there any additional reasons why we can't use the existing job kind instead of creating a new resource type ? Maybe explain a little further what the relationship between the pod and the sparkjob resource type is

A Kubernetes Job is managed by the JobController, which deals with pod creation and deletion decisions. With native integration, we let Spark act as the "controller". So, it's not quite the pod-job type of relationship. The TPR is really just being used as a key-value store which the driver writes to, and all other components read from. I'm unsure if there is a use-case where we want to change the configuration of a job after launching the driver.

The driver pod would create the TPR object when it comes up and update it at regular intervals with metrics and state. Since we are using namespaces as the mechanism to isolate users, it makes sense for the TPR object to live in the same namespace as that user's spark driver and executor pods. The Spark "JobID" could be associated with the TPR object, and be used for finding status, summarizing results, etc and we would not need to directly talk to pods.

[tnachen]

I'm also wondering if we should make it a hard requirement for the TPR to exist for Spark jobs to work on k8s. I don't think there is any reason why the TPR should be a problem for anyone, but just want to make sure this is explicit and we should probably check from the driver side and have a sensible message.

[erikerlandson]

I'd prefer at least the option for the submission client to create the TPR automatically, unless it requires admin privs like the service acct does. It should be easy to do in the fabric8 client, assuming it has a fluent constructor for it.

[foxish]

The ThirdPartyResource itself (which creates the additional API endpoint) needs to be created exactly once for the entire cluster. That would be a prerequisite I imagine. The TPR objects on the other hand could be created automatically by the submission client.

[erikerlandson]

@foxish, if it's not idempotent that makes it slightly harder, but I assume the code can check for existence first if need be. If I'm reading correctly, a TPR gets created on a per-namespace basis.