diff --git a/README.md b/README.md index c2030212e1..7dc1babc4e 100644 --- a/README.md +++ b/README.md @@ -30,6 +30,8 @@ are looking forward to your participation. **NOTE**: the scheduler is built based on [kube-batch](https://github.com/kubernetes-sigs/kube-batch); refer to [#241](https://github.com/volcano-sh/volcano/issues/241) and [#288](https://github.com/volcano-sh/volcano/pull/288) for more detail. +[JobFlow](./docs/design/jobflow/jobflow.md) is a workflow engine based on volcano Job. It proposes two concepts to automate running multiple batch jobs, named JobTemplate and JobFlow, so end users can easily declare their jobs and run them using complex control primitives such as sequential or parallel execution, if-then -else statement, switch-case statement, loop execution, etc. + ![cncf_logo](docs/images/cncf-logo.png) Volcano is an incubating project of the [Cloud Native Computing Foundation](https://cncf.io/) (CNCF). Please consider joining the CNCF if you are an organization that wants to take an active role in supporting the growth and evolution of the cloud native ecosystem. @@ -38,6 +40,8 @@ Volcano is an incubating project of the [Cloud Native Computing Foundation](http ![volcano](docs/images/volcano-architecture.png) +![jobflow-1.png](./docs/design/images/jobflow-1.png) + ## Talks - [Intro: Kubernetes Batch Scheduling @ KubeCon 2019 EU](https://sched.co/MPi7) @@ -110,6 +114,10 @@ job.batch/volcano-admission-init 1/1 48s 96s ``` +### Disable JobFlow + +Disable JobFlow related components if necessary. You need to add --forbid-jobflow=true to the startup parameters of the controller and admission respectively. The default is false to enable the state. + ### Install from code If you don't have a kubernetes cluster, try one-click install from code base: diff --git a/docs/design/images/jobflow-1.png b/docs/design/images/jobflow-1.png new file mode 100644 index 0000000000..94bd0f6db8 Binary files /dev/null and b/docs/design/images/jobflow-1.png differ diff --git a/docs/design/images/jobflow-2.jpg b/docs/design/images/jobflow-2.jpg new file mode 100644 index 0000000000..ac988dfa67 Binary files /dev/null and b/docs/design/images/jobflow-2.jpg differ diff --git a/docs/design/images/jobflow-3.png b/docs/design/images/jobflow-3.png new file mode 100644 index 0000000000..9aa3362d22 Binary files /dev/null and b/docs/design/images/jobflow-3.png differ diff --git a/docs/design/images/jobflow-4.png b/docs/design/images/jobflow-4.png new file mode 100644 index 0000000000..a94e2a7a6a Binary files /dev/null and b/docs/design/images/jobflow-4.png differ diff --git a/docs/design/images/jobflow.gif b/docs/design/images/jobflow.gif new file mode 100644 index 0000000000..0dfd77ba75 Binary files /dev/null and b/docs/design/images/jobflow.gif differ diff --git a/docs/design/jobflow/jobflow.md b/docs/design/jobflow/jobflow.md new file mode 100755 index 0000000000..6eef4b1876 --- /dev/null +++ b/docs/design/jobflow/jobflow.md @@ -0,0 +1,368 @@ +# JobFlow + +## Introduction + +In order to solve the problem of inter-job dependencies. We need many VCJobs to cooperate each other and orchestrate them manually or by another Job Orchestration Platform to get the job done finally.We present an new way of orchestrating VCJobs called JobFlow. We proposed two concepts to running multiple batch jobs automatically named JobTemplate and JobFlow so end users can easily declare their jobs and run them using complex controlling primitives, for example, sequential or parallel executing, if-then-else statement, switch-case statement, loop executing and so on. + +JobFlow helps migrating AI, BigData, HPC workloads to the cloud-native world. Though there are already some workload flow engines, they are not designed for batch job workloads. Those jobs typically have a complex running dependencies and take long time to run, for example days or weeks. JobFlow helps the end users to declare their jobs as an jobTemplate and then reuse them accordingly. Also, JobFlow orchestrating those jobs using complex controlling primitives and launch those jobs automatically. This can significantly reduce the time consumption of an complex job and improve resource utilization. Finally, JobFlow is not an generally purposed workflow engine, it knows the details of VCJobs. End user can have a better understanding of their jobs, for example, job's running state, beginning and ending timestamps, the next jobs to run, pod-failure-ratio and so on. + +## Scope + +### In Scope +- Define the API of JobFlow +- Define the behaviour of JobFlow +- Start sequence between multiple jobs +- Dependency completion state of the job start sequence +- DAG-based job dependency startup + +### Out of Scope +- Supports other job +- Achieve vcjobs level gang + +## Scenarios + +- Some jobs need to depend on the completion of the previous job or other status when running, etc. Otherwise, the correct result cannot be calculated. +- Sometimes inter-job dependencies also require diverse dependency types, such as conditional dependencies, circular dependencies, probes, and so on. + +![jobflow-2.jpg](../images/jobflow-2.jpg) + +## Design + +![jobflow-1.png](../images/jobflow-1.png) + +![jobflow-3.png](../images/jobflow-3.png) + +The blue part is the component of k8s itself, the green and brown are the components of volcano, and the yellow is the crd resource of volcano. + +jobflow job submission complete process: + +1. After passing the Admission. kubectl will create JobTemplate and JobFlow (Volcano CRD) objects in kube-apiserver. + +2. The JobFlowController uses the JobTemplate as a template according to the configuration of the JobFlow, and creates the corresponding VcJob according to the flow dependency rules. + +3. After VcJob is created, VcJobController creates corresponding Pods and podgroups according to the configuration of VcJob. + +4. After Pod and PodGroup are created, vc-scheduler will go to kube-apiserver to get Pod/PodGroup and node information. + +5. After obtaining the information, vc-scheduler will select the appropriate node for each Pod according to its configured scheduling policy. + +6. After assigning nodes to Pods, kubelet will get the Pod's configuration from kube-apiserver and start the corresponding containers. + + + +### Controller + +![jobflow-4.png](../images/jobflow-4.png) + +### Webhook + +``` +Create a JobFlow check +1、There cannot be a template with the same name in a JobFlow dependency + Such as: A->B->A->C A appears twice +2、Closed loops cannot occur in JobFlow + E.g:A -> B -> C + ^ / + | / + < - D + +Create a JobTemplte check (following the vcjob parameter specification) +E.g: job minAvailable must be greater than or equal to zero + job maxRetry must be greater than or equal to zero + tasks cannot be empty, and cannot have tasks with the same name + The number of task replicas cannot be less than zero + task minAvailable cannot be greater than task replicas... +``` + +### JobFlow + +#### Introduction + +JobFlow defines the running flow of a set of jobs. Fields in JobFlow define how jobs are orchestrated. + +JobFlow is abbreviated as jf, and the resource can be viewed through kubectl get jf + +JobFlow aims to realize job-dependent operation between vcjobs in volcano. According to the dependency between vcjob, vcjob is issued. + +#### Key Fields + +##### Top-Level Attributes + +The top-level attributes of a jobflow define its apiVersion, kind, metadata and spec. + +| Attribute | Type | Required | Default Value | Description | +| ------------ | ----------------------- | -------- | -------------------------- | ------------------------------------------------------------ | +| `apiVersion` | `string` | Y | `flow.volcano.sh/v1alpha1` | A string that identifies the version of the schema the object should have. The core types uses `flow.volcano.sh/v1alpha1` in this version of documentation. | +| `kind` | `string` | Y | `JobFlow` | Must be `JobFlow` | +| `metadata` | [`Metadata`](#Metadata) | Y | | Information about the JobFlow resource. | +| `spec` | [`Spec`](#spec) | Y | | A specification for the JobFlow resource attributes. | +| `status` | [`Status`](#Status) | Y | | A specification for the JobFlow status attributes. | + + + +##### Metadata + +Metadata provides basic information about the JobFlow. + +| Attribute | Type | Required | Default Value | Description | +| ------------- | ------------------- | -------- | ------------- | ------------------------------------------------------------ | +| `name` | `string` | Y | | A name for the schematic. `name` is subject to the restrictions listed beneath this table. | +| `namespace` | `string` | Y | | A namespace for the schematic. `namespace` is subject to the restrictions listed beneath this table. | +| `labels` | `map[string]string` | N | | A set of string key/value pairs used as arbitrary labels on this component. Labels follow the [Kubernetes specification](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/). | +| `annotations` | `map[string]string` | N | | A set of string key/value pairs used as arbitrary descriptive text associated with this object. Annotations follows the [Kubernetes specification](https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/#syntax-and-character-set). | + + + +##### Spec + +The specification of cloud-native services defines service metadata, version list, service capabilities and plugins. + +| Attribute | Type | Required | Default Value | Description | +| ----------------- | ------------------------------------ | -------- | ------------- | ------------------------------------------------------------ | +| `flows` | [`Flow array`](#Flow) | Y | | Describes the dependencies between vcjobs. | +| `jobRetainPolicy` | `string` | Y | retain | After JobFlow succeed, keep the generated job. Otherwise, delete it. | + + + +##### Flow + +| Attribute | Type | Required | Default Value | Description | +| ----------------- | ------------------------------------ | -------- | ------------- | ------------------------------------------------------------ | +| `name` | `string` | Y | | JobTemplate name | +| `dependsOn` | [`DependsOn`](#DependsOn) | Y | | JobTemplate dependencies | +| `patch` | [`Patch`](#Patch) | N | | Patch JobTemplate | + + + +##### DependsOn + +| Attribute | Type | Required | Default Value | Description | +| ----------------- | ------------------------------------ | -------- | ------------- | ------------------------------------------------------------ | +| `targets` | `string array` | Y | | All jobtemplate names that JobTemplate depends on | +| `probe` | [`Probe`](#Probe) | N | | Probe Type Dependency | +| `strategy` | `string` | Y | all | Whether the dependencies need to be all satisfied | + + + +##### Patch + +| Attribute | Type | Required | Default Value | Description | +| ----------------- | ------------------------------------ | -------- | ------------- | ------------------------------------------------------------ | +| `spec` | `spec` | Y | | Patch the contents of the jobtemplate's spec | + + + +##### Probe + +| Attribute | Type | Required | Default Value | Description | +| ----------------- | ------------------------------------ | -------- | ------------- | ------------------------------------------------------------ | +| `httpGetList` | [`HttpGet array`](#HttpGet) | N | | HttpGet type dependencies | +| `tcpSocketList` | [`TcpSocket array`](#TcpSocket) | N | | TcpSocket type dependencies | +| `taskStatusList` | [`TaskStatus array`](#TaskStatus) | N | | TaskStatus type dependencies | + + + +##### HttpGet + +| Attribute | Type | Required | Default Value | Description | +| ----------------- | ------------------------------------ | -------- | ------------- | ------------------------------------------------------------ | +| `TaskName` | `string` | Y | | The name of the task under vcjob | +| `Path` | [`Probe`](#Probe) | Y | | The path of httpget | +| `Port` | `int` | Y | | The port of httpget | +| `httpHeader` | `HTTPHeader` | N | | The httpHeader of httpget | + + + +##### TcpSocket + +| Attribute | Type | Required | Default Value | Description | +| ----------------- | ------------------------------------ | -------- | ------------- | ------------------------------------------------------------ | +| `TaskName` | `string` | Y | | The name of the task under vcjob | +| `Port` | `int` | Y | | The port of TcpSocket | + + + +##### TaskStatus + +| Attribute | Type | Required | Default Value | Description | +| ----------------- | ------------------------------------ | -------- | ------------- | ------------------------------------------------------------ | +| `TaskName` | `string` | Y | | The name of the task under vcjob | +| `Phase` | `string` | Y | | The phase of task | + + + +##### Status + +| Attribute | Type | Required | Default Value | Description | +| ----------------- | ------------------------------------ | -------- | ------------- | ------------------------------------------------------------ | +| `pendingJobs` | `string array` | N | | Vcjobs in pending state | +| `runningJobs` | `string array` | N | | Vcjobs in running state | +| `failedJobs` | `string array` | N | | Vcjobs in failed state | +| `completedJobs` | `string array` | N | | Vcjobs in completed and completing state | +| `terminatedJobs` | `string array` | N | | Vcjobs in terminated and terminating state | +| `unKnowJobs` | `string array` | N | | Vcjobs in pending state | +| `jobStatusList` | [`JobStatus array`](#JobStatus) | N | | Status information of all split vcjobs | +| `conditions` | [`map[string]Condition`](#Condition) | N | | It is used to describe the current state, creation time, completion time and information of all vcjobs. The vcjob state here additionally adds the waiting state to describe the vcjob whose dependencies do not meet the requirements. | +| `state` | [`State`](#State) | N | | State of JobFlow | + + + +##### JobStatus + +| Attribute | Type | Required | Default Value | Description | +| ----------------- | ------------------------------------ | -------- | ------------- | ------------------------------------------------------------ | +| `name` | `string` | N | | Name of vcjob | +| `state` | `string` | N | | State of vcJob | +| `startTimestamp` | `Time` | N | | StartTimestamp of vcjob | +| `endTimestamp` | `Time` | N | | EndTimestamp of vcjob | +| `restartCount` | `int32` | N | | RestartCount of vcjob | +| `runningHistories` | [`JobRunningHistory array`](#JobRunningHistory) | N | | Historical information of various states of vcjob | + + + +##### Condition + +| Attribute | Type | Required | Default Value | Description | +| ----------------- | ------------------------------------ | -------- | ------------- | ------------------------------------------------------------ | +| `phase` | `string` | N | | phase of vcjob | +| `createTime` | `Time` | N | | CreateTime of vcjob | +| `runningDuration` | `Duration` | N | | RunningDuration of vcjob | +| `taskStatusCount` | `map[string]TaskState` | N | | The number of tasks in different states | + + + +##### State + +| Attribute | Type | Required | Default Value | Description | +| ----------------- | ------------------------------------ | -------- | ------------- | ------------------------------------------------------------ | +| `phase` | `string` | N | | Succeed: All vcjobs have reached completed state。
Terminating: Jobflow is deleting。
Failed: A vcjob in the flow is in the failed state, so the vcjob in the flow cannot continue to be delivered。
Running: Flow contains vcjob in Running state。
Pending: The flow contains no vcjob in the Running state。 | + + + +##### JobRunningHistory + +| Attribute | Type | Required | Default Value | Description | +| ----------------- | ------------------------------------ | -------- | ------------- | ------------------------------------------------------------ | +| `startTimestamp` | `Time` | N | | The start time of a certain state of the vcjob | +| `endTimestamp` | `Time` | N | | The end time of a certain state of the vcjob | +| `state` | `string` | N | | Vcjob status | + +JobFlow supports the functionality of the JobTemplate patch. The example in JobFlow is as follows: + +``` +apiVersion: flow.volcano.sh/v1alpha1 +kind: JobFlow +metadata: + name: test + namespace: default +spec: + jobRetainPolicy: delete + flows: + - name: a + patch: + spec: + tasks: + - name: "default-nginx" + template: + spec: + containers: + - name: nginx + command: + - sh + - -c + - sleep 10s +``` + +Here is an example of jobflow: + +[the sample file of JobFlow](../../../example/jobflow/JobFlow.yaml) + +### JobTemplate + +#### Introduction + +* JobTemplate is the template of vcjob, after JobTemplate is created, it will not be processed by vc-controller like vcjob, it will wait to be referenced by JobFlow. +* JobFlow can reference multiple jobtemplates +* A jobtemplate can be referenced by multiple jobflows +* JobTemplate can be converted to and from vcjob. +* Jobtemplate is abbreviated as jt, and the resource can be viewed through kubectl get jt +* The difference between jobtemplate and vcjob is that jobtemplate will not be issued by the job controller, and jobflow can directly reference the name of the JobTemplate to implement the issuance of vcjob. +* JobFlow supports making changes to jobtemplate when referencing jobtemplate + +#### Definition + +#### Key Fields + +##### Top-Level Attributes + +The top-level attributes of a jobtemplate define its apiVersion, kind, metadata and spec. + +| Attribute | Type | Required | Default Value | Description | +| ------------ | ----------------------- | -------- | -------------------------- | ------------------------------------------------------------ | +| `apiVersion` | `string` | Y | `flow.volcano.sh/v1alpha1` | A string that identifies the version of the schema the object should have. The core types uses `flow.volcano.sh/v1alpha1` in this version of documentation. | +| `kind` | `string` | Y | `JobTemplate` | Must be `JobTemplate` | +| `metadata` | [`Metadata`](#JobTemplateMetadata) | Y | | Information about the JobTemplate resource. | +| `spec` | [`Spec`](#JobTemplateSpec) | Y | | A specification for the JobTemplate resource attributes. | +| `status` | [`Status`](# JobTemplateStatus) | Y | | A specification for the JobTemplate status attributes. | + + + +##### Metadata + +Metadata provides basic information about the JobTemplate. + +| Attribute | Type | Required | Default Value | Description | +| ------------- | ------------------- | -------- | ------------- | ------------------------------------------------------------ | +| `name` | `string` | Y | | A name for the schematic. `name` is subject to the restrictions listed beneath this table. | +| `namespace` | `string` | Y | | A namespace for the schematic. `namespace` is subject to the restrictions listed beneath this table. | +| `labels` | `map[string]string` | N | | A set of string key/value pairs used as arbitrary labels on this component. Labels follow the [Kubernetes specification](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/). | +| `annotations` | `map[string]string` | N | | A set of string key/value pairs used as arbitrary descriptive text associated with this object. Annotations follows the [Kubernetes specification](https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/#syntax-and-character-set). | + + + +##### JobTemplateSpec + +The spec of jobtemplate directly follows the spec of vcjob. + + + +##### JobTemplateStatus +| Attribute | Type | Required | Default Value | Description | +| ----------------- | ------------------------------------ | -------- | ------------- | ------------------------------------------------------------ | +| `jobDependsOnList` | `string array` | Y | | Vcjobs created by this jobtemplate as a template. | + +You can view [the sample file of JobTemplate](../../../example/jobflow/JobTemplate.yaml) + +## JobFlow task scheduling + +![jobflowAnimation](../images/jobflow.gif) + +## Demo video + +https://www.bilibili.com/video/BV1c44y1Y7FX + +## Usage + +- Create the jobTemplate that needs to be used +- Create a jobflow. The flow field of the jobflow is filled with the corresponding jobtemplate used to create a vcjob. +- The field jobRetainPolicy indicates whether to delete the vcjob created by the jobflow after the jobflow succeeds. (delete/retain) default is retain. + +## JobFlow Features + +### Features that have been implemented + +* Create JobFlow and JobTemplate CRD +* Support sequential start of vcjob +* Support vcjob to depend on other vcjobs to start +* Support the conversion of vcjob and JobTemplate to each other +* Supports viewing of the running status of JobFlow + +### Features not yet implemented + +* JobFlow supports making changes to jobtemplate when referencing jobtemplate +* `if` statements +* `switch` statements +* `for` statements +* Support job failure retry in JobFlow +* Integration with volcano-scheduler +* Support for scheduling plugins at JobFlow level \ No newline at end of file diff --git a/example/jobflow/JobFlow.yaml b/example/jobflow/JobFlow.yaml new file mode 100755 index 0000000000..66e7985056 --- /dev/null +++ b/example/jobflow/JobFlow.yaml @@ -0,0 +1,22 @@ +apiVersion: flow.volcano.sh/v1alpha1 +kind: JobFlow +metadata: + name: test + namespace: default +spec: + jobRetainPolicy: delete # After jobflow runs, keep the generated job. Otherwise, delete it. + flows: + - name: a + - name: b + dependsOn: + targets: ['a'] + - name: c + dependsOn: + targets: ['b'] + - name: d + dependsOn: + targets: ['b'] + - name: e + dependsOn: + targets: ['c','d'] + \ No newline at end of file diff --git a/example/jobflow/JobTemplate.yaml b/example/jobflow/JobTemplate.yaml new file mode 100755 index 0000000000..01e5150985 --- /dev/null +++ b/example/jobflow/JobTemplate.yaml @@ -0,0 +1,184 @@ +apiVersion: flow.volcano.sh/v1alpha1 +kind: JobTemplate +metadata: + name: a +spec: + minAvailable: 1 + schedulerName: volcano + priorityClassName: high-priority + policies: + - event: PodEvicted + action: RestartJob + plugins: + ssh: [] + env: [] + svc: [] + maxRetry: 5 + queue: default + tasks: + - replicas: 1 + name: "default-nginx" + template: + metadata: + name: web + spec: + containers: + - image: nginx:1.14.2 + command: + - sh + - -c + - sleep 10s + imagePullPolicy: IfNotPresent + name: nginx + resources: + requests: + cpu: "1" + restartPolicy: OnFailure +--- +apiVersion: flow.volcano.sh/v1alpha1 +kind: JobTemplate +metadata: + name: b +spec: + minAvailable: 1 + schedulerName: volcano + priorityClassName: high-priority + policies: + - event: PodEvicted + action: RestartJob + plugins: + ssh: [] + env: [] + svc: [] + maxRetry: 5 + queue: default + tasks: + - replicas: 1 + name: "default-nginx" + template: + metadata: + name: web + spec: + containers: + - image: nginx:1.14.2 + command: + - sh + - -c + - sleep 10s + imagePullPolicy: IfNotPresent + name: nginx + resources: + requests: + cpu: "1" + restartPolicy: OnFailure +--- +apiVersion: flow.volcano.sh/v1alpha1 +kind: JobTemplate +metadata: + name: c +spec: + minAvailable: 1 + schedulerName: volcano + priorityClassName: high-priority + policies: + - event: PodEvicted + action: RestartJob + plugins: + ssh: [] + env: [] + svc: [] + maxRetry: 5 + queue: default + tasks: + - replicas: 1 + name: "default-nginx" + template: + metadata: + name: web + spec: + containers: + - image: nginx:1.14.2 + command: + - sh + - -c + - sleep 10s + imagePullPolicy: IfNotPresent + name: nginx + resources: + requests: + cpu: "1" + restartPolicy: OnFailure +--- +apiVersion: flow.volcano.sh/v1alpha1 +kind: JobTemplate +metadata: + name: d +spec: + minAvailable: 1 + schedulerName: volcano + priorityClassName: high-priority + policies: + - event: PodEvicted + action: RestartJob + plugins: + ssh: [] + env: [] + svc: [] + maxRetry: 5 + queue: default + tasks: + - replicas: 1 + name: "default-nginx" + template: + metadata: + name: web + spec: + containers: + - image: nginx:1.14.2 + command: + - sh + - -c + - sleep 10s + imagePullPolicy: IfNotPresent + name: nginx + resources: + requests: + cpu: "1" + restartPolicy: OnFailure +--- +apiVersion: flow.volcano.sh/v1alpha1 +kind: JobTemplate +metadata: + name: e +spec: + minAvailable: 1 + schedulerName: volcano + priorityClassName: high-priority + policies: + - event: PodEvicted + action: RestartJob + plugins: + ssh: [] + env: [] + svc: [] + maxRetry: 5 + queue: default + tasks: + - replicas: 1 + name: "default-nginx" + template: + metadata: + name: web + spec: + containers: + - image: nginx:1.14.2 + command: + - sh + - -c + - sleep 10s + imagePullPolicy: IfNotPresent + name: nginx + resources: + requests: + cpu: "1" + restartPolicy: OnFailure \ No newline at end of file