From 4be538df0a6f5e5588272f93fa9a9e2b02a45f44 Mon Sep 17 00:00:00 2001 From: Maciej Szulik Date: Wed, 26 Apr 2017 12:09:35 +0200 Subject: [PATCH 1/7] Update proposal to match current job spec --- contributors/design-proposals/job.md | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/contributors/design-proposals/job.md b/contributors/design-proposals/job.md index 160b38ddb4f..31fb0e3f118 100644 --- a/contributors/design-proposals/job.md +++ b/contributors/design-proposals/job.md @@ -79,8 +79,12 @@ type JobSpec struct { // job should be run with. Defaults to 1. Completions *int + // Optional duration in seconds relative to the startTime that the job may be active + // before the system tries to terminate it; value must be positive integer + ActiveDeadlineSeconds *int + // Selector is a label query over pods running a job. - Selector map[string]string + Selector LabelSelector // Template is the object that describes the pod that will be created when // executing a job. @@ -109,12 +113,12 @@ type JobStatus struct { // Active is the number of actively running pods. Active int - // Successful is the number of pods successfully completed their job. - Successful int + // Succeeded is the number of pods successfully completed their job. + Succeeded int - // Unsuccessful is the number of pods failures, this applies only to jobs + // Failed is the number of pods failures, this applies only to jobs // created with RestartPolicyNever, otherwise this value will always be 0. - Unsuccessful int + Failed int } type JobConditionType string From bd5d3d9bf3ea9733eadecd3932839123209b2ad3 Mon Sep 17 00:00:00 2001 From: Maciej Szulik Date: Wed, 26 Apr 2017 12:09:54 +0200 Subject: [PATCH 2/7] Add backoff policy and failed pod limit --- contributors/design-proposals/job.md | 35 ++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+) diff --git a/contributors/design-proposals/job.md b/contributors/design-proposals/job.md index 31fb0e3f118..ac11a7df3d2 100644 --- a/contributors/design-proposals/job.md +++ b/contributors/design-proposals/job.md @@ -18,6 +18,7 @@ Several existing issues and PRs were already created regarding that particular s 1. Be able to get the job status. 1. Be able to specify the number of instances performing a job at any one time. 1. Be able to specify the number of successfully finished instances required to finish a job. +1. Be able to specify backoff policy, when job is continuously failing. ## Motivation @@ -26,6 +27,31 @@ Jobs are needed for executing multi-pod computation to completion; a good exampl here would be the ability to implement any type of batch oriented tasks. +## Backoff policy and failed pod limit + +By design, Jobs do not have any notion of failure, other than Pod's `restartPolicy` +which is mistakenly taken as Job's restart policy ([#30243](https://github.com/kubernetes/kubernetes/issues/30243), +[#[43964](https://github.com/kubernetes/kubernetes/issues/43964)]). There are +situation where one wants to fail a Job after some amount of retries over certain +period of time, due to a logical error in configuration etc. To do so we are going +following fields will be introduced, which will control the exponential backoff +when retrying Job: number of retries and time to retry. The two fields will allow +creating a fine grain control over the backoff policy, limiting the number of retries +over specified period of time. In the case when only one of them is specified +an exponential backoff with duration of 10 seconds and factor of 2 will be applied +in such a way that either time or number is reached. After reaching the limit +a Job will be marked as failed. + +Additionally, to help debug the issue with a job, and limit the impact of having +too many failed pods left around (as mentioned in [#30243](https://github.com/kubernetes/kubernetes/issues/30243)) +we are going to introduce a field which will allow specifying the maximum number +of failed pods to keep around. This number will also take effect if none of the +limits, described above, are set. + +All of the above fields will be optional and will apply no matter which `restartPolicy` +is set on a `PodTemplate`. + + ## Implementation Job controller is similar to replication controller in that they manage pods. @@ -83,6 +109,15 @@ type JobSpec struct { // before the system tries to terminate it; value must be positive integer ActiveDeadlineSeconds *int + // Optional number of retries, before marking this job failed. + BackoffLimit *int + + // Optional time (in seconds), how log a job should be retried before marking it failed. + BackoffDeadlineSeconds *int + + // Optional number of failed pods to retain. + FailedPodsLimit *int + // Selector is a label query over pods running a job. Selector LabelSelector From caee4947ba30dd2267accaeeb00e6bd0feb2c82f Mon Sep 17 00:00:00 2001 From: Maciej Szulik Date: Thu, 27 Apr 2017 17:23:58 +0200 Subject: [PATCH 3/7] Address the first batch of comments --- contributors/design-proposals/job.md | 40 +++++++++++++++------------- 1 file changed, 22 insertions(+), 18 deletions(-) diff --git a/contributors/design-proposals/job.md b/contributors/design-proposals/job.md index ac11a7df3d2..9653e6cc57d 100644 --- a/contributors/design-proposals/job.md +++ b/contributors/design-proposals/job.md @@ -18,7 +18,7 @@ Several existing issues and PRs were already created regarding that particular s 1. Be able to get the job status. 1. Be able to specify the number of instances performing a job at any one time. 1. Be able to specify the number of successfully finished instances required to finish a job. -1. Be able to specify backoff policy, when job is continuously failing. +1. Be able to specify a backoff policy, when job is continuously failing. ## Motivation @@ -29,27 +29,31 @@ here would be the ability to implement any type of batch oriented tasks. ## Backoff policy and failed pod limit -By design, Jobs do not have any notion of failure, other than Pod's `restartPolicy` +By design, Jobs do not have any notion of failure, other than a pod's `restartPolicy` which is mistakenly taken as Job's restart policy ([#30243](https://github.com/kubernetes/kubernetes/issues/30243), [#[43964](https://github.com/kubernetes/kubernetes/issues/43964)]). There are -situation where one wants to fail a Job after some amount of retries over certain +situation where one wants to fail a Job after some amount of retries over a certain period of time, due to a logical error in configuration etc. To do so we are going -following fields will be introduced, which will control the exponential backoff -when retrying Job: number of retries and time to retry. The two fields will allow -creating a fine grain control over the backoff policy, limiting the number of retries -over specified period of time. In the case when only one of them is specified -an exponential backoff with duration of 10 seconds and factor of 2 will be applied -in such a way that either time or number is reached. After reaching the limit -a Job will be marked as failed. - -Additionally, to help debug the issue with a job, and limit the impact of having -too many failed pods left around (as mentioned in [#30243](https://github.com/kubernetes/kubernetes/issues/30243)) +to introduce following fields will be introduced, which will control the exponential +backoff when retrying a Job: number of retries and time to retry. The two fields +will allow creating a fine-grained control over the backoff policy, limiting the +number of retries over a specified period of time. If only one of the two fields +is supplied, an exponential backoff with an intervening duration of ten seconds +and a factor of two will be applied, such that either: +* the number of retries will not exceed a specified count, if present, or +* the maximum time elapsed will not exceed the specified duration, if present. + +Additionally, to help debug the issue with a Job, and limit the impact of having +too many failed pods left around (as mentioned in [#30243](https://github.com/kubernetes/kubernetes/issues/30243)), we are going to introduce a field which will allow specifying the maximum number of failed pods to keep around. This number will also take effect if none of the -limits, described above, are set. +limits described above are set. All of the above fields will be optional and will apply no matter which `restartPolicy` -is set on a `PodTemplate`. +is set on a `PodTemplate`. The only difference applies to how failures are counted. +For restart policy `Never` we count actual pod failures (reflected in `.status.failed` +field). With restart policy `OnFailure` we look at pod restarts (calculated from +`.status.containerStatuses[*].restartCount`). ## Implementation @@ -106,13 +110,13 @@ type JobSpec struct { Completions *int // Optional duration in seconds relative to the startTime that the job may be active - // before the system tries to terminate it; value must be positive integer + // before the system tries to terminate it; value must be a positive integer. ActiveDeadlineSeconds *int - // Optional number of retries, before marking this job failed. + // Optional number of retries before marking this job failed. BackoffLimit *int - // Optional time (in seconds), how log a job should be retried before marking it failed. + // Optional time (in seconds) specifying how long a job should be retried before marking it failed. BackoffDeadlineSeconds *int // Optional number of failed pods to retain. From 00def1fb867fa9f29acc006f6fe4acfe725f285b Mon Sep 17 00:00:00 2001 From: Maciej Szulik Date: Tue, 2 May 2017 21:56:09 +0200 Subject: [PATCH 4/7] Address comments --- contributors/design-proposals/job.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/contributors/design-proposals/job.md b/contributors/design-proposals/job.md index 9653e6cc57d..9c0f13c19c5 100644 --- a/contributors/design-proposals/job.md +++ b/contributors/design-proposals/job.md @@ -34,12 +34,12 @@ which is mistakenly taken as Job's restart policy ([#30243](https://github.com/k [#[43964](https://github.com/kubernetes/kubernetes/issues/43964)]). There are situation where one wants to fail a Job after some amount of retries over a certain period of time, due to a logical error in configuration etc. To do so we are going -to introduce following fields will be introduced, which will control the exponential -backoff when retrying a Job: number of retries and time to retry. The two fields -will allow creating a fine-grained control over the backoff policy, limiting the -number of retries over a specified period of time. If only one of the two fields -is supplied, an exponential backoff with an intervening duration of ten seconds -and a factor of two will be applied, such that either: +to introduce following fields, which will control the exponential backoff when +retrying a Job: number of retries and time to retry. The two fields will allow +fine-grained control over the backoff policy, limiting the number of retries over +a specified period of time. If only one of the two fields is supplied, an exponential +backoff with an intervening duration of ten seconds and a factor of two will be +applied, such that either: * the number of retries will not exceed a specified count, if present, or * the maximum time elapsed will not exceed the specified duration, if present. From 5b6d2ce880cad0deb7199e09b1a7ef86ec5f6660 Mon Sep 17 00:00:00 2001 From: Maciej Szulik Date: Thu, 22 Jun 2017 13:43:08 +0200 Subject: [PATCH 5/7] Address comments --- contributors/design-proposals/job.md | 38 +++++++++++++++------------- 1 file changed, 21 insertions(+), 17 deletions(-) diff --git a/contributors/design-proposals/job.md b/contributors/design-proposals/job.md index 9c0f13c19c5..b23c3e02a87 100644 --- a/contributors/design-proposals/job.md +++ b/contributors/design-proposals/job.md @@ -34,11 +34,11 @@ which is mistakenly taken as Job's restart policy ([#30243](https://github.com/k [#[43964](https://github.com/kubernetes/kubernetes/issues/43964)]). There are situation where one wants to fail a Job after some amount of retries over a certain period of time, due to a logical error in configuration etc. To do so we are going -to introduce following fields, which will control the exponential backoff when -retrying a Job: number of retries and time to retry. The two fields will allow -fine-grained control over the backoff policy, limiting the number of retries over -a specified period of time. If only one of the two fields is supplied, an exponential -backoff with an intervening duration of ten seconds and a factor of two will be +to introduce following fields, which will control the backoff policy: a number of +retries and a time to retry (counted from the first failure). The two fields will +allow fine-grained control over the backoff policy, limiting the number of retries +over a specified period of time. If only one of the two fields is supplied, +a backoff with an intervening duration of ten seconds and a factor of two will be applied, such that either: * the number of retries will not exceed a specified count, if present, or * the maximum time elapsed will not exceed the specified duration, if present. @@ -47,13 +47,15 @@ Additionally, to help debug the issue with a Job, and limit the impact of having too many failed pods left around (as mentioned in [#30243](https://github.com/kubernetes/kubernetes/issues/30243)), we are going to introduce a field which will allow specifying the maximum number of failed pods to keep around. This number will also take effect if none of the -limits described above are set. +limits described above are set. By default it will take value of 1, to allow debugging +job issues, but not to flood the cluster with too many failed jobs and their +accompanying pods. All of the above fields will be optional and will apply no matter which `restartPolicy` is set on a `PodTemplate`. The only difference applies to how failures are counted. For restart policy `Never` we count actual pod failures (reflected in `.status.failed` -field). With restart policy `OnFailure` we look at pod restarts (calculated from -`.status.containerStatuses[*].restartCount`). +field). With restart policy `OnFailure` we take an approximate value of pod restarts +(as reported in `.status.containerStatuses[*].restartCount`). ## Implementation @@ -103,24 +105,26 @@ type JobSpec struct { // run at any given time. The actual number of pods running in steady state will // be less than this number when ((.spec.completions - .status.successful) < .spec.parallelism), // i.e. when the work left to do is less than max parallelism. - Parallelism *int + Parallelism *int32 // Completions specifies the desired number of successfully finished pods the // job should be run with. Defaults to 1. - Completions *int + Completions *int32 // Optional duration in seconds relative to the startTime that the job may be active // before the system tries to terminate it; value must be a positive integer. - ActiveDeadlineSeconds *int + // It applies to overall job run time, no matter of the value of completions + // or parallelism parameters. + ActiveDeadlineSeconds *int64 // Optional number of retries before marking this job failed. - BackoffLimit *int + BackoffLimit *int32 // Optional time (in seconds) specifying how long a job should be retried before marking it failed. - BackoffDeadlineSeconds *int + BackoffDeadlineSeconds *int64 // Optional number of failed pods to retain. - FailedPodsLimit *int + FailedPodsLimit *int32 // Selector is a label query over pods running a job. Selector LabelSelector @@ -150,14 +154,14 @@ type JobStatus struct { CompletionTime unversioned.Time // Active is the number of actively running pods. - Active int + Active int32 // Succeeded is the number of pods successfully completed their job. - Succeeded int + Succeeded int32 // Failed is the number of pods failures, this applies only to jobs // created with RestartPolicyNever, otherwise this value will always be 0. - Failed int + Failed int32 } type JobConditionType string From f719b7e3835cae255fcf23b372f212766a5438b3 Mon Sep 17 00:00:00 2001 From: Maciej Szulik Date: Wed, 23 Aug 2017 22:23:41 +0200 Subject: [PATCH 6/7] Renamed BackoffDeadlineSeconds to BackoffSeconds --- contributors/design-proposals/job.md | 23 +++++++++++++---------- 1 file changed, 13 insertions(+), 10 deletions(-) diff --git a/contributors/design-proposals/job.md b/contributors/design-proposals/job.md index b23c3e02a87..c5a21927b3d 100644 --- a/contributors/design-proposals/job.md +++ b/contributors/design-proposals/job.md @@ -34,14 +34,12 @@ which is mistakenly taken as Job's restart policy ([#30243](https://github.com/k [#[43964](https://github.com/kubernetes/kubernetes/issues/43964)]). There are situation where one wants to fail a Job after some amount of retries over a certain period of time, due to a logical error in configuration etc. To do so we are going -to introduce following fields, which will control the backoff policy: a number of -retries and a time to retry (counted from the first failure). The two fields will -allow fine-grained control over the backoff policy, limiting the number of retries -over a specified period of time. If only one of the two fields is supplied, -a backoff with an intervening duration of ten seconds and a factor of two will be -applied, such that either: -* the number of retries will not exceed a specified count, if present, or -* the maximum time elapsed will not exceed the specified duration, if present. +to introduce the following fields, which will control the backoff policy: a number of +retries and an initial time of retry. The two fields will allow fine-grained control +over the backoff policy. Each of the two fields will use a default value if none +is provided, `BackoffLimit` is set by default to 6 and `BackoffSeconds` to 10s. +This will result in the following retry sequence: 10s, 20s, 40s, 1m20s, 2m40s, +5m20s. After which the job will be considered failed. Additionally, to help debug the issue with a Job, and limit the impact of having too many failed pods left around (as mentioned in [#30243](https://github.com/kubernetes/kubernetes/issues/30243)), @@ -56,6 +54,9 @@ is set on a `PodTemplate`. The only difference applies to how failures are coun For restart policy `Never` we count actual pod failures (reflected in `.status.failed` field). With restart policy `OnFailure` we take an approximate value of pod restarts (as reported in `.status.containerStatuses[*].restartCount`). +When `.spec.parallelism` is set to a value higher than 1, the failures are an +overall number (as coming from `.status.failed`) because the controller does not +hold information about failures coming from separate pods. ## Implementation @@ -118,10 +119,12 @@ type JobSpec struct { ActiveDeadlineSeconds *int64 // Optional number of retries before marking this job failed. + // Defaults to 6. BackoffLimit *int32 - // Optional time (in seconds) specifying how long a job should be retried before marking it failed. - BackoffDeadlineSeconds *int64 + // Optional time (in seconds) specifying how long the initial backoff will last. + // Defaults to 10s. + BackoffSeconds *int64 // Optional number of failed pods to retain. FailedPodsLimit *int32 From f65a602d626479e93820324c4ffbe9a203da0dd4 Mon Sep 17 00:00:00 2001 From: Maciej Szulik Date: Fri, 25 Aug 2017 17:29:05 +0200 Subject: [PATCH 7/7] Which backoff fields apply to which restart policy --- contributors/design-proposals/job.md | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/contributors/design-proposals/job.md b/contributors/design-proposals/job.md index c5a21927b3d..4028de33993 100644 --- a/contributors/design-proposals/job.md +++ b/contributors/design-proposals/job.md @@ -49,11 +49,14 @@ limits described above are set. By default it will take value of 1, to allow deb job issues, but not to flood the cluster with too many failed jobs and their accompanying pods. -All of the above fields will be optional and will apply no matter which `restartPolicy` -is set on a `PodTemplate`. The only difference applies to how failures are counted. -For restart policy `Never` we count actual pod failures (reflected in `.status.failed` -field). With restart policy `OnFailure` we take an approximate value of pod restarts -(as reported in `.status.containerStatuses[*].restartCount`). +All of the above fields will be optional and will apply when `restartPolicy` is +set to `Never` on a `PodTemplate`. With restart policy `OnFailure` only `BackoffLimit` +applies. The reason for that is that failed pods are already restarted by the +kubelet with an [exponential backoff](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#restart-policy). +Additionally, failures are counted differently depending on `restartPolicy` +setting. For `Never` we count actual pod failures (reflected in `.status.failed` +field). With `OnFailure`, we take an approximate value of pod restarts (as reported +in `.status.containerStatuses[*].restartCount`). When `.spec.parallelism` is set to a value higher than 1, the failures are an overall number (as coming from `.status.failed`) because the controller does not hold information about failures coming from separate pods.