MetricController: Run only a single job per task

This changes the `spec.concurrencyPolicy` of the metric collector cron-job from "Allow" (default) to "Forbid". The cronjob used to create a new job even if the previous job had not succeeded. On high-load clusters this could lead to a high number of jobs which never finished. This fixed #659
kubeflow · Jun 22, 2019 · 7471333 · 7471333
1 parent c81692c
commit 7471333
Show file tree

Hide file tree

Showing 2 changed files with 2 additions and 0 deletions.
diff --git a/manifests/v1alpha1/studyjobcontroller/metricsControllerConfigMap.yaml b/manifests/v1alpha1/studyjobcontroller/metricsControllerConfigMap.yaml
@@ -14,6 +14,7 @@ data:
       schedule: "*/1 * * * *"
       successfulJobsHistoryLimit: 0
       failedJobsHistoryLimit: 1
+      concurrencyPolicy: Forbid
       jobTemplate:
         spec:
           backoffLimit: 0

diff --git a/manifests/v1alpha2/katib-controller/metricsControllerConfigMap.yaml b/manifests/v1alpha2/katib-controller/metricsControllerConfigMap.yaml
@@ -14,6 +14,7 @@ data:
       schedule: "*/1 * * * *"
       successfulJobsHistoryLimit: 0
       failedJobsHistoryLimit: 1
+      concurrencyPolicy: Forbid
       jobTemplate:
         spec:
           backoffLimit: 0