feat(datahub-upgrade): Restore Indices Resources (#152)

* <feat>(datahub-upgrade): Restore Indices Resources

This PR adds the ability for end-users to specify resource requests and limits
for the datahub upgrade job 'restore indices'. This is needed for situations
where the restore indices job will OOM due to a large backfill of data. I've
seen this happening with our deployment of datahub and am unable to tune
the resources for the restore indices job because the current requests
and limits are hard-coded.

charts/datahub/Chart.yaml

@@ -4,10 +4,10 @@ description: A Helm chart for LinkedIn DataHub
 type: application
 # This is the chart version. This version number should be incremented each time you make changes
 # to the chart and its templates, including the app version.
-version: 0.2.90
+version: 0.2.91
 # This is the version number of the application being deployed. This version number should be
 # incremented each time you make changes to the application.
-appVersion: 0.8.43
+appVersion: 0.8.44
 dependencies:
   - name: datahub-gms
     version: 0.2.5

charts/datahub/README.md

@@ -34,6 +34,7 @@ helm install datahub datahub/datahub --values <<path-to-values-file>>
 | datahubUpgrade.podSecurityContext | object | `{}` | Pod security context for datahubUpgrade jobs |
 | datahubUpgrade.securityContext | object | `{}` | Container security context for datahubUpgrade jobs |
 | datahubUpgrade.podAnnotations | object | `{}` | Pod annotations for datahubUpgrade jobs |
+| datahubUpgrade.restoreIndices.resources | object | '{}' | Kube Resource definitions for the datahub upgrade job 'restore indices' |
 | elasticsearchSetupJob.enabled | bool | `true` | Enable setup job for elasicsearch |
 | elasticsearchSetupJob.image.repository | string | `"linkedin/datahub-elasticsearch-setup"` | Image repository for elasticsearchSetupJob |
 | elasticsearchSetupJob.image.tag | string | `"v0.8.43"` | Image repository for elasticsearchSetupJob |

charts/datahub/templates/datahub-upgrade/datahub-restore-indices-job-template.yml

@@ -87,12 +87,7 @@ spec:
                 {{- toYaml . | nindent 16 }}
               {{- end }}
               resources:
-                limits:
-                  cpu: 500m
-                  memory: 512Mi
-                requests:
-                  cpu: 300m
-                  memory: 256Mi
+                {{- toYaml .Values.datahubUpgrade.restoreIndices.resources | nindent 16}}
           {{- with .Values.datahubUpgrade.nodeSelector }}
           nodeSelector:
             {{- toYaml . | nindent 12 }}

charts/datahub/values.yaml

@@ -101,6 +101,14 @@ datahubUpgrade:
   securityContext: {}
     # runAsUser: 1000
   podAnnotations: {}
+  restoreIndices:
+    resources:
+      limits:
+        cpu: 500m
+        memory: 512Mi
+      requests:
+        cpu: 300m
+        memory: 256Mi
 
 global:
   graph_service_impl: neo4j