Skip to content

Commit

Permalink
feat(k8s): Add resource limits to all per-instance resources (#1842)
Browse files Browse the repository at this point in the history
Setting resource limits is good for 2 reasons:
- Java containers set the heap size based off how much memory they have available. No limits mean they think they've got the whole machine. Setting limits fixes this effectively. Backend memory usage goes down from 3GB to 500MB. Finally things get garbage collected.
- Step towards limiting swapping which currently can happen. We want to prioritize central services and non-kubernetes resources over preview deployments.

I set memory requests and limits for all, but only cpu requests, as cpu limits are not very useful (we are not CPU limited usually and throttling works well usually).

(cherry picked from commit ead6afc)
corneliusroemer committed May 10, 2024
1 parent 9d2b418 commit d799b67
Showing 8 changed files with 65 additions and 3 deletions.
Original file line number Diff line number Diff line change
@@ -22,6 +22,12 @@ spec:
containers:
- name: loculus-keycloak-database
image: postgres:latest
resources:
requests:
memory: "100Mi"
cpu: "100m"
limits:
memory: "100Mi"
ports:
- containerPort: 5432
env:
9 changes: 9 additions & 0 deletions kubernetes/loculus/templates/keycloak-deployment.yaml
Original file line number Diff line number Diff line change
@@ -40,6 +40,12 @@ spec:
- name: keycloak
# TODO #1221
image: quay.io/keycloak/keycloak:23.0
resources:
requests:
memory: "500Mi"
cpu: "20m"
limits:
memory: "3Gi"
env:
- name: DB_ADDR
valueFrom:
@@ -87,6 +93,9 @@ spec:
value: "{{ include "keycloakUrl" . }}"
- name: KC_FEATURES
value: "declarative-user-profile"
# see https://github.com/keycloak/keycloak/blob/77b58275ca06d1cbe430c51db74479a7e1b409b5/quarkus/dist/src/main/content/bin/kc.sh#L95-L150
- name: KC_RUN_IN_CONTAINER
value: "true"
args:
- "start"
- "--import-realm"
18 changes: 18 additions & 0 deletions kubernetes/loculus/templates/lapis-silo-deployment.yaml
Original file line number Diff line number Diff line change
@@ -27,6 +27,12 @@ spec:
containers:
- name: silo
image: ghcr.io/genspectrum/lapis-silo:{{ $.Values.imageTags.lapisSilo }}
resources:
requests:
memory: "100Mi"
cpu: "10m"
limits:
memory: "2Gi"
ports:
- containerPort: 8081
args:
@@ -36,6 +42,12 @@ spec:
mountPath: /data
- name: lapis
image: ghcr.io/genspectrum/lapis-v2:{{ $.Values.imageTags.lapis }}
resources:
requests:
memory: "100Mi"
cpu: "10m"
limits:
memory: "500Mi"
ports:
- containerPort: 8080
args:
@@ -50,6 +62,12 @@ spec:
- name: silo-preprocessing
image: ghcr.io/genspectrum/lapis-silo:{{ $.Values.imageTags.lapisSilo }}
imagePullPolicy: IfNotPresent
resources:
requests:
memory: "100Mi"
cpu: "10m"
limits:
memory: "1Gi"
command:
- sh
- /silo_import_wrapper.sh
6 changes: 6 additions & 0 deletions kubernetes/loculus/templates/loculus-backend.yaml
Original file line number Diff line number Diff line change
@@ -26,6 +26,12 @@ spec:
- name: backend
image: "ghcr.io/loculus-project/backend:{{ $dockerTag }}"
imagePullPolicy: Always
resources:
requests:
memory: "640Mi"
cpu: "100m"
limits:
memory: "1Gi" # Backend requires at least 635741K of memory
livenessProbe:
httpGet:
path: "/actuator/health/liveness"
6 changes: 6 additions & 0 deletions kubernetes/loculus/templates/loculus-database-standin.yaml
Original file line number Diff line number Diff line change
@@ -21,6 +21,12 @@ spec:
containers:
- name: database
image: postgres:latest
resources:
requests:
memory: "200Mi"
cpu: "100m"
limits:
memory: "2Gi"
ports:
- containerPort: 5432
env:
12 changes: 12 additions & 0 deletions kubernetes/loculus/templates/loculus-ingest-deployment.yaml
Original file line number Diff line number Diff line change
@@ -26,6 +26,12 @@ spec:
- name: ingest-{{ $key }}
image: {{ $value.ingest.image}}:{{ $dockerTag }}
imagePullPolicy: Always
resources:
requests:
memory: "500Mi"
cpu: "100m"
limits:
memory: "10Gi"
env:
- name: KEYCLOAK_INGEST_PASSWORD
valueFrom:
@@ -74,6 +80,12 @@ spec:
- name: ingest-{{ $key }}
image: {{ $value.ingest.image}}:{{ $dockerTag }}
imagePullPolicy: Always
resources:
requests:
memory: "1Gi"
cpu: "100m"
limits:
memory: "10Gi"
env:
- name: KEYCLOAK_INGEST_PASSWORD
valueFrom:
Original file line number Diff line number Diff line change
@@ -34,6 +34,12 @@ spec:
- name: preprocessing-{{ $organism }}
image: {{ $processingConfig.image}}:{{ $dockerTag }}
imagePullPolicy: Always
resources:
requests:
memory: "40Mi"
cpu: "10m"
limits:
memory: "1Gi"
env:
- name: KEYCLOAK_PASSWORD
valueFrom:
5 changes: 2 additions & 3 deletions kubernetes/loculus/templates/loculus-website.yaml
Original file line number Diff line number Diff line change
@@ -28,11 +28,10 @@ spec:
imagePullPolicy: Always
resources:
requests:
memory: "1300Mi"
memory: "200Mi"
cpu: "100m"
limits:
memory: "2000Mi"
cpu: "1000m"
memory: "1Gi"
ports:
- containerPort: 3000
volumeMounts:

0 comments on commit d799b67

Please sign in to comment.