OOMKilled - gatekeeper:v3.1.0-beta.0

[bjethwan]

It worked fine in one k8s cluster (less used) while on another (heavily used) the container kept restarting. This is right after deploying gatekeeper:v3.1.0-beta.0 (i.e. no templates, no constraints).

kubectl apply -f https://raw.githubusercontent.com/open-policy-agent/gatekeeper/master/deploy/gatekeeper.yaml

Containers:
  manager:
    Container ID:  docker://e3c803389aede20bcece35f48ac2e474ad2e5518093eba2afd043734f34d7aac
    Image:         quay.io/open-policy-agent/gatekeeper:v3.1.0-beta.0
    Port:            8443/TCP
    Host Port:   0/TCP
    Command: 
      /manager
    Args:
      --auditInterval=30
      --port=8443
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137
      Started:      Tue, 10 Dec 2019 03:57:52 +0000
      Finished:     Tue, 10 Dec 2019 03:57:55 +0000
    Ready:          False
    Restart Count:  7
    Limits:
      cpu:     1
      memory:  512Mi
    Requests:
      cpu:     100m
      memory:  256Mi

 Message
 -------
 Successfully assigned gatekeeper-system/gatekeeper-controller-manager-76c4b57755-
 Started container manager
 Pulling image "quay.io/open-policy-agent/gatekeeper:v3.1.0-beta.0"
 Successfully pulled image "quay.io/open-policy-agent/gatekeeper:v3.1.0-beta.0"
 Created container manager
 Back-off restarting failed container

[bjethwan]

    Limits:
      cpu:     1
      memory:  1Gi
    Requests:
      cpu:     100m
      memory:  512Mi

$ kg get pods
NAME                                             READY   STATUS             RESTARTS   AGE
gatekeeper-controller-manager-5f96b7f746-wlxff   0/1     CrashLoopBackOff   3          2m5s

$ kg get pods
NAME                                             READY   STATUS      RESTARTS   AGE
gatekeeper-controller-manager-5f96b7f746-wlxff   0/1     OOMKilled   4          2m23s

[bjethwan]

It's running now.

    Limits:
      cpu:     1
      memory:  2Gi
    Requests:
      cpu:     100m
      memory:  1Gi

$ kg get pods
NAME                                             READY   STATUS    RESTARTS   AGE
gatekeeper-controller-manager-54cb9dbcf6-zmgdf   1/1     Running   1          6m37s

[bjethwan]

One thing doesn't make sense - everything worked fine when I deployed an older version v3.0.4-beta.2, I didn't have to bump up the memory limits/requests.

[maxsmythe]

Was the older version working fine on the same cluster where the new one OOMs?

How large is the OOMing cluster in terms of:

• # of cached resources (if any)
• # of constraints
• # of templates
• Amount of memory used by old version and new, respectively?

[maxsmythe]

Sorry, just saw the no constraints/templates, I am interested in the other two data points, though,

[bjethwan]

@maxsmythe I am sorry for the slow response. It was a fresh install...it my sanbox cluster in aws....3 masters, 20 worker....No gatekeep sync srt at the moment

[bjethwan]

But this is team's sandbox .....so a lot of churns

[maxsmythe]

Is this still happening? I haven't heard of any other reports of OOMs.

[maxsmythe]

Closing this out for staleness, please re-open if it is still ongoing.