Verification : verifying "adrianmo/pravega-operator:pr-145-1" build to validate Bookie restarted with exception "java.lang.OutOfMemoryError: Java heap space" Issue

[vedanthh]

Verification : verifying adrianmo/pravega-operator:pr-145-1 build to validate Bookie restarted with exception "java.lang.OutOfMemoryError: Java heap space" Issue - #141 #144 PR #145

Environment details: PKS / K8 with medium cluster:

3 master: xlarge: 4 CPU, 16 GB Ram, 32 GB Disk
5 worker: 2xlarge: 8 CPU, 32 GB Ram, 64 GB Disk
Tier-1 storage is from VSAN datastore
Tier-2 storage curved on NFS Client provisioner using Isilon as backend

Pravega version: zk-closed-client-issue-0.5.0-2161.60655bf
Zookeeper Operator : pravega/zookeeper-operator:0.2.1
Pravega Operator: adrianmo/pravega-operator:pr-145-1

[vedanthh]

Observing Bookie restart during longevity run with adrianmo/pravega-operator:pr-145-1 build after ~10 hrs with custom resource requests and limits.

# kubectl describe po/pravega-bookie-0
    State:          Running
      Started:      Tue, 26 Mar 2019 18:38:49 -0400
    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137
      Started:      Tue, 26 Mar 2019 07:47:32 -0400
      Finished:     Tue, 26 Mar 2019 18:38:49 -0400
    Ready:          True
    Restart Count:  1
    Limits:
      cpu:     2
      memory:  4Gi
    Requests:
      cpu:      1
      memory:   2Gi

# kubectl get po
NAME                                                 READY     STATUS    RESTARTS   AGE
pravega-bookie-0                                     1/1       Running   1          16h
pravega-bookie-1                                     1/1       Running   1          16h
pravega-bookie-2                                     1/1       Running   0          16h
pravega-bookie-3                                     1/1       Running   1          16h
pravega-operator-778b8c9485-tnhjl                    1/1       Running   1          16h
pravega-pravega-controller-c67d6b758-gcmht           1/1       Running   1          16h
pravega-pravega-controller-c67d6b758-j6btb           1/1       Running   1          16h
pravega-pravega-segmentstore-0                       1/1       Running   0          16h
pravega-pravega-segmentstore-1                       1/1       Running   0          16h
pravega-pravega-segmentstore-2                       1/1       Running   0          16h
pravega-pravega-segmentstore-3                       1/1       Running   0          16h
pravega-zk-0                                         1/1       Running   0          16h
pravega-zk-1                                         1/1       Running   0          16h
pravega-zk-2                                         1/1       Running   0          16h
shaka-zulu-nfs-client-provisioner-59d7f8f84c-xvktb   1/1       Running   0          5d
zookeeper-operator-685bfcbbc5-tjtjr                  1/1       Running   0          5d

Will be validating issue with adrianmo/pravega-operator:pr-145-2 #151

[adrianmo]

@vedanthh this issue was reported in #141 and fixed in #145. Please use the latest official operator, i.e., pravega/pravega-operator:0.3.2, which contains the fix for this.

[vedanthh]

Longevity Ran fine for ~20 hours with adrianmo/pravega-operator:pr-145-2 still no bookie restart and OOM observed , Hence closing the issue.