eviction strategy can be aggressive sometimes unexpectedly

[lmatz]

Slack history:
https://risingwave-labs.slack.com/archives/C03GTKX3C8G/p1676901892312759
https://risingwave-labs.slack.com/archives/C03GTKX3C8G/p1676957678684269

You can see the memory starts growing at about 00:53, and that is when watermark start to grow to evict cache. But the memory keeps growing until just pass 00:55. So the memory eviction start to be aggressive to prevent OOM. The memory drop to 0 because the watermark hit the upper bound (current epoch), so it evicts all cache.
by @yuhao-su

[lmatz]

It OOMed when The memory allocated by Jemalloc hits 0

[yuhao-su]

It OOMed when The memory allocated by Jemalloc hits 0

Is it possible the memory drop to 0 because it got killed?

[lmatz]

It OOMed when The memory allocated by Jemalloc hits 0

Is it possible the memory drop to 0 because it got killed?

I think so, it OOMed first, i.e. use more memory resources than it requested from the risingwave-operator, and then it gets killed I suppose

[lmatz]

I mean kube-bench never voluntarily kills a pod if the pod does noting wrong. cc @huangjw806
And this phenomenon almost(I manually checked 3-4 Grafana dashboards in the recent month) always happens in every performance testing pipeline every day

[huangjw806]

I mean kube-bench never voluntarily kills a pod if the pod does noting wrong.

During the test, kube-bench will not actively kill the risingwave pod. It may be restarted by k8s only if it encounters issues.

[lmatz]

OOM may be caused by #8125

[lmatz]

Reopen if it appears again