Meta node will restart when running nexmark q9

[huangjw806]

Describe the bug

meta node will restart when running nexmark q9.

To Reproduce

image-tag:nightly-20230207

1. Cloud deploys a single-node risingwave cluster.
2. prepare 1 billion messages in kafka source.
3. run nexmark q9.

pipeline link: https://buildkite.com/risingwave-test/nexmark-benchmark/builds/366#01862ca0-2a86-4bff-911a-d0178ec9287f

The environment is gone, cloudwatch logs link: https://ap-southeast-1.console.aws.amazon.com/cloudwatch/home?region=ap-southeast-1#logsV2:log-groups/log-group/$252Frisingwave$252Frls-apse1-eks-a$252Fnexmark-7-8-9-16-daily-20230207

[shanicky]

It seems that the etcd connection is frequently disconnected causing the election watch client on the meta node to be disconnected causing meta-0 to restart

[huangjw806]

The sameissue appeared in the pipeline last night, this time when it happened in drop mv：

buildkite link：https://buildkite.com/risingwave-test/nexmark-benchmark/builds/398#018655d3-beb0-4394-87e3-9b128a7a9753

cloudwatch logs link：https://ap-southeast-1.console.aws.amazon.com/cloudwatch/home?region=ap-southeast-1#logsV2:log-groups/log-group/$252Frisingwave$252Frls-apse1-eks-a$252Fnexmark-7r-8-9-16-daily-20230215

[fuyufjh]

Any updates?

[shanicky]

This issue should have been resolved with meta in new version.