etcdserver: request is too large and grpc: received message larger than max

[lmatz]

Describe the bug

psql:./nexmark/queries/q15.sql:17: ERROR:  QueryError: internal error: MetadataModel error: Meta store error: internal error: grpc request error: status: ResourceExhausted, message: "grpc: received message larger than max (2182088 vs. 2097152)", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc"} }CREATE_MATERIALIZED_VIEWCREATE_MATERIALIZED_VIEW
psql:./nexmark/queries/q5.sql:35: ERROR:  QueryError: internal error: MetadataModel error: Meta store error: internal error: grpc request error: status: InvalidArgument, message: "etcdserver: request is too large", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc"} }
psql:./nexmark/queries/q16.sql:19: ERROR:  QueryError: internal error: MetadataModel error: Meta store error: internal error: grpc request error: status: ResourceExhausted, message: "grpc: received message larger than max (2417002 vs. 2097152)", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc"} }

To Reproduce

No response

Expected behavior

No response

Additional context

namespace: rwc-3-longevity-20230111-060844
buildkite: https://buildkite.com/risingwave-test/longevity-test/builds/289#01859f71-d994-40ea-a5ab-c4791805061e

Related: #6726

[zwang28]

Seems some too large write to etcd when creating MV.
We've customized etcd config in risedev.
We can enlarge at least --max-request-bytes in longevity test.

BTW this issue is not the same as #6726, because the latter happens during recovery, while this one happens during creating MV. @yezizp2012 Are you aware of any potential large write when creating MV?

[yezizp2012]

BTW this issue is not the same as #6726, because the latter happens during recovery, while this one happens during creating MV. @yezizp2012 Are you aware of any potential large write when creating MV?

Nope, only metadata of streaming job (including actor/fragment/location infos) will be wrote into etcd during MV creation. The issue found in #6726 already included all metadata of tpch MVs, which is about 5M+ in a ci-3cn-1fe environment and obviously larger than the ones in this issue. It's quite weird that the metadata of nexrmark q15/q5/q16 are all larger than 2M in this issue. 🤔

Anyway, I also agree that we'd better enlarge --max-request-bytes in longevity test.

[zwang28]

Because in the test the compute node's parallelism is 32 and there are 3 compute nodes. The write size is proportional to to them.
We can reproduce it locally with ./risedev d ci-3cn-1fe, compute node's parallelism=32 and etcd's default max-request-bytes instead of the enlarged one we've used in risedev, then creating nexmark q15.

With the enlarged max-request-bytes, the error is gone.

[kwannoel]

To enable --max-request-bytes in rwc: https://github.com/risingwavelabs/risingwave-cloud/issues/1493

[kwannoel]

Some additional notes on reproducing this via risedev, thanks to @zwang28 for providing more details offline:

• I used c5a.8xlarge instance (32 vCPU, so parallelism can match)
• override risedev.yml:ci-3cn-1fe to add parallelism: 32.
• erase

  risingwave/src/risedevtool/src/task/etcd_service.rs

  Line 64 in e8831f0

  .arg("--max-request-bytes")

  , so we will match cloud config.

Run the queries:

• https://github.com/risingwavelabs/risingwave-test/blob/main/longevity/nexmark/base_config.sql
• https://github.com/risingwavelabs/risingwave-test/blob/main/longevity/nexmark/queries/q15.sql

And you will see the error.

After https://github.com/risingwavelabs/risingwave-test/pull/212 is running on the buildkite job, we should no longer see this error message in longevity-test.

[fuyufjh]

Fixed by https://github.com/risingwavelabs/risingwave-cloud/pull/1566 ? @mikechesterwang