bug: hummock error NoSuchKey

[zwang28]

Describe the bug

Namespace: rwc-3-longevity-20230106-065737
Pod: risingwave-compute-1

2023-01-06T07:02:32.250276Z ERROR risingwave_storage::monitor::monitored_store: Failed in iter: Hummock error: ObjectStore failed with IO error Internal error: read \""rls-apse1-eks-a-rwc-3-longevity-20230106-065737/240/54.data\"" in block Some(BlockLocation { offset: 14477472, size: 65902 }) failed, error: NoSuchKey: The specified key does not exist."

cn-1.csv

To Reproduce

No response

Expected behavior

No response

Additional context

compute-node-0 hits #7002 meanwhile.

[zwang28]

This issue is not a kernel bug. It's caused by

1. Firstly the driving test script (test_runner.sh) fails due to create mv timeout.
   subprocess.TimeoutExpired: Command '['psql', 'postgres://dev:dev@rls-apse1-eks-a.risingwave-cloud.xyz:4566/dev?options=--tenant%3Drwc-3-longevity-20230107-045111', '-a', '-f', './nexmark/queries/q101.sql']' timed out after 89.999904184 seconds
2. Then the test script does clean up: rwc delete tenant, which will shutdown risingwave cluster and empty S3 bucket.

So I think what happens is after calling rwc delete tenant, risingwave cluster is still alive while S3 is being emptying, thus results in risingwave reading a nonexistent S3 object. @mikechesterwang Can you help to clarify the order of deleting risingwave cluster and S3 in rwc delete tenant?

To confirm my guessing, I run other tests (with S3 sdk's connection timeout increased to workaround #7002 temporarily).

• If I use the original test script, this issue is reproduced every time.
• If I tune the test script to not delete tenant after failing, this issue doesn't appear anymore.