bug: barrier stuck for minutes

[fuyufjh]

Describe the bug

The metrics diagram looks broken, because no barriers are collected during a long time.

Both barrier number and latency are very high.

During the whole running https://g-2927a1b4d9.grafana-workspace.us-east-1.amazonaws.com/d/EpkBw5W4k/risingwave-test-dashboard?org[…]90631&from=1683537069000&to=1683623809000

To Reproduce

Longevity test

Expected behavior

No response

Additional context

Await-Tree Dump of Compute Node 3.txt
Await-Tree Dump of Compute Node 4.txt
Await-Tree Dump of Compute Node 5.txt

See more context in Slack channel: https://risingwave-labs.slack.com/archives/C034TRPKN1F/p1683627627166979

[kwannoel]

Slack notification: https://risingwave-labs.slack.com/archives/C048NM5LNKX/p1683623832327539
Buildkite: https://buildkite.com/risingwave-test/longevity-kubebench/builds/302

[kwannoel]

Test configuration:

[fuyufjh]

The problem is caused by:

1. The state of distinct table of aggregation has bad locality
2. When the amount of data reached become larger than meta cache, the meta cache miss caused high read latency
3. The read latency caused longer barrier latency, since the number of rows in one epoch is constant (not well backpressured, which may be a potential problem to be improved)