v2.0.0-rt.1 tpch q15 performance degradation

[cyliu0]

Describe the bug

http://metabase.risingwave-cloud.xyz/question/26990-rw-avg-source-throughput-release?start_date=2024-07-01&testbed=medium-1cn-affinity&workload=tpch-q15

Error message/log

No response

To Reproduce

No response

Expected behavior

No response

How did you deploy RisingWave?

No response

The version of RisingWave

No response

Additional context

No response

[lmatz]

Is it an issue of recording the wrong end time due to some unknown reason?

https://grafana.test.risingwave-cloud.xyz/d/EpkBw5W4k/risingwave-dev-dashboard?orgId=1&var-datasource=Prometheus:%20test-useast1-eks-a&from=1723828156000&to=1723829176000&var-namespace=tpch-benchmark-wyw1vg

https://grafana.test.risingwave-cloud.xyz/d/EpkBw5W4k/risingwave-dev-dashboard?from=1724158206000&orgId=1&to=1724160025000&var-datasource=Prometheus:+test-useast1-eks-a&var-namespace=tpch-benchmark-ngn0u

When clicking by the URL, the second screenshot has a period of 0 throughput at the second half period, but somehow it is still counted into the calculation

I think the system is ok, no regression if you exclude the second half period

[lmatz]

what are the criteria for determining the end time?

[cyliu0]

We use Write Batch Ops = 0 as the end condition. https://grafana.test.risingwave-cloud.xyz/d/EpkBw5W4k/risingwave-dev-dashboard?from=1724158206000&orgId=1&to=1724160025000&var-datasource=Prometheus:%20test-useast1-eks-a&var-namespace=tpch-benchmark-ngn0u&editPanel=128

[hzxa21]

Is it an issue of recording the wrong end time due to some unknown reason?

https://grafana.test.risingwave-cloud.xyz/d/EpkBw5W4k/risingwave-dev-dashboard?orgId=1&var-datasource=Prometheus:%20test-useast1-eks-a&from=1723828156000&to=1723829176000&var-namespace=tpch-benchmark-wyw1vg

https://grafana.test.risingwave-cloud.xyz/d/EpkBw5W4k/risingwave-dev-dashboard?from=1724158206000&orgId=1&to=1724160025000&var-datasource=Prometheus:+test-useast1-eks-a&var-namespace=tpch-benchmark-ngn0u

When clicking by the URL, the second screenshot has a period of 0 throughput at the second half period, but somehow it is still counted into the calculation

I think the system is ok, no regression if you exclude the second half period

From the second grafana, I saw that there is actor throughput between fragment 135 and 141 after source throughput drops to 0:

[lmatz]

solved by #18302