nightly-20241203 perf degradation

[cyliu0]

Describe the bug

+--------------------------------------------------------------+--------------+------------+----------------------------------------------------+---------------------+-----------------------------+-------------------------------+
| BENCHMARK NAME                                               | EXECUTION ID | STATUS     | KEY METRICS                                        | FLUCTUATION OF BEST | FLUCTUATION OF LAST 10 DAYS | FLUCTUATION OF LAST EXECUTION |
+--------------------------------------------------------------+--------------+------------+----------------------------------------------------+---------------------+-----------------------------+-------------------------------+
| nexmark-q15-blackhole-8c32g-1cn                              |        44386 | Negative   | avg-source-output-rows-per-second                  | -18.87%             | -13.88%                     | -14.15%                       |
| nexmark-q4-blackhole-8c32g-1cn                               |        44388 | Negative   | avg-source-output-rows-per-second                  | -17.05%             | -10.47%                     | -17.05%                       |
| nexmark-q16-blackhole-8c32g-1cn                              |        44389 | Negative   | avg-source-output-rows-per-second                  | -26.66%             | -23.07%                     | -22.29%                       |
| nexmark-q17-blackhole-8c32g-1cn                              |        44395 | Negative   | avg-source-output-rows-per-second                  | -23.29%             | -13.97%                     | -10.30%                       |
| nexmark-q18-blackhole-8c32g-1cn                              |        44399 | Negative   | avg-source-output-rows-per-second                  | -27.74%             | -22.46%                     | -21.98%                       |
| nexmark-q19-blackhole-8c32g-1cn                              |        44405 | Negative   | avg-source-output-rows-per-second                  | -24.95%             | -19.74%                     | -19.53%                       |
| nexmark-q20-blackhole-8c32g-1cn                              |        44409 | Negative   | avg-source-output-rows-per-second                  | -23.25%             | -17.72%                     | -17.16%                       |
| nexmark-q101-blackhole-8c32g-1cn                             |        44414 | Negative   | avg-source-output-rows-per-second                  | -24.35%             | -19.67%                     | -19.72%                       |
| nexmark-q102-blackhole-8c32g-1cn                             |        44418 | Negative   | avg-source-output-rows-per-second                  | -34.84%             | -25.38%                     | -24.49%                       |
| nexmark-q103-blackhole-8c32g-1cn                             |        44423 | Negative   | avg-source-output-rows-per-second                  | -22.24%             | -15.52%                     | -12.42%                       |
| nexmark-q104-blackhole-8c32g-1cn                             |        44428 | Negative   | avg-source-output-rows-per-second                  | -24.22%             | -17.49%                     | -15.48%                       |
| nexmark-q105-blackhole-8c32g-1cn                             |        44430 | Negative   | avg-source-output-rows-per-second                  | -32.73%             | -26.26%                     | -26.34%                       |

New testbed dashboard http://metabase.risingwave-cloud.xyz/dashboard/1405-nexmark-8c32g-1cn-rw-source-throughput

Most degradation queries belongs to this job https://buildkite.com/risingwave-test/nexmark-benchmark/builds/4878

https://grafana.test.risingwave-cloud.xyz/d/EpkBw5W4k/risingwave-dev-dashboard?orgId=1&from=1733241886137&to=1733266417993&var-datasource=cdtasocg64074c&var-namespace=nexmark-bs-15-105-daily-20241203&var-instance=benchmark-risingwave&var-pod=All&var-component=All&var-table=All

Recently we migrated the testbed machine from 8c16g to 8c32g. You can refer to the old perf dash for this http://metabase.risingwave-cloud.xyz/dashboard/241-nexmark-blackhole-1cn-anti-affinity-rw-avg-source-throughput?namespace=daily

Error message/log

No response

To Reproduce

No response

Expected behavior

No response

How did you deploy RisingWave?

No response

The version of RisingWave

nightly-20241203

Additional context

https://github.com/risingwavelabs/rw-commits-history?tab=readme-ov-file#nightly-20241203

nightly-20241203

compare changes with previous nightly build

• e3337876903cbfa53231407f39a58f1ec53b03d5 feat(storage): Provide table id filter for sstable_iterator (#19582)
• c2ee827eeed2912a1c94ceca657d3fa36da43851 refactor(meta): generalize replace table to replace job (#18935)
• 70d0275561601f5ad020e2e4e373b8675c923d06 feat: remove flink mock sink from Java connector codebase (#19653)
• d60649262c4fa4ce841124e8ee75b24175923b5e fix(storage): fix the atomicity problem after split/merge operation (#19625)
• 78e39a8b517561a9054a5d422b74a46df5c85ca4 chore(dep): switch foyer to v0.13.0 release version (#19648)
• e9e3b24442cc55755c826ec9264fff909e553ef7 chore: set recursion_limit for sink bench crate (#19651)
• a41866fcd3b6e722e19355c60a839b76319f6301 feat(metrics): add internal latency of actors, MVs and sinks (#19639)
• c3750ebbbe80cd979ff236d621acb48e3c8f7e16 chore(deps): Bump pprof from 0.13.0 to 0.14.0 (#19340)
• 84566dc586c8c2e4d03971180a77427ab0b512c7 feat: make Nats jetstream Source work in parallel (#19529)
• 563c70fda37b83d44415dc448416ecaaca85058f refactor(meta): unify DdlType and StreamingJobType (#19630)
• 2c0d286ddb2ef25e722dc330550eec938676d2fd fix: bind under schema path for function, connection, subscription to avoid misjudgment of not found error (#19640)
• e9c15eb55cc246097eff55b7624124d45167343b chore(deps): Bump bytes from 1.7.2 to 1.9.0 (#19626)
• 69158c47d16d88569a3ef3adf061cfa2f0721211 feat: telemetry for connection create and ref (#19638)
• 6f2d0c33848871a5f24b2537e0c713fc66afcf96 feat(frontend): support alter source pause/resume (#19636)
• 845ed68183de5cb63525417063eb4b30064b9707 feat: add rw_internal_table_info to identity which streaming job the internal table belongs (#19642)

[MrCroxx]

The block cache hit ratio dropped. Let me check.