Benchmark runner

[Veeupup]

No description provided.

[ZhiHanZ]

I think you could take a look at the runner on test-infra

[ZhiHanZ]

FYI, the runner controller version is outdated, bump runner version would be nice as well

[Veeupup]

It seems test-infra runner can be used for SQL bench?

here I mean that we could bench for some functions in common/functions , such as aggregate and scalars functions. And now we have benches in query/benches, maybe we could combine them( SQL bench, function bench and maybe more performence sensitive components in future) all together in databend?

do we need to combine them together ?
cc @sundy-li @BohuTANG

[ZhiHanZ]

test-infra could do the following things:

1. Remote build and containerization support
2. provision temporary k8s cluster to compare target commit with reference commit(main, master, other commits etc)
3. upload compare results to S3 repo

perf-tool is another containered wrapper for databend-benchmark binary, it supports:

1. run performance test again given input queries(https://github.com/datafuselabs/databend/blob/main/tests/perfs/perfs.yaml)
2. generate html form reports

[ZhiHanZ]

I think we need to finish the following parts for continuous benchmark

1. support to provision eks as the temporary k8s cluster to host query workload
2. run workload on given cluster after each nightly release
3. upload result html to website

[sundy-li]

SQL bench is enough for database benchmarks.
query/benches are aimed to bench unit function benchmark, but it needs to code manually, yet it is hard to cover the performance in SQL's lifetime.

[PsiACE]

https://github.com/boa-dev/criterion-compare-action

We can try to support this action with test-infra

[BohuTANG]

For the benchmark and output with a uniform format, I think https://github.com/datafuselabs/databend/blob/main/query/src/bin/databend-benchmark.rs can do it well.

So, the runner here is that how we run the difference benchmarks with the test-infra, @ZhiHanZ has give some suggestions above.

[BohuTANG]

SQL bench is enough for database benchmarks. query/benches are aimed to bench unit function benchmark, but it needs to code manually, yet it is hard to cover the performance in SQL's lifetime.

Does the query/benches not used anymore? It seems a long time we do not run them

[PsiACE]

I think this task is obsolete and we now have a working benchmark mechanism.