[Feature]: Evaluate Profile-Guided Optimization (PGO) and LLVM BOLT

[zamazan4ik]

Hi!

Recently I did many Profile-Guided Optimization (PGO) benchmarks on multiple projects (including many databases like PostgreSQL, MongoDB, Redis, and many others) - the results are available here, all databases results are located here. So that's why I think it's worth trying to apply PGO to OceanBase.

I can suggest the following things to do:

• Evaluate PGO's applicability and benchmark results to OceanBase.
• If PGO helps to achieve better performance - add a note to OceanBase's documentation about that. In this case, users and maintainers will be aware of another optimization opportunity for OceanBase.
• Provide PGO integration into the build scripts. It can help users and maintainers easily apply PGO for their own workloads.
• PGO-optimize prebuilt OceanBase binaries (if it's possible to prepare a generic enough test workload).

Here are some PGO integration examples in the existing build scripts in other projects:

• Rustc: a CI script for the multi-stage build
• GCC:
  • Official docs, section "Building with profile feedback" (even AutoFDO build is supported)
  • A part in a "wonderful" configure script
• Clang: Docs
• Python:
  • CPython: README
  • Pyston: README
• Go: Bash script
• V8: Bazel flag
• ChakraCore: Scripts
• Chromium: Script
• Firefox: Docs
  • Thunderbird has PGO support too
• PHP - Makefile command and old Centminmod scripts
• MySQL: CMake script
• YugabyteDB: GitHub commit
• FoundationDB: Script
• Zstd: Makefile
• Foot: Scripts
• Windows Terminal: GitHub PR
• Pydantic-core: GitHub PR

Some PGO documentation examples:

• ClickHouse: https://clickhouse.com/docs/en/operations/optimizing-performance/profile-guided-optimization
• Databend: https://databend.rs/doc/contributing/pgo
• Vector: https://vector.dev/docs/administration/tuning/pgo/
• Nebula: https://docs.nebula-graph.io/3.5.0/8.service-tuning/enable_autofdo_for_nebulagraph/
• GCC: Official docs, section "Building with profile feedback" (even AutoFDO build is supported)
• Clang:
  • https://llvm.org/docs/HowToBuildWithPGO.html
  • https://llvm.org/docs/AdvancedBuilds.html

After PGO, I can suggest evaluating LLVM BOLT as an additional optimization step after PGO.

[caifeizhi]

Thank you for your suggestion.
Currently, the code in OceanBase already supports PGO, and you can find the code at:

oceanbase/cmake/Env.cmake

Line 55 in e4e09ec

if(ENABLE_AUTO_FDO)

You are welcome to provide any further optimization suggestions for this.
Additionally, we would appreciate it if you could add our repository to the awesome repository related to FGO.

[zamazan4ik]

Currently, the code in OceanBase already supports PGO, and you can find the code at

Thanks for pointing to the script! I didn't find it because I had been searching for PGO instead of FDO... :)

I have some questions about the current FDO implementation in OceanBase:

1. Do you have the benchmark results of enabling FDO on OceanBase? I am interested in comparing Release vs Release + PGO OceanBase versions. So I will be able to put your benchmarks to my repo as an example.
2. I see you use Sampling (AutoFDO) approach. Did you try to use the Instrumented FDO instead? If yes, could you please describe why you chose AutoFDO instead? My guess is that Instrumented FDO has just too big performance overhead compared to AutoFDO but it's just my guess.
3. How did you collect the committed to the repo FDO profiles? How can I try to reproduce these profiles?
4. Do you FDO-optimize prebuilt binaries in your releases (e.g. available on GitHub Releases page)?

Additionally, we would appreciate it if you could add our repository to the awesome repository related to FGO.

Of course! Just did it :)

[caifeizhi]

1. Do you have the benchmark results of enabling FDO on OceanBase? I am interested in comparing Release vs Release + PGO OceanBase versions. So I will be able to put your benchmarks to my repo as an example.

about 15% improvement;

2. I see you use Sampling (AutoFDO) approach. Did you try to use the Instrumented FDO instead? If yes, could you please describe why you chose AutoFDO instead? My guess is that Instrumented FDO has just too big performance overhead compared to AutoFDO but it's just my guess.
3. How did you collect the committed to the repo FDO profiles? How can I try to reproduce these profiles?

@Naynahs can you please take a look

4. Do you FDO-optimize prebuilt binaries in your releases (e.g. available on GitHub Releases page)?

yes, we use AUTOFDO=ON when rpm build, see it:

oceanbase/build.sh

Line 207 in 223cf90

do_build "$@" -DOB_BUILD_RPM=ON -DCMAKE_BUILD_TYPE=RelWithDebInfo -DOB_USE_LLD=$LLD_OPTION -DENABLE_FATAL_ERROR_HANG=OFF -DENABLE_AUTO_FDO=ON -DENABLE_THIN_LTO=ON -DOB_STATIC_LINK_LGPL_DEPS=$STATIC_LINK_LGPL_DEPS_OPTION

[ruclz]

Yeah, I also want to know more about oceanbase, how to collect the committed to the repo FDO profiles? Thanks very much.