Evaluate using Profile-Guided Optimization (PGO) and Post-Link Optimization (PLO)

[zamazan4ik]

Hi!

Recently I checked Profile-Guided Optimization (PGO) improvements on multiple projects. The results are available here. According to the tests, PGO can help with achieving better performance in many cases, including compilers and interpreters. Since Koto targets fast compilation (according to the documentation), I think trying to optimize Koto's tooling performance with PGO can be a good idea.

I already did some benchmarks and want to share my results.

Test environment

• Fedora 39
• Linux kernel 6.8.6
• AMD Ryzen 9 5900x
• 48 Gib RAM
• SSD Samsung 980 Pro 2 Tib
• Compiler - Rustc 1.78-nightly
• koto version: the latest for now from the main branch on commit 1856abb432e44c95d40791815771bdc96e1ae34f
• Disabled Turbo boost

Benchmark

For benchmark purposes, I use these benchmarks. For PGO optimization I use cargo-pgo tool. Release bench result I got with cargo bench --workspace command. PGO training phase is done with cargo pgo bench -- --workspace, PGO optimization phase - with cargo pgo optimize bench -- --workspace.

For all commands taskset -c 0 is used for reducing the OS scheduler "noise". The measurements are done multiple times - the results are consistent across runs.

Results

I got the following results:

• Release: https://gist.github.com/zamazan4ik/90521ffd9f25f6f2ba4909bfc93ac302
• PGO optimized compared to Release: https://gist.github.com/zamazan4ik/77a46c1bd726212311358373fba2660e
• (just for reference) PGO instrumentation compared to Release: https://gist.github.com/zamazan4ik/c9361f6caf7cae5a38fa4d57b2ffa358

According to the results, PGO measurably improves Koto's performance at least in the cases above.

Further steps

I can suggest the following action points:

• Perform more PGO benchmarks with Koto. If it shows improvements - add a note to the documentation about possible improvements in Koto's performance with PGO.
• Providing an easier way (e.g. a build option) to build scripts with PGO can be helpful for the end-users and maintainers since they will be able to optimize Koto tooling according to their workloads.
• Optimize pre-built Koto binaries

Testing Post-Link Optimization techniques (like LLVM BOLT) would be interesting too (Clang and Rustc already use BOLT as an addition to PGO) but I recommend starting from the usual PGO.

Here are some examples of how PGO optimization is integrated into other projects:

• Rustc: a CI script for the multi-stage build
• GCC:
  • Official docs, section "Building with profile feedback"
  • A part in a "wonderful" configure script
• Clang: Docs
• Python:
  • CPython: README
  • Pyston: README
• Go: Bash script
• V8: Bazel flag
• ChakraCore: Scripts
• Chromium: Script
• Firefox: Docs
  • Thunderbird has PGO support too
• PHP - Makefile command and old Centminmod scripts
• MySQL: CMake script
• YugabyteDB: GitHub commit
• FoundationDB: Script
• Zstd: Makefile
• Foot: Scripts
• Windows Terminal: GitHub PR
• Pydantic-core: GitHub PR
• file.d: GitHub PR
• OceanBase: CMake flag

I would be happy to answer your questions about PGO and PLO.

[irh]

This is wonderful, thanks @zamazan4ik! I didn't know that the PGO results would be quite this impressive, I'll take a closer look during the week.