Evaluate using Profile-Guided Optimization (PGO) and Post-Link Optimization (PLO)

[zamazan4ik]

Hi!

Recently I checked Profile-Guided Optimization (PGO) and Post-Link Optimization (PLO) improvements on multiple projects. The results are available here. According to the tests, these optimizations can help with achieving better performance in many cases for many applications: networking apps like HAproxy and Envoy, many databases, parsing utilities like Vector, runtimes like V8, etc. Since this, I think trying to optimize workerd with PGO and PLO can be a good idea. According to the issue some people are already interested in further workerd optimization.

I can suggest the following action points:

• Perform PGO benchmarks on workerd. If it shows improvements - add a note to the documentation about possible improvements in workerd performance with PGO.
• Providing an easier way (e.g. a build option) to build scripts with PGO can be helpful for the end-users and maintainers since they will be able to optimize workerd according to their workloads.
• Optimize pre-built workerd binaries

Testing Post-Link Optimization techniques (like LLVM BOLT) would be interesting too (Clang and Rustc already use BOLT as an addition to PGO) but I recommend starting from the usual PGO.

Examples of how PGO optimization is integrated into other projects:

• Rustc: a CI script for the multi-stage build
• GCC:
  • Official docs, section "Building with profile feedback" (even AutoFDO build is supported)
  • A part in a "wonderful" configure script
• Clang: Docs
• Python:
  • CPython: README
  • Pyston: README
• Go: Bash script
• V8: Bazel flag
• ChakraCore: Scripts
• Chromium: Script
• Firefox: Docs
  • Thunderbird has PGO support too
• PHP - Makefile command and old Centminmod scripts
• MySQL: CMake script
• YugabyteDB: GitHub commit
• FoundationDB: Script
• Zstd: Makefile
• Foot: Scripts
• Windows Terminal: GitHub PR
• Pydantic-core: GitHub PR
• file.d: GitHub PR
• OceanBase: CMake flag

I have some examples of how PGO information looks in the documentation:

• ClickHouse: https://clickhouse.com/docs/en/operations/optimizing-performance/profile-guided-optimization
• Databend: https://databend.rs/doc/contributing/pgo
• Vector: https://vector.dev/docs/administration/tuning/pgo/
• Nebula: https://docs.nebula-graph.io/3.5.0/8.service-tuning/enable_autofdo_for_nebulagraph/
• GCC: Official docs, section "Building with profile feedback" (even AutoFDO build is supported)
• Clang:
  • https://llvm.org/docs/HowToBuildWithPGO.html
  • https://llvm.org/docs/AdvancedBuilds.html
• tsv-utils: https://github.com/eBay/tsv-utils/blob/master/docs/BuildingWithLTO.md

Regarding LLVM BOLT integration, I have the following examples:

• Rustc:
  • Rustc itself (GitHub PR)
  • LLVM in Rustc (Reddit)
• CPython: GitHub PR
• YDB: GitHub comment
• Clang:
  • Slides
  • Results on building Clang
  • Linaro results
  • on AMD 7950X3D
  • GitHub comment from LLVM
• LDC: GitHub comment
• HHVM, Proxygen and others: Facebook paper
• NodeJS: Blog
• Chromium: Blog
• MySQL, MongoDB, memcached, Verilator: Paper

I hope these ideas could be interesting for someone from the workerd community.

[kentonv]

Hi @zamazan4ik,

Thanks for the suggestion. We're aware that PGO is a thing, and we'll probably look into it eventually, but it isn't a priority in the near term.

Using PGO correctly is challenging, since we'd need to collect representative profiles to drive the optimizer. In an ideal world we'd have some sort of automated load test we could run in CI which emulates production traffic so we could generate a new profile before every release and optimize based on that. Of course, creating such a representative test suite is essentially impossible without capturing real applications and real data from the wild, which would be problematic. The more realistic approach would be to capture profiles directly from production. But that's a capability we'd have to build, it's by no means trivial. We'd need an automated system to refresh these profiles so they're always up to date, etc. And we'd need to make sure we understand all the implications of using profiles that were generated based on a slightly older version of the code than the one we're optimizing.

This is all a lot of work, and not something we feel is the best cost/benefit for our limited engineering hours right now. Maybe someday!