Profile-Guided Optimization (PGO) benchmark results

[zamazan4ik]

Hi!

Recently I tested Profile-Guided Optimization (PGO) compiler optimization on different projects in different software domains - all the results are available in https://github.com/zamazan4ik/awesome-pgo . Since PGO shows measurable improvements in many cases, I decided to perform PGO benchmarks on prost. Here are my results.

Test environment

• Fedora 39
• Linux kernel 6.8.6
• AMD Ryzen 9 5900x
• 48 Gib RAM
• SSD Samsung 980 Pro 2 Tib
• Compiler - Rustc 1.77.2
• Prost version: the latest for now from the master branch on commit 1fc736ef7840ad986e7f6e185c242489e97f3400
• Disabled Turbo boost

Benchmark

For benchmark purposes, I use built-in into the project benchmarks. For PGO optimization I use cargo-pgo tool. Release bench result I got with cargo bench --workspace command. PGO training phase is done with cargo pgo bench -- --workspace, PGO optimization phase - with cargo pgo optimize bench -- --workspace.

All measurements are done on the same machine, with the same background "noise" (as much as I can guarantee).

Results

I got the following results:

• Release: https://gist.github.com/zamazan4ik/1783b04847288cadb251abef3b4e43ad
• PGO optimized compared to Release: https://gist.github.com/zamazan4ik/654fe668dd5ad191359c6c4e7f7cd737
• (just for reference) PGO instrumentation compared to Release: https://gist.github.com/zamazan4ik/32b33157c2a3d2c855530be38beb45bf

According to the results, PGO measurably improves prost performance in many cases.

Further steps

I can suggest the following action points:

• Perform more PGO benchmarks with other datasets. If it shows improvements - add a note to the documentation (README file?) about possible improvements in prost's performance with PGO.

Testing Post-Link Optimization techniques (like LLVM BOLT) would be interesting too (Clang and Rustc already use BOLT as an addition to PGO) but I recommend starting from the usual PGO.

I would be happy to answer your questions about PGO.

[caspermeijn]

Do I understand correctly that some benchmarks are 500% faster with these optimizations?

This is only useful when doing PGO on the actual application, right? We can't provide these optimizations with the library, or can we?

I am not sure about the current state of our benchmark. I believe they can be improved a lot. Are you willing to do some work on our benchmarks?

[zamazan4ik]

Do I understand correctly that some benchmarks are 500% faster with these optimizations?

According to https://gist.github.com/zamazan4ik/654fe668dd5ad191359c6c4e7f7cd737 I don't see 500% improvement. The biggest improvement I see is ~40% . However, it's still a big improvement, IMO.

This is only useful when doing PGO on the actual application, right?

In general case, yes. However, they also can be useful in the case of prebuilt libraries (not the case for the regular Rust libraries).

We can't provide these optimizations with the library, or can we?

There are several ways. One way - prebuilt library with PGO-enabled optimization (very exotic case for the Rust ecosystem). Another way - try to analyze the generated assembly before and after PGO. Then, change the source code correspondingly.

Are you willing to do some work on our benchmarks?

I don't have enough expertise in the library so I don't think that can contribute valuable and useful changes to the benchmark suite.

[caspermeijn]

This is only useful when doing PGO on the actual application, right?

In general case, yes. However, they also can be useful in the case of prebuilt libraries (not the case for the regular Rust libraries).

We don't ship prebuilt libraries for prost, so that will not work for us.

We can't provide these optimizations with the library, or can we?

There are several ways. One way - prebuilt library with PGO-enabled optimization (very exotic case for the Rust ecosystem). Another way - try to analyze the generated assembly before and after PGO. Then, change the source code correspondingly.

Analyzing assembly doesn't sound fun to me. But if you are willing to do so, I am interested in what PRs that could result in.