Profile-Guided Optimization (PGO) benchmark report

[zamazan4ik]

Hi!

As I have done many times before, I decided to test the Profile-Guided Optimization (PGO) technique to optimize the library performance. For reference, results for other projects are available at https://github.com/zamazan4ik/awesome-pgo . Since PGO helped a lot for many libraries, I decided to apply it to gimli to see if the performance win (or loss) can be achieved. Here are my benchmark results.

This information can be interesting for anyone who wants to achieve more performance with the library in their use cases.

Test environment

• Fedora 40
• Linux kernel 6.11.3
• AMD Ryzen 9 5900x
• 48 Gib RAM
• SSD Samsung 980 Pro 2 Tib
• Compiler - Rustc 1.81.0
• gimli version: master branch on commit 7e9d923a98c5eeed4d7a8b8cb32475d1ce16ced2
• Disabled Turbo boost

Benchmark

For benchmark purposes, I use built-in into the project benchmarks. For PGO optimization I use cargo-pgo tool. Release bench results I got with taskset -c 0 cargo +nightly bench command. The PGO training phase is done with taskset -c 0 cargo +nightly pgo bench, PGO optimization phase - with taskset -c 0 cargo +nightly pgo optimize bench.

taskset -c 0 is used for reducing the OS scheduler's influence on the results. All measurements are done on the same machine, with the same background "noise" (as much as I can guarantee).

Results

I got the following results.

Release:

test bench_evaluating_debug_info_expressions                   ... bench:     427,432.10 ns/iter (+/- 1,603.88)
test bench_evaluating_debug_loc_expressions                    ... bench:     567,973.50 ns/iter (+/- 3,968.06)
test bench_executing_line_number_programs                      ... bench:     177,523.46 ns/iter (+/- 4,116.21)
test bench_parsing_debug_abbrev                                ... bench:      11,275.47 ns/iter (+/- 37.54)
test bench_parsing_debug_aranges                               ... bench:       3,702.12 ns/iter (+/- 16.72)
test bench_parsing_debug_info                                  ... bench:   1,684,371.30 ns/iter (+/- 5,754.22)
test bench_parsing_debug_info_expressions                      ... bench:      66,512.52 ns/iter (+/- 221.12)
test bench_parsing_debug_info_raw                              ... bench:   1,040,226.50 ns/iter (+/- 10,144.59)
test bench_parsing_debug_info_tree                             ... bench:   1,673,263.30 ns/iter (+/- 2,402.60)
test bench_parsing_debug_info_with_endian_rc_slice             ... bench:   1,804,673.30 ns/iter (+/- 4,026.97)
test bench_parsing_debug_loc                                   ... bench:     191,329.24 ns/iter (+/- 2,170.18)
test bench_parsing_debug_loc_expressions                       ... bench:      68,838.54 ns/iter (+/- 342.78)
test bench_parsing_debug_pubnames                              ... bench:      53,378.72 ns/iter (+/- 217.06)
test bench_parsing_debug_pubtypes                              ... bench:      22,203.66 ns/iter (+/- 2,960.54)
test bench_parsing_debug_ranges                                ... bench:      50,701.59 ns/iter (+/- 205.55)
test bench_parsing_line_number_program_opcodes                 ... bench:     180,742.56 ns/iter (+/- 1,424.95)
test cfi::eval_longest_fde_instructions_new_ctx_everytime      ... bench:         268.54 ns/iter (+/- 2.65)
test cfi::eval_longest_fde_instructions_same_ctx               ... bench:         221.56 ns/iter (+/- 1.85)
test cfi::iterate_entries_and_do_not_parse_any_fde             ... bench:      44,695.92 ns/iter (+/- 384.10)
test cfi::iterate_entries_and_parse_every_fde                  ... bench:     269,820.70 ns/iter (+/- 2,314.91)
test cfi::iterate_entries_and_parse_every_fde_and_instructions ... bench:     883,491.50 ns/iter (+/- 22,425.72)
test cfi::iterate_entries_evaluate_every_fde                   ... bench:     815,062.30 ns/iter (+/- 3,201.45)
test cfi::parse_longest_fde_instructions                       ... bench:         218.80 ns/iter (+/- 0.40)

PGO optimized:

test bench_evaluating_debug_info_expressions                   ... bench:     351,174.44 ns/iter (+/- 1,517.56)
test bench_evaluating_debug_loc_expressions                    ... bench:     430,575.82 ns/iter (+/- 841.13)
test bench_executing_line_number_programs                      ... bench:     144,242.90 ns/iter (+/- 600.31)
test bench_parsing_debug_abbrev                                ... bench:      12,211.98 ns/iter (+/- 69.88)
test bench_parsing_debug_aranges                               ... bench:       3,171.28 ns/iter (+/- 7.23)
test bench_parsing_debug_info                                  ... bench:   1,134,422.00 ns/iter (+/- 9,102.07)
test bench_parsing_debug_info_expressions                      ... bench:      38,630.04 ns/iter (+/- 128.55)
test bench_parsing_debug_info_raw                              ... bench:     940,866.75 ns/iter (+/- 6,882.31)
test bench_parsing_debug_info_tree                             ... bench:   1,259,128.15 ns/iter (+/- 10,731.43)
test bench_parsing_debug_info_with_endian_rc_slice             ... bench:   1,259,793.70 ns/iter (+/- 7,193.95)
test bench_parsing_debug_loc                                   ... bench:      82,677.42 ns/iter (+/- 150.93)
test bench_parsing_debug_loc_expressions                       ... bench:      43,838.92 ns/iter (+/- 169.23)
test bench_parsing_debug_pubnames                              ... bench:     125,822.83 ns/iter (+/- 169.41)
test bench_parsing_debug_pubtypes                              ... bench:      42,468.04 ns/iter (+/- 123.37)
test bench_parsing_debug_ranges                                ... bench:      51,323.69 ns/iter (+/- 221.35)
test bench_parsing_line_number_program_opcodes                 ... bench:     275,697.45 ns/iter (+/- 698.49)
test cfi::eval_longest_fde_instructions_new_ctx_everytime      ... bench:         160.92 ns/iter (+/- 2.22)
test cfi::eval_longest_fde_instructions_same_ctx               ... bench:         121.82 ns/iter (+/- 0.55)
test cfi::iterate_entries_and_do_not_parse_any_fde             ... bench:      20,551.87 ns/iter (+/- 42.44)
test cfi::iterate_entries_and_parse_every_fde                  ... bench:     209,861.77 ns/iter (+/- 1,664.25)
test cfi::iterate_entries_and_parse_every_fde_and_instructions ... bench:     300,605.73 ns/iter (+/- 1,009.40)
test cfi::iterate_entries_evaluate_every_fde                   ... bench:     513,710.60 ns/iter (+/- 3,727.22)
test cfi::parse_longest_fde_instructions                       ... bench:          47.86 ns/iter (+/- 0.05)

(just for reference) PGO instrumented:

test bench_evaluating_debug_info_expressions                   ... bench:     499,808.20 ns/iter (+/- 3,149.01)
test bench_evaluating_debug_loc_expressions                    ... bench:     622,441.80 ns/iter (+/- 3,318.93)
test bench_executing_line_number_programs                      ... bench:     168,378.92 ns/iter (+/- 2,109.33)
test bench_parsing_debug_abbrev                                ... bench:      15,448.84 ns/iter (+/- 71.79)
test bench_parsing_debug_aranges                               ... bench:       8,509.96 ns/iter (+/- 20.81)
test bench_parsing_debug_info                                  ... bench:   2,237,421.55 ns/iter (+/- 9,375.71)
test bench_parsing_debug_info_expressions                      ... bench:      87,804.87 ns/iter (+/- 243.56)
test bench_parsing_debug_info_raw                              ... bench:   1,496,320.00 ns/iter (+/- 24,766.99)
test bench_parsing_debug_info_tree                             ... bench:   2,235,192.30 ns/iter (+/- 4,157.22)
test bench_parsing_debug_info_with_endian_rc_slice             ... bench:   1,567,395.50 ns/iter (+/- 4,141.62)
test bench_parsing_debug_loc                                   ... bench:     301,707.91 ns/iter (+/- 936.29)
test bench_parsing_debug_loc_expressions                       ... bench:     124,260.93 ns/iter (+/- 475.22)
test bench_parsing_debug_pubnames                              ... bench:     136,493.26 ns/iter (+/- 173.66)
test bench_parsing_debug_pubtypes                              ... bench:      50,752.37 ns/iter (+/- 200.23)
test bench_parsing_debug_ranges                                ... bench:     114,024.10 ns/iter (+/- 380.75)
test bench_parsing_line_number_program_opcodes                 ... bench:     384,739.18 ns/iter (+/- 730.68)
test cfi::eval_longest_fde_instructions_new_ctx_everytime      ... bench:         313.50 ns/iter (+/- 3.12)
test cfi::eval_longest_fde_instructions_same_ctx               ... bench:         266.56 ns/iter (+/- 1.89)
test cfi::iterate_entries_and_do_not_parse_any_fde             ... bench:      64,691.00 ns/iter (+/- 832.04)
test cfi::iterate_entries_and_parse_every_fde                  ... bench:     383,923.60 ns/iter (+/- 2,736.85)
test cfi::iterate_entries_and_parse_every_fde_and_instructions ... bench:   1,087,289.20 ns/iter (+/- 7,695.76)
test cfi::iterate_entries_evaluate_every_fde                   ... bench:   1,081,502.80 ns/iter (+/- 11,114.39)
test cfi::parse_longest_fde_instructions                       ... bench:         237.96 ns/iter (+/- 2.28)

According to the results, we see improvements in the library's performance.

Further steps

At the very least, the library's users can find this performance report and decide to enable PGO for their applications if they care about gimli performance in their workloads. Maybe a small note somewhere in the documentation (the README file?) will be enough to raise awareness about this work.

Also, Post-Link Optimization (PLO) can be tested after PGO. It can be done by applying tools like LLVM BOLT to applications with apps that use gimli. However, it's a much less mature optimization technique compared to PGO.

Thank you.

[philipc]

Interesting, that's some fairly large differences. It'd be nice to test on a real workload instead of micro benchmarks (e.g. https://github.com/gimli-rs/addr2line/blob/master/scripts/benchmark-addr2line.sh)

cargo-benchcmp comparison:

 name                                                       release ns/iter  pgo ns/iter  diff ns/iter   diff %  speedup 
 bench_evaluating_debug_info_expressions                    427,432          351,174           -76,258  -17.84%   x 1.22 
 bench_evaluating_debug_loc_expressions                     567,973          430,575          -137,398  -24.19%   x 1.32 
 bench_executing_line_number_programs                       177,523          144,242           -33,281  -18.75%   x 1.23 
 bench_parsing_debug_abbrev                                 11,275           12,211                936    8.30%   x 0.92 
 bench_parsing_debug_aranges                                3,702            3,171                -531  -14.34%   x 1.17 
 bench_parsing_debug_info                                   1,684,371        1,134,422        -549,949  -32.65%   x 1.48 
 bench_parsing_debug_info_expressions                       66,512           38,630            -27,882  -41.92%   x 1.72 
 bench_parsing_debug_info_raw                               1,040,226        940,866           -99,360   -9.55%   x 1.11 
 bench_parsing_debug_info_tree                              1,673,263        1,259,128        -414,135  -24.75%   x 1.33 
 bench_parsing_debug_info_with_endian_rc_slice              1,804,673        1,259,793        -544,880  -30.19%   x 1.43 
 bench_parsing_debug_loc                                    191,329          82,677           -108,652  -56.79%   x 2.31 
 bench_parsing_debug_loc_expressions                        68,838           43,838            -25,000  -36.32%   x 1.57 
 bench_parsing_debug_pubnames                               53,378           125,822            72,444  135.72%   x 0.42 
 bench_parsing_debug_pubtypes                               22,203           42,468             20,265   91.27%   x 0.52 
 bench_parsing_debug_ranges                                 50,701           51,323                622    1.23%   x 0.99 
 bench_parsing_line_number_program_opcodes                  180,742          275,697            94,955   52.54%   x 0.66 
 cfi::eval_longest_fde_instructions_new_ctx_everytime       268              160                  -108  -40.30%   x 1.68 
 cfi::eval_longest_fde_instructions_same_ctx                221              121                  -100  -45.25%   x 1.83 
 cfi::iterate_entries_and_do_not_parse_any_fde              44,695           20,551            -24,144  -54.02%   x 2.17 
 cfi::iterate_entries_and_parse_every_fde                   269,820          209,861           -59,959  -22.22%   x 1.29 
 cfi::iterate_entries_and_parse_every_fde_and_instructions  883,491          300,605          -582,886  -65.98%   x 2.94 
 cfi::iterate_entries_evaluate_every_fde                    815,062          513,710          -301,352  -36.97%   x 1.59 
 cfi::parse_longest_fde_instructions                        218              47                   -171  -78.44%   x 4.64