Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stabilize support for Profile-guided Optimization #61268

Merged
merged 2 commits into from
Jul 2, 2019

Conversation

michaelwoerister
Copy link
Member

This PR makes profile-guided optimization available via the -C profile-generate / -C profile-use pair of commandline flags and adds end-user documentation for the feature to the rustc book. The PR thus ticks the last two remaining checkboxes of the stabilization tracking issue.

From the tracking issue:

Profile-guided optimization (PGO) is a common optimization technique for ahead-of-time compilers. It works by collecting data about a program's typical execution (e.g. probability of branches taken, typical runtime values of variables, etc) and then uses this information during program optimization for things like inlining decisions, machine code layout, or indirect call promotion.

If you are curious about how this can be used, there is a rendered version of the documentation this PR adds available here.

r? @alexcrichton
cc @rust-lang/compiler

@michaelwoerister michaelwoerister added the T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. label May 28, 2019
@rust-highfive rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label May 28, 2019
@michaelwoerister
Copy link
Member Author

Dear @rust-lang/compiler, please add your check mark below.
@rfcbot fcp merge

@rfcbot
Copy link

rfcbot commented May 28, 2019

Team member @michaelwoerister has proposed to merge this. The next step is review by the rest of the tagged team members:

No concerns currently listed.

Once a majority of reviewers approve (and at most 2 approvals are outstanding), this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up!

See this document for info about what commands tagged team members can give me.

@rfcbot rfcbot added proposed-final-comment-period Proposed to merge/close by relevant subteam, see T-<team> label. Will enter FCP once signed off. disposition-merge This issue / PR is in PFCP or FCP with a disposition to merge it. labels May 28, 2019
src/doc/rustc/src/profile-guided-optimization.md Outdated Show resolved Hide resolved

- It is recommended to use *absolute paths* for the argument of
`-Cprofile-generate` and `-Cprofile-use`. Cargo can invoke `rustc` with
varying working directories, meaning that `.profraw` files might show up
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm this isn't quite right I think, don't .profraw files show up in the cwd by default? When compiling with profile-use, however, I think you definitely want to use absolute paths.

That being said absolute paths I think are still always your best bet.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

True, things would still work for the "generate" case. I'll re-word but keep the recommendation for using absolute paths.

src/doc/rustc/src/profile-guided-optimization.md Outdated Show resolved Hide resolved
@rust-highfive
Copy link
Collaborator

The job x86_64-gnu-llvm-6.0 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
travis_time:end:0dcd936a:start=1559056887674898160,finish=1559056888433409042,duration=758510882
$ git checkout -qf FETCH_HEAD
travis_fold:end:git.checkout

Encrypted environment variables have been removed for security reasons.
See https://docs.travis-ci.com/user/pull-requests/#pull-requests-and-security-restrictions
$ export SCCACHE_BUCKET=rust-lang-ci-sccache2
$ export SCCACHE_REGION=us-west-1
$ export GCP_CACHE_BUCKET=rust-lang-ci-cache
$ export AWS_ACCESS_KEY_ID=AKIA46X5W6CZEJZ6XT55
---
travis_time:start:test_codegen
Check compiletest suite=codegen mode=codegen (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
[01:20:41] 
[01:20:41] running 143 tests
[01:20:44] i..iii.....iii..iiii.....i......................i...i................i......i.........ii.i..i..i.ii. 100/143
[01:20:46] test result: ok. 113 passed; 0 failed; 30 ignored; 0 measured; 0 filtered out
[01:20:46] 
[01:20:46]  finished in 4.696
[01:20:46] travis_fold:end:test_codegen
---
travis_time:start:test_assembly
Check compiletest suite=assembly mode=assembly (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
[01:20:48] 
[01:20:48] running 9 tests
[01:20:48] iiiiiiiii
[01:20:48] 
[01:20:48]  finished in 0.158
[01:20:48] travis_fold:end:test_assembly

---
travis_time:start:test_debuginfo
Check compiletest suite=debuginfo mode=debuginfo-gdb+lldb (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
[01:21:04] 
[01:21:04] running 122 tests
[01:21:30] .iiiii...i.....i..i...i..i.i.i..i.ii..i.i.....i..i....i..........iiii..........i...ii...i.......ii.i 100/122
[01:21:35] .i.i......iii.i.....ii
[01:21:35] 
[01:21:35]  finished in 30.513
[01:21:35] travis_fold:end:test_debuginfo

---
[01:45:25] 
[01:45:25] stdout ----
[01:45:25] 
[01:45:25] running 1 test
[01:45:25] test /checkout/src/doc/rustc/src/profile-guided-optimization.md - Profile_Guided_Optimization::Usage (line 43) ... FAILED
[01:45:25] failures:
[01:45:25] 
[01:45:25] 
[01:45:25] ---- /checkout/src/doc/rustc/src/profile-guided-optimization.md - Profile_Guided_Optimization::Usage (line 43) stdout ----
[01:45:25] error: expected one of `!`, `.`, `::`, `;`, `?`, `{`, `}`, or an operator, found `component`
[01:45:25]  --> /checkout/src/doc/rustc/src/profile-guided-optimization.md:44:8
[01:45:25] 3 | rustup component add llvm-tools-preview
[01:45:25]   |        ^^^^^^^^^ expected one of 8 possible tokens here
[01:45:25] 
[01:45:25] error[E0425]: cannot find value `rustup` in this scope
---
[01:45:25] 
[01:45:25] error: aborting due to 2 previous errors
[01:45:25] 
[01:45:25] For more information about this error, try `rustc --explain E0425`.
[01:45:25] thread '/checkout/src/doc/rustc/src/profile-guided-optimization.md - Profile_Guided_Optimization::Usage (line 43)' panicked at 'couldn't compile the test', src/librustdoc/test.rs:320:13
[01:45:25] 
[01:45:25] 
[01:45:25] failures:
[01:45:25] failures:
[01:45:25]     /checkout/src/doc/rustc/src/profile-guided-optimization.md - Profile_Guided_Optimization::Usage (line 43)
[01:45:25] test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out
[01:45:25] 
[01:45:25] 
[01:45:25] stderr ----
[01:45:25] stderr ----
[01:45:25] 
[01:45:25] 
[01:45:25] 
[01:45:25] failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test
[01:45:25] Build completed unsuccessfully in 0:36:45
[01:45:25] make: *** [check] Error 1
[01:45:25] Makefile:48: recipe for target 'check' failed
The command "stamp sh -x -c "$RUN_SCRIPT"" exited with 2.
travis_time:start:354b0800
$ date && (curl -fs --head https://google.com | grep ^Date: | sed 's/Date: //g' || true)
Tue May 28 17:07:05 UTC 2019
---
travis_time:end:094dae8e:start=1559063227275306941,finish=1559063227280379075,duration=5072134
travis_fold:end:after_failure.3
travis_fold:start:after_failure.4
travis_time:start:00e441d8
$ ln -s . checkout && for CORE in obj/cores/core.*; do EXE=$(echo $CORE | sed 's|obj/cores/core\.[0-9]*\.!checkout!\(.*\)|\1|;y|!|/|'); if [ -f "$EXE" ]; then printf travis_fold":start:crashlog\n\033[31;1m%s\033[0m\n" "$CORE"; gdb --batch -q -c "$CORE" "$EXE" -iex 'set auto-load off' -iex 'dir src/' -iex 'set sysroot .' -ex bt -ex q; echo travis_fold":"end:crashlog; fi; done || true
travis_fold:end:after_failure.4
travis_fold:start:after_failure.5
travis_time:start:0d57b450
travis_time:start:0d57b450
$ cat ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers || true
cat: ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers: No such file or directory
travis_fold:end:after_failure.5
travis_fold:start:after_failure.6
travis_time:start:0985bd58
$ dmesg | grep -i kill

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@bors
Copy link
Contributor

bors commented May 28, 2019

☔ The latest upstream changes (presumably #61274) made this pull request unmergeable. Please resolve the merge conflicts.

@michaelwoerister
Copy link
Member Author

michaelwoerister commented May 29, 2019

@alexcrichton, I removed the -Ccodegen-units=1 flags from the examples. What do you think of defaulting to one codegen unit (with the option to opt out) when compiling with PGO? I think that would remove a common footgun. And, perhaps more importantly, it removes ThinLTO from the equation which can only make things better/more reliable.

@alexcrichton
Copy link
Member

I mentioned above that I wouldn't switch CGUs by default just yet, but I would also expect ThinLTO to take PGO data into account to know what to inline, but I could very well be wrong about that!

@michaelwoerister
Copy link
Member Author

I mentioned above that I wouldn't switch CGUs by default just yet, but I would also expect ThinLTO to take PGO data into account to know what to inline, but I could very well be wrong about that!

OK, let's keep the two settings independent then. ThinLTO should indeed take PGO data into account since PGO adds regular cold and inlinehint annotation to functions.

This is mostly my general slight mistrust in (Thin)LTO speaking. It's just one more step where things can go wrong (and have done so in the past). I personally would always default to one CGU for any kind of production code. But that's a topic for another discussion :)

@bors
Copy link
Contributor

bors commented May 30, 2019

☔ The latest upstream changes (presumably #61005) made this pull request unmergeable. Please resolve the merge conflicts.

@rfcbot rfcbot added final-comment-period In the final comment period and will be merged soon unless new substantive objections are raised. and removed proposed-final-comment-period Proposed to merge/close by relevant subteam, see T-<team> label. Will enter FCP once signed off. labels Jun 4, 2019
@rfcbot
Copy link

rfcbot commented Jun 4, 2019

🔔 This is now entering its final comment period, as per the review above. 🔔

@Centril Centril added S-waiting-on-team Status: Awaiting decision from the relevant subteam (see the T-<team> label). S-waiting-on-fcp Status: PR is in FCP and is awaiting for FCP to complete. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. S-waiting-on-team Status: Awaiting decision from the relevant subteam (see the T-<team> label). labels Jun 10, 2019
@rfcbot rfcbot added finished-final-comment-period The final comment period is finished for this PR / Issue. and removed final-comment-period In the final comment period and will be merged soon unless new substantive objections are raised. labels Jun 14, 2019
@rfcbot
Copy link

rfcbot commented Jun 14, 2019

The final comment period, with a disposition to merge, as per the review above, is now complete.

As the automated representative of the governance process, I would like to thank the author for their work and everyone else who contributed.

The RFC will be merged soon.

@bors
Copy link
Contributor

bors commented Jun 16, 2019

☔ The latest upstream changes (presumably #61739) made this pull request unmergeable. Please resolve the merge conflicts.

@bors bors added the S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. label Jun 16, 2019
@michaelwoerister
Copy link
Member Author

@bors r=alexcrichton (right?)

@bors
Copy link
Contributor

bors commented Jun 21, 2019

📌 Commit b7fe2ca has been approved by alexcrichton

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Jun 21, 2019
@jonas-schievink jonas-schievink removed the S-waiting-on-fcp Status: PR is in FCP and is awaiting for FCP to complete. label Jun 22, 2019
@mati865
Copy link
Contributor

mati865 commented Jul 2, 2019

This PR doesn't appear on https://buildbot2.rust-lang.org/homu/queue/rust
@michaelwoerister you will have to re-queue it.

@alexcrichton alexcrichton reopened this Jul 2, 2019
@alexcrichton
Copy link
Member

@bors: r+

@bors
Copy link
Contributor

bors commented Jul 2, 2019

💡 This pull request was already approved, no need to approve it again.

@bors
Copy link
Contributor

bors commented Jul 2, 2019

📌 Commit b7fe2ca has been approved by alexcrichton

@bors
Copy link
Contributor

bors commented Jul 2, 2019

⌛ Testing commit b7fe2ca with merge 30fcafb37fb86899f2be848f1507848f4049c8eb...

@Centril
Copy link
Contributor

Centril commented Jul 2, 2019

@bors retry Yielding to stable emergency backport (#62311).

@Centril
Copy link
Contributor

Centril commented Jul 2, 2019

@bors p=50

@Centril Centril added this to the 1.37 milestone Jul 2, 2019
@bors
Copy link
Contributor

bors commented Jul 2, 2019

⌛ Testing commit b7fe2ca with merge 0beb2ba...

bors added a commit that referenced this pull request Jul 2, 2019
Stabilize support for Profile-guided Optimization

This PR makes profile-guided optimization available via the `-C profile-generate` / `-C profile-use` pair of commandline flags and adds end-user documentation for the feature to the [rustc book](https://doc.rust-lang.org/rustc/). The PR thus ticks the last two remaining checkboxes of the [stabilization tracking issue](#59913).

From the tracking issue:
> Profile-guided optimization (PGO) is a common optimization technique for ahead-of-time compilers. It works by collecting data about a program's typical execution (e.g. probability of branches taken, typical runtime values of variables, etc) and then uses this information during program optimization for things like inlining decisions, machine code layout, or indirect call promotion.

If you are curious about how this can be used, there is a rendered version of the documentation this PR adds available [here](
https://github.com/michaelwoerister/rust/blob/stabilize-pgo/src/doc/rustc/src/profile-guided-optimization.md).

r? @alexcrichton
cc @rust-lang/compiler
@Centril Centril added the relnotes Marks issues that should be documented in the release notes of the next release. label Jul 2, 2019
@bors
Copy link
Contributor

bors commented Jul 2, 2019

☀️ Test successful - checks-azure, checks-travis, status-appveyor
Approved by: alexcrichton
Pushing 0beb2ba to master...

@bors bors added the merged-by-bors This PR was explicitly merged by bors. label Jul 2, 2019
@bors bors merged commit b7fe2ca into rust-lang:master Jul 2, 2019
netbsd-srcmastr pushed a commit to NetBSD/pkgsrc that referenced this pull request Aug 29, 2019
Pkgsrc changes:
 * Add a patch to llvm to deal with const dli_saddr.
 * Adapt two other patches.
 * Cross-build currently fails, so i386, powerpc and sparc64 bootstrap
   kits for 1.37.0 are built natively.  Missing aarch64 hardware, so that's
   not available yet.
 * Bump bootstrap requirements to 1.36.0 except for armv7-unknown-netbsd-eabihf
   which I've not managed to cross-build.

Upstream changes:

Version 1.37.0 (2019-08-15)
==========================

Language
--------
- `#[must_use]` will now warn if the type is contained in a [tuple][61100],
  [`Box`][62228], or an [array][62235] and unused.
- [You can now use the `cfg` and `cfg_attr` attributes on
  generic parameters.][61547]
- [You can now use enum variants through type alias.][61682] e.g. You can
  write the following:
  ```rust
  type MyOption = Option<u8>;

  fn increment_or_zero(x: MyOption) -> u8 {
      match x {
          MyOption::Some(y) => y + 1,
          MyOption::None => 0,
      }
  }
  ```
- [You can now use `_` as an identifier for consts.][61347] e.g. You can write
  `const _: u32 = 5;`.
- [You can now use `#[repr(align(X)]` on enums.][61229]
- [The  `?`/_"Kleene"_ macro operator is now available in the
  2015 edition.][60932]

Compiler
--------
- [You can now enable Profile-Guided Optimization with the `-C profile-generate`
  and `-C profile-use` flags.][61268] For more information on how to use profile
  guided optimization, please refer to the [rustc book][rustc-book-pgo].
- [The `rust-lldb` wrapper script should now work again.][61827]

Libraries
---------
- [`mem::MaybeUninit<T>` is now ABI-compatible with `T`.][61802]

Stabilized APIs
---------------
- [`BufReader::buffer`]
- [`BufWriter::buffer`]
- [`Cell::from_mut`]
- [`Cell<[T]>::as_slice_of_cells`][`Cell<slice>::as_slice_of_cells`]
- [`DoubleEndedIterator::nth_back`]
- [`Option::xor`]
- [`Wrapping::reverse_bits`]
- [`i128::reverse_bits`]
- [`i16::reverse_bits`]
- [`i32::reverse_bits`]
- [`i64::reverse_bits`]
- [`i8::reverse_bits`]
- [`isize::reverse_bits`]
- [`slice::copy_within`]
- [`u128::reverse_bits`]
- [`u16::reverse_bits`]
- [`u32::reverse_bits`]
- [`u64::reverse_bits`]
- [`u8::reverse_bits`]
- [`usize::reverse_bits`]

Cargo
-----
- [`Cargo.lock` files are now included by default when publishing executable crates
  with executables.][cargo/7026]
- [You can now specify `default-run="foo"` in `[package]` to specify the
  default executable to use for `cargo run`.][cargo/7056]

Misc
----

Compatibility Notes
-------------------
- [Using `...` for inclusive range patterns will now warn by default.][61342]
  Please transition your code to using the `..=` syntax for inclusive
  ranges instead.
- [Using a trait object without the `dyn` will now warn by default.][61203]
  Please transition your code to use `dyn Trait` for trait objects instead.

[62228]: rust-lang/rust#62228
[62235]: rust-lang/rust#62235
[61802]: rust-lang/rust#61802
[61827]: rust-lang/rust#61827
[61547]: rust-lang/rust#61547
[61682]: rust-lang/rust#61682
[61268]: rust-lang/rust#61268
[61342]: rust-lang/rust#61342
[61347]: rust-lang/rust#61347
[61100]: rust-lang/rust#61100
[61203]: rust-lang/rust#61203
[61229]: rust-lang/rust#61229
[60932]: rust-lang/rust#60932
[cargo/7026]: rust-lang/cargo#7026
[cargo/7056]: rust-lang/cargo#7056
[`BufReader::buffer`]: https://doc.rust-lang.org/std/io/struct.BufReader.html#method.buffer
[`BufWriter::buffer`]: https://doc.rust-lang.org/std/io/struct.BufWriter.html#method.buffer
[`Cell::from_mut`]: https://doc.rust-lang.org/std/cell/struct.Cell.html#method.from_mut
[`Cell<slice>::as_slice_of_cells`]: https://doc.rust-lang.org/std/cell/struct.Cell.html#method.as_slice_of_cells
[`DoubleEndedIterator::nth_back`]: https://doc.rust-lang.org/std/iter/trait.DoubleEndedIterator.html#method.nth_back
[`Option::xor`]: https://doc.rust-lang.org/std/option/enum.Option.html#method.xor
[`RefCell::try_borrow_unguarded`]: https://doc.rust-lang.org/std/cell/struct.RefCell.html#method.try_borrow_unguarded
[`Wrapping::reverse_bits`]: https://doc.rust-lang.org/std/num/struct.Wrapping.html#method.reverse_bits
[`i128::reverse_bits`]: https://doc.rust-lang.org/std/primitive.i128.html#method.reverse_bits
[`i16::reverse_bits`]: https://doc.rust-lang.org/std/primitive.i16.html#method.reverse_bits
[`i32::reverse_bits`]: https://doc.rust-lang.org/std/primitive.i32.html#method.reverse_bits
[`i64::reverse_bits`]: https://doc.rust-lang.org/std/primitive.i64.html#method.reverse_bits
[`i8::reverse_bits`]: https://doc.rust-lang.org/std/primitive.i8.html#method.reverse_bits
[`isize::reverse_bits`]: https://doc.rust-lang.org/std/primitive.isize.html#method.reverse_bits
[`slice::copy_within`]: https://doc.rust-lang.org/std/primitive.slice.html#method.copy_within
[`u128::reverse_bits`]: https://doc.rust-lang.org/std/primitive.u128.html#method.reverse_bits
[`u16::reverse_bits`]: https://doc.rust-lang.org/std/primitive.u16.html#method.reverse_bits
[`u32::reverse_bits`]: https://doc.rust-lang.org/std/primitive.u32.html#method.reverse_bits
[`u64::reverse_bits`]: https://doc.rust-lang.org/std/primitive.u64.html#method.reverse_bits
[`u8::reverse_bits`]: https://doc.rust-lang.org/std/primitive.u8.html#method.reverse_bits
[`usize::reverse_bits`]: https://doc.rust-lang.org/std/primitive.usize.html#method.reverse_bits
[rustc-book-pgo]: https://doc.rust-lang.org/rustc/profile-guided-optimization.html
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
disposition-merge This issue / PR is in PFCP or FCP with a disposition to merge it. finished-final-comment-period The final comment period is finished for this PR / Issue. merged-by-bors This PR was explicitly merged by bors. relnotes Marks issues that should be documented in the release notes of the next release. S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants