Implement basic support for PGO in rustbuild for rustc #80033

Mark-Simulacrum · 2020-12-14T18:52:18Z

This implements support for applying PGO to the rustc compilation step (not
standard library or any tooling, including rustdoc). Expanding PGO to more tools
is not terribly difficult but will involve more work and greater CI time
commitment.

For the same reason of avoiding greater time commitment, this currently avoids
implementing for platforms outside of x86_64-unknown-linux-gnu, though in
practice it should be quite simple to extend over time to more platforms. The
initial implementation is intentionally minimal here to avoid too much work
investment before we start seeing wins for a subset of Rust users.

The choice of workloads to profile here is somewhat arbitrary, but the general
rationale was to aim for a small set that largely avoided time regressions on
perf.rust-lang.org's full suite of crates. The set chosen is libcore, cargo (and
its dependencies), and a few ad-hoc stress tests from perf.rlo. The stress tests
are arguably the most controversial, but they benefit those cases (avoiding
regressions) and do not really remove wins from other benchmarks.

The primary next step after this PR lands is to implement support for PGO in
LLVM. It is unclear whether we can afford a full LLVM rebuild in CI, though, so
the approach taken there may need to be more staggered. rustc-only PGO seems
well affordable on linux at least, giving us up to 20% wall time wins on some
crates for 15 minutes of extra CI time (1 hour up from 45 minutes).

Mark-Simulacrum · 2020-12-14T23:04:22Z

So I guess we know that this empty file actively hurts - but at least PGO is having an effect :)

@michaelwoerister - can you share what workload you used? I pushed an update to use libcore, we'll see if that works better

@bors try @rust-timer queue

rust-timer · 2020-12-15T19:03:55Z

Finished benchmarking try commit (9b7eb5a3e3872fa2b9ab043432288b75aa917f4b): comparison url.

Benchmarking this pull request likely means that it is perf-sensitive, so we're automatically marking it as not fit for rolling up. Please note that if the perf results are neutral, you should likely undo the rollup=never given below by specifying rollup- to bors.

Importantly, though, if the results of this run are non-neutral do not roll this PR up -- it will mask other regressions or improvements in the roll up.

@bors rollup=never
@rustbot label: +S-waiting-on-review -S-waiting-on-perf

Mark-Simulacrum · 2020-12-17T17:15:38Z

@bors try

bors · 2020-12-17T17:15:52Z

⌛ Trying commit 1a6402e23a4f3a513af6f9a695bb9393b5f590e2 with merge 4e4f30e50d7c8ee770c0773a1a742cf5e08a7faa...

bors · 2020-12-17T18:17:29Z

☀️ Try build successful - checks-actions
Build commit: 4e4f30e50d7c8ee770c0773a1a742cf5e08a7faa (4e4f30e50d7c8ee770c0773a1a742cf5e08a7faa)

rust-timer · 2020-12-17T18:17:31Z

Queued 4e4f30e50d7c8ee770c0773a1a742cf5e08a7faa with parent caeb333, future comparison URL.

@rustbot label: +S-waiting-on-perf

rust-timer · 2020-12-17T22:03:27Z

Finished benchmarking try commit (4e4f30e50d7c8ee770c0773a1a742cf5e08a7faa): comparison url.

Benchmarking this pull request likely means that it is perf-sensitive, so we're automatically marking it as not fit for rolling up. Please note that if the perf results are neutral, you should likely undo the rollup=never given below by specifying rollup- to bors.

Importantly, though, if the results of this run are non-neutral do not roll this PR up -- it will mask other regressions or improvements in the roll up.

@bors rollup=never
@rustbot label: +S-waiting-on-review -S-waiting-on-perf

bjorn3 · 2020-12-17T22:11:58Z

Apart from a ~30% regression in ctfe-stress-4-doc there are only a few <2% regressions and overall a huge improvement of up 23% for task-clock. Instruction count does show some regressins, but that is expected as PGO optimized for cycle count rather than instruction count.

rust-log-analyzer · 2020-12-18T22:27:24Z

The job mingw-check failed! Check out the build log: (web) (plain)

Click to see the possible cause of the failure (guessed by this bot)

GITHUB_ENV=/home/runner/work/_temp/_runner_file_commands/set_env_ad1ce253-bbfc-4f30-8a1d-f0e6271cb30a
GITHUB_EVENT_NAME=pull_request
GITHUB_EVENT_PATH=/home/runner/work/_temp/_github_workflow/event.json
GITHUB_GRAPHQL_URL=https://api.github.com/graphql
GITHUB_HEAD_REF=pgo-rustc
GITHUB_JOB=pr
GITHUB_PATH=/home/runner/work/_temp/_runner_file_commands/add_path_ad1ce253-bbfc-4f30-8a1d-f0e6271cb30a
GITHUB_REF=refs/pull/80033/merge
GITHUB_REPOSITORY_OWNER=rust-lang
GITHUB_RETENTION_DAYS=90
GITHUB_RUN_ID=431440251
GITHUB_RUN_NUMBER=21643
---
Building wheels for collected packages: PyYAML
  Running setup.py bdist_wheel for PyYAML: started
  Running setup.py bdist_wheel for PyYAML: finished with status 'error'
  Failed building wheel for PyYAML
  Complete output from command /usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-rms5noqd/PyYAML/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/tmph57fhkgmpip-wheel- --python-tag cp36:
     or: -c --help [cmd1 cmd2 ...]
     or: -c --help-commands
     or: -c cmd --help
  
---
configure: rust.channel         := nightly
configure: rust.debug-assertions := True
configure: llvm.assertions      := True
configure: dist.missing-tools   := True
configure: build.configure-args := ['--enable-sccache', '--disable-manage-submodu ...
configure: writing `config.toml` in current directory
configure: 
configure: run `python /checkout/x.py --help`
configure: 
---
Checking which error codes lack tests...
Found 434 error codes
Found 0 error codes with no tests
Done!
tidy error: /checkout/src/ci/docker/host-x86_64/dist-x86_64-linux/pgo.sh:36: line longer than 100 chars
some tidy checks failed

command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/tidy" "/checkout" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "/checkout/obj/build"
expected success, got: exit code: 1

rust-log-analyzer · 2020-12-18T22:43:25Z

The job mingw-check failed! Check out the build log: (web) (plain)

Click to see the possible cause of the failure (guessed by this bot)

GITHUB_ENV=/home/runner/work/_temp/_runner_file_commands/set_env_875465d6-0ae7-4507-ab87-9500f45a9a14
GITHUB_EVENT_NAME=pull_request
GITHUB_EVENT_PATH=/home/runner/work/_temp/_github_workflow/event.json
GITHUB_GRAPHQL_URL=https://api.github.com/graphql
GITHUB_HEAD_REF=pgo-rustc
GITHUB_JOB=pr
GITHUB_PATH=/home/runner/work/_temp/_runner_file_commands/add_path_875465d6-0ae7-4507-ab87-9500f45a9a14
GITHUB_REF=refs/pull/80033/merge
GITHUB_REPOSITORY_OWNER=rust-lang
GITHUB_RETENTION_DAYS=90
GITHUB_RUN_ID=431466429
GITHUB_RUN_NUMBER=21647
---
Building wheels for collected packages: PyYAML
  Running setup.py bdist_wheel for PyYAML: started
  Running setup.py bdist_wheel for PyYAML: finished with status 'error'
  Failed building wheel for PyYAML
  Complete output from command /usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-r2ivmoy8/PyYAML/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/tmp28z5p9pfpip-wheel- --python-tag cp36:
     or: -c --help [cmd1 cmd2 ...]
     or: -c --help-commands
     or: -c cmd --help
  
---
configure: rust.channel         := nightly
configure: rust.debug-assertions := True
configure: llvm.assertions      := True
configure: dist.missing-tools   := True
configure: build.configure-args := ['--enable-sccache', '--disable-manage-submodu ...
configure: writing `config.toml` in current directory
configure: 
configure: run `python /checkout/x.py --help`
configure: 
---
Checking which error codes lack tests...
Found 434 error codes
Found 0 error codes with no tests
Done!
tidy error: /checkout/src/ci/docker/host-x86_64/dist-x86_64-linux/pgo.sh:14: line longer than 100 chars
some tidy checks failed

command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/tidy" "/checkout" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "/checkout/obj/build"
expected success, got: exit code: 1

This implements support for applying PGO to the rustc compilation step (not standard library or any tooling, including rustdoc). Expanding PGO to more tools is not terribly difficult but will involve more work and greater CI time commitment. For the same reason of avoiding greater time commitment, this currently avoids implementing for platforms outside of x86_64-unknown-linux-gnu, though in practice it should be quite simple to extend over time to more platforms. The initial implementation is intentionally minimal here to avoid too much work investment before we start seeing wins for a subset of Rust users. The choice of workloads to profile here is somewhat arbitrary, but the general rationale was to aim for a small set that largely avoided time regressions on perf.rust-lang.org's full suite of crates. The set chosen is libcore, cargo (and its dependencies), and a few ad-hoc stress tests from perf.rlo. The stress tests are arguably the most controversial, but they benefit those cases (avoiding regressions) and do not really remove wins from other benchmarks. The primary next step after this PR lands is to implement support for PGO in LLVM. It is unclear whether we can afford a full LLVM rebuild in CI, though, so the approach taken there may need to be more staggered. rustc-only PGO seems well affordable on linux at least, giving us up to 20% wall time wins on some crates for 15 minutes of extra CI time (1 hour up from 45 minutes).

Mark-Simulacrum · 2020-12-19T03:00:26Z

@bors try @rust-timer queue to just re-check that we're still seeing similar behavior but I am expecting to close this PR after that and open one with intent to merge (rather than just experiment). I think getting these wins out the door to users quickly makes sense and given that we can pull off the direct PGO in one CI builder for rustc at least I think we should, it doesn't make sense to do the more complicated scheme of using the previous build's artifacts if we can avoid it.

rust-timer · 2020-12-19T03:00:27Z

Awaiting bors try build completion.

bors · 2020-12-19T03:00:37Z

⌛ Trying commit 88f3a6e with merge 2ce5173d842eb52a0df4069b1cbb74e212510626...

bors · 2020-12-19T05:10:33Z

☀️ Try build successful - checks-actions
Build commit: 2ce5173d842eb52a0df4069b1cbb74e212510626 (2ce5173d842eb52a0df4069b1cbb74e212510626)

rust-timer · 2020-12-19T05:10:35Z

Queued 2ce5173d842eb52a0df4069b1cbb74e212510626 with parent 50a9097, future comparison URL.

@rustbot label: +S-waiting-on-perf

rust-timer · 2020-12-19T07:53:18Z

Finished benchmarking try commit (2ce5173d842eb52a0df4069b1cbb74e212510626): comparison url.

Benchmarking this pull request likely means that it is perf-sensitive, so we're automatically marking it as not fit for rolling up. Please note that if the perf results are neutral, you should likely undo the rollup=never given below by specifying rollup- to bors.

Importantly, though, if the results of this run are non-neutral do not roll this PR up -- it will mask other regressions or improvements in the roll up.

@bors rollup=never
@rustbot label: +S-waiting-on-review -S-waiting-on-perf

Mark-Simulacrum changed the title ~~Implement basic support for PGO in rustc~~ Implement basic support for PGO in rustbuild for rustc Dec 14, 2020