Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix perf measurements for incremental test cases #42535

Closed
nikomatsakis opened this issue Jun 8, 2017 · 1 comment
Closed

fix perf measurements for incremental test cases #42535

nikomatsakis opened this issue Jun 8, 2017 · 1 comment
Labels
A-incr-comp Area: Incremental compilation E-mentor Call for participation: This issue has a mentor. Use #t-compiler/help on Zulip for discussion. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue.

Comments

@nikomatsakis
Copy link
Contributor

@arielb1 and @Mark-Simulacrum have done awesome work getting perf.rust-lang.org up and running again. Unfortunately, the current setup does not seem to be handling incremental data correctly (in particular, if you view the graphs, you'll see that all the measurements for the incremental tests -- which have names like foo@010-bar -- are zero). It'd be great to figure out what is going wrong! As a second goal, it'd be great to improve this setup and find better ways to visualize the data.

The current setup is described in the README on the rustc-benchmarks repository. The basic idea is that we invoke a series of make targets with funny names like all@010-bar, and each one progressively mutates the setup in some way and then does a compilation (for example, they might apply a patch). This allows us to measure the effect of each change on compilation time.

To explain the idea, here are some of the regex targets:

  • regex-0.1.80@010-baseline -- builds without incremental, to establish a baseline time
  • regex-0.1.80@020-incr-from-scratch -- builds for the first time with incremental; comparing to the previous result gives us an idea of the base overhead for incremental tracking
  • regex-0.1.80@030-compile_one -- applies a patch that modifies a particular method; comparing to the 010-baseline result gives us an idea of the gain from using incremental versus comparing by scratch every time, or this particular change.

Perhaps @Mark-Simulacrum or @arielb1 can give some tips as to what might be wrong and how to go about debugging the problem? I'm not really sure (for example) what code perf is running or how to make it run locally. I'm labeling this as "E-mentor" but volunteering them as mentors. =)

@nikomatsakis nikomatsakis added A-incr-comp Area: Incremental compilation E-mentor Call for participation: This issue has a mentor. Use #t-compiler/help on Zulip for discussion. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels Jun 8, 2017
@nikomatsakis
Copy link
Contributor Author

@Mark-Simulacrum figured out that performance measurements were working, but the GUI was not displaying them correctly. (And, indeed, we found a performance regression as a result!) Closing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-incr-comp Area: Incremental compilation E-mentor Call for participation: This issue has a mentor. Use #t-compiler/help on Zulip for discussion. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue.
Projects
None yet
Development

No branches or pull requests

1 participant