Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DO NOT LAND] Regress the token-stream-stress benchmark. #67248

Closed

Conversation

nnethercote
Copy link
Contributor

I have a suspicion that there is a bug in rustc-perf or rust-timer
causing the wrong revisions to be measured by CI. See #66405 and #67079
for more details.

This commit deliberately causes a massive regression to the
token-stream-stress benchmark. On my machine, the instruction count
goes from 313M to 6084M, an 1843.4% regression. I want to see if a CI
run replicates that.

cc @Mark-Simulacrum
r? @ghost

I have a suspicion that there is a bug in rustc-perf or rust-timer
causing the wrong revisions to be measured by CI. See rust-lang#66405 and rust-lang#67079
for more details.

This commit deliberately causes a massive regression to the
`token-stream-stress` benchmark. On my machine, the instruction count
goes from 313M to 6084M, an 1843.4% regression. I want to see if a CI
run replicates that.
@nnethercote
Copy link
Contributor Author

@bors try @rust-timer queue

@rust-timer
Copy link
Collaborator

Awaiting bors try build completion

@bors
Copy link
Collaborator

bors commented Dec 12, 2019

⌛ Trying commit 2d5843d with merge 5e1e02e...

bors added a commit that referenced this pull request Dec 12, 2019
[DO NOT LAND] Regress the `token-stream-stress` benchmark.

I have a suspicion that there is a bug in rustc-perf or rust-timer
causing the wrong revisions to be measured by CI. See #66405 and #67079
for more details.

This commit deliberately causes a massive regression to the
`token-stream-stress` benchmark. On my machine, the instruction count
goes from 313M to 6084M, an 1843.4% regression. I want to see if a CI
run replicates that.

cc @Mark-Simulacrum
r? @ghost
@nnethercote
Copy link
Contributor Author

Here are the top 10 check-clean entries from a local run:

token-stream-stress-check
        avg: 1843.4%    min: 1843.4%    max: 1843.4%
helloworld-check
        avg: -0.1%      min: -0.1%      max: -0.1%
issue-46449-check
        avg: -0.1%      min: -0.1%      max: -0.1%
unify-linearly-check
        avg: -0.0%      min: -0.0%      max: -0.0%
html5ever-check
        avg: 0.0%       min: 0.0%       max: 0.0%
deeply-nested-check
        avg: -0.0%      min: -0.0%      max: -0.0%
await-call-tree-check
        avg: -0.0%      min: -0.0%      max: -0.0%
coercions-check
        avg: -0.0%?     min: -0.0%?     max: -0.0%?
serde-check
        avg: 0.0%       min: 0.0%       max: 0.0%
syn-check
        avg: -0.0%      min: -0.0%      max: -0.0%

It shows a huge regression for token-stream-stress, and negligible other changes. Let's see if the CI run matches that.

@bors
Copy link
Collaborator

bors commented Dec 12, 2019

☀️ Try build successful - checks-azure
Build commit: 5e1e02e (5e1e02e73da308be8b0908637628027265f123a7)

@rust-timer
Copy link
Collaborator

Queued 5e1e02e with parent de0abf7, future comparison URL.

@rust-timer
Copy link
Collaborator

Finished benchmarking try commit 5e1e02e, comparison URL.

@nnethercote
Copy link
Contributor Author

CI results match my local results. Top 10 check-clean results:

token-stream-stress-check
        avg: 1695.9%    min: 1695.9%    max: 1695.9%
helloworld-check
        avg: 0.4%       min: 0.4%       max: 0.4%
issue-46449-check
        avg: 0.2%       min: 0.2%       max: 0.2%
await-call-tree-check
        avg: 0.2%       min: 0.2%       max: 0.2%
unify-linearly-check
        avg: 0.2%       min: 0.2%       max: 0.2%
deeply-nested-check
        avg: 0.2%       min: 0.2%       max: 0.2%
ripgrep-check
        avg: 0.1%       min: 0.1%       max: 0.1%
futures-check
        avg: 0.1%       min: 0.1%       max: 0.1%
regression-31157-check
        avg: 0.1%       min: 0.1%       max: 0.1%
regex-check
        avg: 0.0%       min: 0.0%       max: 0.0% 

A bit more variation among the barely-changing ones, but it disproves the theory that the wrong revisions are being tested.

@nnethercote nnethercote deleted the regress-token-stream-stress branch December 12, 2019 09:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants