Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Benchmark improvements, including passing exceptions upwards #5289

Closed
wants to merge 34 commits into from

Conversation

trexfeathers
Copy link
Contributor

@trexfeathers trexfeathers commented Apr 28, 2023

🚀 Pull Request

This benchmark run was requested for the penultimate commit (before the What's New entry).

Clearer exceptions/errors

  • ab32be1 & b793971: ensure failures - i.e. breakages rather than just regressions - are elevated out of subprocess.run, causing the workflow to fail.
  • 28bb0d2: adds a step to the GHA workflow that will raise an issue when it fails, since we quite often miss this with the overnight runs (GitHub's default is to just email the last user to touch the workflow file).

CI usability improvements

  • 5cd49c2 & 78b34f3: switch PR benchmarking to use asv continuous, which uses an exit code of 1 where performance regressions are found - so we're not relying on developers to manually read the GitHub workflow.
  • 1ebf20e: improves how the benchmark runner prints things, specifically to make GHA easier to interpret.

Fixes


Consult Iris pull request check list

@trexfeathers trexfeathers added the benchmark_this Request that this pull request be benchmarked to check if it introduces performance shifts label Apr 28, 2023
@codecov
Copy link

codecov bot commented Apr 28, 2023

Codecov Report

Patch coverage has no change and project coverage change: +0.02 🎉

Comparison is base (a49b5ac) 89.32% compared to head (21da04c) 89.35%.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #5289      +/-   ##
==========================================
+ Coverage   89.32%   89.35%   +0.02%     
==========================================
  Files          89       89              
  Lines       22390    22417      +27     
  Branches     5374     5382       +8     
==========================================
+ Hits        20000    20030      +30     
+ Misses       1640     1639       -1     
+ Partials      750      748       -2     

see 10 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

@trexfeathers trexfeathers added benchmark_this Request that this pull request be benchmarked to check if it introduces performance shifts and removed benchmark_this Request that this pull request be benchmarked to check if it introduces performance shifts labels Apr 28, 2023
@trexfeathers trexfeathers added benchmark_this Request that this pull request be benchmarked to check if it introduces performance shifts and removed benchmark_this Request that this pull request be benchmarked to check if it introduces performance shifts labels Apr 28, 2023
@trexfeathers trexfeathers added benchmark_this Request that this pull request be benchmarked to check if it introduces performance shifts and removed benchmark_this Request that this pull request be benchmarked to check if it introduces performance shifts labels Apr 28, 2023
@trexfeathers trexfeathers added benchmark_this Request that this pull request be benchmarked to check if it introduces performance shifts and removed benchmark_this Request that this pull request be benchmarked to check if it introduces performance shifts labels Apr 28, 2023
@trexfeathers trexfeathers added benchmark_this Request that this pull request be benchmarked to check if it introduces performance shifts and removed benchmark_this Request that this pull request be benchmarked to check if it introduces performance shifts labels Apr 28, 2023
@trexfeathers trexfeathers added Type: Performance Type: Infrastructure Type: Testing and removed benchmark_this Request that this pull request be benchmarked to check if it introduces performance shifts labels Apr 28, 2023
@trexfeathers trexfeathers added benchmark_this Request that this pull request be benchmarked to check if it introduces performance shifts and removed benchmark_this Request that this pull request be benchmarked to check if it introduces performance shifts labels May 26, 2023
@trexfeathers trexfeathers added benchmark_this Request that this pull request be benchmarked to check if it introduces performance shifts and removed benchmark_this Request that this pull request be benchmarked to check if it introduces performance shifts labels May 30, 2023
@trexfeathers trexfeathers added benchmark_this Request that this pull request be benchmarked to check if it introduces performance shifts and removed benchmark_this Request that this pull request be benchmarked to check if it introduces performance shifts labels May 30, 2023
@trexfeathers trexfeathers added benchmark_this Request that this pull request be benchmarked to check if it introduces performance shifts and removed benchmark_this Request that this pull request be benchmarked to check if it introduces performance shifts labels May 30, 2023
@trexfeathers trexfeathers added benchmark_this Request that this pull request be benchmarked to check if it introduces performance shifts and removed benchmark_this Request that this pull request be benchmarked to check if it introduces performance shifts labels May 30, 2023
@trexfeathers trexfeathers added benchmark_this Request that this pull request be benchmarked to check if it introduces performance shifts and removed benchmark_this Request that this pull request be benchmarked to check if it introduces performance shifts labels May 30, 2023
@trexfeathers trexfeathers added benchmark_this Request that this pull request be benchmarked to check if it introduces performance shifts and removed benchmark_this Request that this pull request be benchmarked to check if it introduces performance shifts labels May 30, 2023
@trexfeathers trexfeathers added benchmark_this Request that this pull request be benchmarked to check if it introduces performance shifts and removed benchmark_this Request that this pull request be benchmarked to check if it introduces performance shifts labels May 30, 2023
@trexfeathers trexfeathers added benchmark_this Request that this pull request be benchmarked to check if it introduces performance shifts and removed benchmark_this Request that this pull request be benchmarked to check if it introduces performance shifts labels May 30, 2023
@trexfeathers trexfeathers deleted the bm_error_catch branch May 3, 2024 15:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
benchmark_this Request that this pull request be benchmarked to check if it introduces performance shifts Type: Infrastructure Type: Performance Type: Testing
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants