Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Track performance changes in CI #6149

Open
adriencaccia opened this issue Jul 29, 2024 · 4 comments · May be fixed by #6150
Open

Track performance changes in CI #6149

adriencaccia opened this issue Jul 29, 2024 · 4 comments · May be fixed by #6150
Labels
enhancement Any new improvement worthy of a entry in the changelog

Comments

@adriencaccia
Copy link

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

Add a system to track performance over time, using the repository's existing benchmarks. This will help ensure that no unwanted performance regressions are merged, and give more visibility on performance improvement work like #6107

Describe the solution you'd like

Use https://codspeed.io/ with a new workflow that runs on every PR and the master branch.

Additional context

I already installed CodSpeed on a fork of arrow-rs, and you can see what it looks like on a PR that improves performance (based on #6145):
CodSpeedHQ#51
https://codspeed.io/CodSpeedHQ/arrow-rs/branches/fork-6145-samuelcolvin/improve-like-regex

Thanks @samuelcolvin for validating the need

@adriencaccia adriencaccia added the enhancement Any new improvement worthy of a entry in the changelog label Jul 29, 2024
@adriencaccia adriencaccia linked a pull request Jul 29, 2024 that will close this issue
@Xuanwo
Copy link
Member

Xuanwo commented Jul 29, 2024

Hi, thanks for bringing this up! Many people are concerned about performance changes in arrow-rs.

I have a question about codspeed: Where do these performance tests take place? Are they run on GitHub Actions or on a dedicated environment? How can we ensure stable performance metrics and avoid false positives?

@adriencaccia
Copy link
Author

I have a question about codspeed: Where do these performance tests take place? Are they run on GitHub Actions or on a dedicated environment?

They are run on GitHub Actions default runners, using https://github.com/CodSpeedHQ/action/.

How can we ensure stable performance metrics and avoid false positives?

Thanks to the action, the benchmarks are run on a simulated CPU and thus the measurements are stable (you can read more about it in our docs).

On the existing benchmarks, you can see that the results are quite consistent:

The links above is the CodSpeed integration of a forked arrow-rs.

@samuelcolvin
Copy link
Contributor

Thanks @adriencaccia I'm strongly in favor of adding codspeed. I discussed this with @alamb and I think he was broadly in favor.

We've been using codspeed on pydantic-core since 2022, and it's absolutely key to maintaining and improving performance. See for example pydantic/pydantic-core#1155 (comment).

@Xuanwo
Copy link
Member

Xuanwo commented Jul 29, 2024

Thanks to the action, the benchmarks are run on a simulated CPU and thus the measurements are stable (you can read more about it in our docs).

That's great, thanks a lot for the link.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Any new improvement worthy of a entry in the changelog
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants