Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Get asv benchmark suite running again. #641

Merged
merged 1 commit into from
Jan 31, 2024

Conversation

danielballan
Copy link
Member

@danielballan danielballan commented Jan 25, 2024

Tiled has a nascent (nascent!) suite of benchmarks using asv. They have languished; the last update was in October 2021.

This PR gets them back into working order. Here is an example, running on this PR branch:

$ asv run --environment existing:same
· Discovering benchmarks
· Running 3 total benchmarks (1 commits * 1 environments * 3 benchmarks)
[ 0.00%] ·· Benchmarking existing-py_home_dallan_micromamba_envs_py311_bin_python3.11
[16.67%] ··· Running (benchmarks.TimeSuite.time_list_tree--)...
[66.67%] ··· benchmarks.TimeSuite.time_list_tree                                                                                                                                                 8.92±0.5ms
[83.33%] ··· benchmarks.TimeSuite.time_lookup                                                                                                                                                    4.63±0.2ms
[100.00%] ··· benchmarks.TimeSuite.time_lookup_and_read                                                                                                                                           12.9±0.4ms

Note: asv is designed to create dedicated conda environments to run its benchmarks in. This facilitates testing against different versions of Python, numpy, etc. if you configure it so. It is designed to run across ranged of commits, tracking a project's history, and publishing the results as nice plots in a static web site.

For quick tests though, I like the above incantation, which just runs the current commit in the current Python environment.

def setup(self):
self.server_process = subprocess.Popen(
Copy link
Member Author

@danielballan danielballan Jan 25, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Notice that instead of starting a server in a subprocess, this PR runs the server and client in the same process. Messages are passed directly via Python function calls, the same as it works in unit tests. This takes networking out of the equation and will give more stable results, focused on actual Tiled performance.

Copy link
Contributor

@padraic-shafer padraic-shafer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@danielballan danielballan merged commit fa00781 into bluesky:main Jan 31, 2024
8 checks passed
@danielballan danielballan deleted the update-benchmarks branch January 31, 2024 11:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants