-
Notifications
You must be signed in to change notification settings - Fork 19
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add test variants with compressible data #714
Conversation
I ran an A/B tests on distributed#7593 and I'm observing a very modest (5%), but consistent speedup in I need to scrap the current algorithm and synthetically create something similar to the zarr dataset. |
9961df3
to
60a078b
Compare
This is ready for review and merge. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM. 👍
ty @milesgranger for review |
Closes #696
Run the following tests twice, once with uncompressible data and another with highly compressible data, to display differences on the network stack in one use case rather than another:
test_anom_mean
test_climatic_mean
(currently skipped)test_vorticity
test_double_diff
test_dot_product
test_map_overlap_sample
This PR increases the overall runtime from 48min to 50min.
I've deliberately not touched
test_basic_sum
, which is always compressible, andtest_rechunk_*
, which are always uncompressible, because they have already a fair amount of permutations and I didn't feel that doubling everything (with the additional challenges in readability more than in runtime) would yield a benefit worth it.