initial stab at the numba.typed.List benchmarks #10
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This is an intial stab at the ASV tests for the `numba.typed.List`. Things that still need to be decided on: * Do we want to measure compile time too? If so, how? * Is compile time included in these benchmarks or not? Lastly, here is a snapshot of what it looks like when run on my laptop: ``` 💣 zsh» asv run --show-stderr -b 'bench_typed_list*' · Fetching recent changes. · Creating environments · Discovering benchmarks · Running 4 total benchmarks (1 commits * 1 environments * 4 benchmarks) [ 0.00%] · For numba commit 1949c62e <master>: [ 0.00%] ·· Benchmarking conda-py3.6-cudatoolkit-llvmlite-numpy [ 12.50%] ··· Running (bench_typed_list.ConstructionSuite.time_construct_from_python_list--).... [ 62.50%] ··· bench_typed_list.ConstructionSuite.time_construct_from_python_list 97.8±1ms [ 75.00%] ··· bench_typed_list.ConstructionSuite.time_construct_in_njit_function 2.61±0.1ms [ 87.50%] ··· bench_typed_list.ReductionSuite.time_reduction_sum 203±9μs [100.00%] ··· bench_typed_list.SortSuite.time_sort 117±4ms asv run --show-stderr -b 'bench_typed_list*' 23.89s user 1.37s system 85% cpu 29.624 total ```
sklam
reviewed
Sep 18, 2020
| class ConstructionSuite: | ||
|
|
||
| def setup(self): | ||
| self.pl = make_random_python_list(SIZE) |
benchmarks/bench_typed_list.py
Outdated
| class ReductionSuite: | ||
|
|
||
| def setup(self): | ||
| self.tl = make_random_typed_list(SIZE) |
|
We do want compile time. It can make a fresh dispatcher and run the |
benchmarks/bench_typed_list.py
Outdated
| agg += i | ||
| return agg | ||
|
|
||
| self.reduction_sum = njit(reduction_sum) |
There was a problem hiding this comment.
Without fastmath this won't vectorize (should probably test both with and without). I also wonder if the iterator will vectorize vs an explicit induced loop, something to look at. The iterator loop will likely be full of refops.
There was a problem hiding this comment.
I am testing both now, they seem to have very similar run-time characteristics, will need to look at vectorization. But testing both is worthwhile in it's own right. Testing compilation time too now.
To ensure the reproducibility of the results, we seed the (pseudo-) random number generator.
Since we want to catch compile time regressions too, we benchmark the compile-time.
Implement the steps in `dispatcher.compile` to clear out the cache etc.. to ensure we really do get a fresh compile during every benchmark.
This makes the function reusable.
|
I have update the tests to seed the RNG and to time compilation. Here is snapshot of a run on my machine: |
Distinguish between for-loop based and iterator based iteration and between integers and floats.
|
With recent updates, these are the current benchmarks for the changes introduced by: numba/numba#6278 |
README.md
Outdated
| Run `asv` on the first commit: | ||
|
|
||
| ```bash | ||
| asv run "-1 abcdefg |
There was a problem hiding this comment.
Suggested change
| asv run "-1 abcdefg | |
| asv run "-1 abcdefg" |
also, that's hexadecimal? Why's there a g in it?
There was a problem hiding this comment.
Not necessarily, it can be any commit-ish - which includes tags and branches. Though I understand that this is misleading because it looks like hex.
README.md
Outdated
| Run `asv` on the second commit: | ||
|
|
||
| ```bash | ||
| asv run "-1 1235567 |
There was a problem hiding this comment.
Suggested change
| asv run "-1 1235567 | |
| asv run "-1 1235567" |
README.md
Outdated
Comment on lines
51
to
57
| ```bash | ||
| asv run --verbose --show-stderr -b 'bench_typed_list' "-1 abcdefg" | ||
| ``` | ||
|
|
||
| ```bash | ||
| asv compare --machine machine.local "abcdefg" "1234567" | ||
| ``` |
There was a problem hiding this comment.
Add 1 line to say what they do/why they are useful?
Fixed some syntax issues, made the metsyntactic placeholders for commit-ish's less ambiguous and expanded the descriptions.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is an intial stab at the ASV tests for the
numba.typed.List.Things that still need to be decided on:
Lastly, here is a snapshot of what it looks like when run on my laptop: