Target 8192 blocks instead of split to large grid for large reduction #35997
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
When the number of blocks is large enough, we are already achieving blalanced SM allocation. But we still should keep the number of inputs per thread large, because thread reduce is cheap. Benchmark for Half on V100: https://github.com/zasdfgbnm/things/blob/master/2020Q2/reduction-benchmark.ipynb On large tensor, it is: 1.37ms vs 1.25ms [ghstack-poisoned]
💊 CircleCI build failures summary and remediationsAs of commit 974fe24 (more details on the Dr. CI page):
1 failure not recognized by patterns:
❄️ 5 tentatively flaky failures5 failures tentatively classified as flaky but have not triggered reruns to confirm:
|
…e reduction" When the number of blocks is large enough, we are already achieving blalanced SM allocation. But we still should keep the number of inputs per thread large, because thread reduce is cheap. Benchmark for Half on V100: https://github.com/zasdfgbnm/things/blob/master/2020Q2/reduction-benchmark.ipynb On large tensor, it is: 1.37ms vs 1.25ms [ghstack-poisoned]
jjsjann123
reviewed
Apr 4, 2020
jjsjann123
approved these changes
Apr 4, 2020
…e reduction" When the number of blocks is large enough, we are already achieving blalanced SM allocation. But we still should keep the number of inputs per thread large, because thread reduce is cheap. Benchmark for Half on V100: https://github.com/zasdfgbnm/things/blob/master/2020Q2/reduction-benchmark.ipynb On large tensor, it is: 1.37ms vs 1.25ms [ghstack-poisoned]
…e reduction" When the number of blocks is large enough, we are already achieving blalanced SM allocation. But we still should keep the number of inputs per thread large, because thread reduce is cheap. Benchmark for Half on V100: https://github.com/zasdfgbnm/things/blob/master/2020Q2/reduction-benchmark.ipynb On large tensor, it is: 1.37ms vs 1.25ms Differential Revision: [D20927533](https://our.internmc.facebook.com/intern/diff/D20927533) [ghstack-poisoned]
…e reduction" When the number of blocks is large enough, we are already achieving blalanced SM allocation. But we still should keep the number of inputs per thread large, because thread reduce is cheap. Benchmark for Half on V100: https://github.com/zasdfgbnm/things/blob/master/2020Q2/reduction-benchmark.ipynb On large tensor, it is: 1.37ms vs 1.25ms Differential Revision: [D20927533](https://our.internmc.facebook.com/intern/diff/D20927533) [ghstack-poisoned]
zasdfgbnm
changed the title
ngimel
approved these changes
Apr 9, 2020
aten/src/ATen/native/cuda/Reduce.cuh
Outdated
| @@ -789,15 +789,23 @@ inline void gpu_reduce_kernel(TensorIterator& iter, const ops_t& ops, ident_t id | |||
| config.output_mult[1] = config.split_output(block_height); | |||
| } | |||
|
|
|||
| if (config.input_mult[1] != 0 && config.values_per_thread() >= 256 && num_outputs <= 4096) { | |||
| constexpr int target_grid_size = 4096; | |||
There was a problem hiding this comment.
This generally looks good, but probably you can make it even less than 4096? You should be targeting full occupancy, which will come out to less than 4096?
ashishfarmer
pushed a commit
to ashishfarmer/pytorch
that referenced
this pull request
Apr 13, 2020
…pytorch#35997) Summary: Pull Request resolved: pytorch#35997 When the number of blocks is large enough, we are already achieving blalanced SM allocation. But we still should keep the number of inputs per thread large, because thread reduce is cheap. Benchmark for Half on V100: https://github.com/zasdfgbnm/things/blob/master/2020Q2/reduction-benchmark.ipynb On large tensor, it is: 1.37ms vs 1.25ms Test Plan: Imported from OSS Differential Revision: D20927533 Pulled By: ngimel fbshipit-source-id: 40df52e439cc1c01cda66c6195b600f301c5e984
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Stack from ghstack:
When the number of blocks is large enough, we are already achieving
blalanced SM allocation. But we still should keep the number of inputs
per thread large, because thread reduce is cheap.
Benchmark for Half on V100:
https://github.com/zasdfgbnm/things/blob/master/2020Q2/reduction-benchmark.ipynb
On large tensor, it is: 1.37ms vs 1.25ms
Differential Revision: D20927533