Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make MostInlined and BestEffort inline propagation no longer assert replayed #1868

Merged
merged 13 commits into from
Jul 29, 2022

Conversation

zasdfgbnm
Copy link
Collaborator

Currently, when doing MostInlined propagation, at each step propagateX2Y, the inline propagator TORCH_CHECK that Y is already replayed consistently with X at -1, then it will inline Y with the innermost dimension of X. BestEffort is similar except that it allows specifying a position.

When writing my transpose PR #1854, I don't feel this behavior very convenient to use. For example, if I have a DAG:

T0[I1, I2, I3] = input
T1[I1, I2, I3] = sin(T0)
T2[I1, I2, I3] = cos(T1)
T3[I1, I3, I2] = transpose(T2)
T4[I1, I3, I2] = exp(T3)
output = T5[I1, I3, I2] = tan(T4)

If I do a MostInlined inline propagation starting from T5, I will get an error message when propagating from T3 to T2 saying that dims do not match and replay is required. But what I really want is to just don't inline into the mismatched dimension and continue the propagation so that the sin will be inlined into the innermost loop of cos.

I think generally, it makes sense to make MostInlined and BestEffort to just accept the current replay status as is, and just don't inline into mismatched dimensions.

@zasdfgbnm zasdfgbnm mentioned this pull request Jul 26, 2022
4 tasks
@zasdfgbnm zasdfgbnm marked this pull request as draft July 26, 2022 16:10
@zasdfgbnm
Copy link
Collaborator Author

Making this a draft to play with something and add unit tests.

Copy link
Owner

@csarofeen csarofeen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@zasdfgbnm zasdfgbnm changed the base branch from devel to merge-trivial-reduction July 27, 2022 05:58
@zasdfgbnm zasdfgbnm marked this pull request as ready for review July 27, 2022 06:35
@zasdfgbnm
Copy link
Collaborator Author

Rebased to #1871

Base automatically changed from merge-trivial-reduction to devel July 29, 2022 03:31
@zasdfgbnm zasdfgbnm merged commit 172fb36 into devel Jul 29, 2022
@zasdfgbnm zasdfgbnm deleted the inline-propagator branch July 29, 2022 03:51
jjsjann123 added a commit that referenced this pull request Aug 29, 2022
Syncing nvfuser devel branch to upstream master. https://github.com/csarofeen/pytorch/

Code changes includes:

- codegen improvements:
  1. removes un-necessary sync from redundant thread compute analysis
  2. symmetric API for BestEffortReplay
  3. support merge on trivial reductions
  4. Ampere async copy improvements
- bug fixes:
  1. vectorization bug fixes
  2. type inference patch : fixes upstream pytorch#81725
  3. segmenter bug fix with deterministic iteration ordering
- parser update
  1. added leaky_relu
- scheduler
  1. normalization scheduler clean up.
  2. simplifies matmul scheduling with new transform propagator
  3. merge all dimensions in PW scheduler
  4. various gemm related improvements
- debuggability
  1. nsight compute support
  2. debug dump for InlinePropagator
  3. Add `UnaryOpType::Print`

Squashed commits to WAR github API
Commits that's actually in this PR from the devel branch:

```
dfe02f3 Merge remote-tracking branch 'csarofeen/devel' into HEAD
1617373 Add `TensorViewBuilder::shape(std::vector<Val*> shape)` (#1884)
7cfb779 Merge pull request #1887 from csarofeen/upstream_merge_0803
3399f6d Merge remote-tracking branch 'origin/viable/strict' into HEAD
01208f5 Add `UnaryOpType::Print` which can be helpful for debugging (#1878)
0646522 Remove redundant TORCH_INTERNAL_ASSERT in lower_magic_zero.cpp (#1881)
7bc76aa Fix most inlined propagator for mismatched dims (#1875)
501f4aa Nonaffine swizzle formulation ep.2: Loop swizzle variant. (#1826)
d863d69 Ampere async copy ep.2: circular buffering extension to support pipelined matmul operand load (#1827)
e0ae11a Larger sized mma instructions to support full vectorization (#1824)
9bb4cf7 fragment iteration to support fully unrolled mma ops (#1823)
a48270a Merge all dims in pointwise scheduler (#1872)
172fb36 Make MostInlined and BestEffort inline propagation no longer assert replayed (#1868)
a64462a Allow trivial reduction to be merged (#1871)
440102b Symmetric API for BestEffortReplay (#1870)
d1caf33 Some misc cleanups/refactor split out from #1854 (#1867)
1013eda Remove some welford specific logic. (#1864)
51589d3 Some cleanups on tests and heuristics params (#1866)
a6b3e70 Segmenter bug fix, and deterministic iteration ordering.  (#1865)
1b665b9 Add nullptr checks to IrBuilder (#1861)
1cd9451 Simplify matmul scheduling with the new transform propagator.  (#1817)
bbc1fb9 Add leaky_relu operation (#1852)
e842a9b Minor cleanup in pointwise scheduler (#1858)
9ee850c Fix stringstream usage (#1857)
20a36c1 Improve nsight compute support (#1855)
4059103 Remove debugging `true ||` from getPointwiseHeuristics (#1822)
01117bf Misc cleanup (#1853)
5cc6494 Apply the magic-zero protection to each indexed domain individually for predicate indexing (#1846)
92e6f02 Cleanup normalization scheduler (#1845)
db89c65 Type inference patch (#1848)
102fe93 Add debug dump for InlinePropagator (#1847)
b7a4d93 Redundant thread compute analysis to avoid un-necessary sync insertion (#1687)
942be5b Upstream ci build fixes (#1842)
0b83645 Fix vectorization bug introduced in #1831 (#1840)
63630f1 Move MaxProducerPosUpdater into InlinePropagator::tearDown (#1825)
9135a96 Fix transpose benchmark dtype (#1839)
2c9a6c0 Add extra configurability to `parallelizeAllLike` (#1831)
```

RUN_TORCHBENCH: nvfuser

Differential Revision: [D38543000](https://our.internmc.facebook.com/intern/diff/D38543000)
Pull Request resolved: pytorch#83067
Approved by: https://github.com/davidberard98
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants