Grouping grid allreduces across iterations #1755

naoyam · 2022-06-08T00:04:02Z

ParallelType::Group is added. It is only allowed with grid allreduces, currently. Welford is not supported at this time.

A grouped IterDomain is treated like a vectorized IterDomain. It is fully unrolled and each reduction operation is packed to a single call to the runtime template function.

Here's a packed grid reduction example from FusionCrossIterationGroupedGridAllreduce1:

  T2_reduction.reduceGroup(
    RefTuple<float, float>(T2[0], T2[1]),
    ConstRefTuple<float, float>(T1[0], T1[1]),
    VolatilePtrTuple<float, float>(&T5[0], &T5[((((nvfuser_index_t)gridDim.x) * ((nvfuser_index_t)gridDim.y)) * 32) * 1]),
    LocalTuple<float, float>(0, 0),
    &T6[0],
    shared_mem,
    LocalTuple<bool, bool>(true, true),
    LocalTuple<bool, bool>(true, true),
    [](float &a, float b) { a = a + b; }, [](float &a, float b) { a = a + b; });

Confirmed that the performance benefit experimentally obtained at PR #1731 can be reproduced by just using this Group parallel type as shown in FusionGroupedReductionPersistentChannelsLastBatchNormLike

Follow-up TODOs

Enables grid persistence in the reduction scheduler
Use the grouping features in the reduction scheduler

The kernel itself should work with an arbitrary number of inputs, but the underlying data structure, Tuple, still explicitly needs to be specialized for the number of values, which is currently limited to 8.

csarofeen

Codegen changes seem a bit complex and only reviewed them quickly. Seems fine to me though. It should be part of the future looking part that we reevaluate this strategy and how it could extend to Multi-GPU.

csarofeen · 2022-06-24T19:37:40Z

torch/csrc/jit/codegen/cuda/lower_index_hoist.cpp

    return {index, false};
  }

  const CommonIndexKey key(
      indexed_consumer_id, consumer_td, ref_td, ref_index_map, loops);

+  // Hoisting is not possible if any of used loops is grouped.


Would this be an issue for MMA? @shmsong

I don't think this would be an issue. In matmul patterns the values to reduce would be in registers, which would have to be constant indices anyway, so no need to hoist the tensor index at this point, and most use cases should be cascading reductions, i.e. rfactors, so predicate would be removed.

csarofeen · 2022-06-24T19:43:15Z

torch/csrc/jit/codegen/cuda/lower_index.cpp

  // Use a unique buffer for work and sync flag when called within a
  // loop unless it's persistent. Grid all reduce means persistence is
  // required. However, not being a grid all reduce does not mean
  // non-persistence. Currently, if a cooperative grid reduction is
  // required anywhere in the kernel, all grid reducitons are done in
  // a persistent manner, so all grid reductions should be consulted.
  // TODO: fix this
-  const bool privatize_buffer = !rop->isAllreduce();
+  const bool is_persistent = rop->isAllreduce();


Why is it called is_persistent if it's really if all reduce? Is this a persistent rule of just a special rule if it's a persistent "grid all reduce"?

Yes, is_persistent is now equivalent to isAlllreduce() , but it was not the case before. I think the name makes more sense.

torch/csrc/jit/codegen/cuda/lower_validation.cpp

torch/csrc/jit/codegen/cuda/lower2device.cpp

csarofeen · 2022-06-24T19:51:57Z

torch/csrc/jit/codegen/cuda/compute_at.cpp

@@ -48,6 +48,18 @@ bool validateDomain(TensorView* tv, TensorDomain* new_td) {
      first_mismatch >= (int)tv->getComputeAtPosition();
 }

+// Check if inlining can be done at an ID.


Just marking a conflict warning for @zasdfgbnm

torch/csrc/jit/codegen/cuda/codegen.cpp

csarofeen · 2022-07-01T15:09:28Z

@naoyam could we try to get this merged in before it gets too stale?

naoyam · 2022-07-01T16:16:32Z

@naoyam could we try to get this merged in before it gets too stale?

Yes, I should be able to do so.

csarofeen · 2022-07-05T10:34:33Z

I'm satisfied with the updates, please resolve conflicts and feel free to merge in.

@csarofeen

* Refactor TransormPropagator to allow specifying a position and propagating to part of the DAG (#1775) `MaxInfoPropagator` is renamed to `MaxInfoSpanningTree`, it now only does path-finding, and the propagation is in a separate class `MaxInfoSpanningTree::Propagator`. Same for `MaxRootDomainInfoPropagator`. `MaxInfoSpanningTree` and `MaxRootDomainInfoSpanningTree` now allow specifying a selector, which controls which subgraph should be included in path-finding. `MaxRootDomainInfoSpanningTree` also gets a few new constructors for convenience to use. `TransormPropagator` is now a subclass of `MaxInfoSpanningTree::Propagator`, so the way to use it has changed. Now `MaxInfoSpanningTree` and `MaxRootDomainInfoSpanningTree` will store the path after generation so that the same path can be traversed multiple times. This will be useful to support use cases like new `computeAt`. Pseudo-code: ```C++ void TensorView::computeAt(TensorView tv, int pos) { auto ComputeAtSubgraphSelector selector(this, tv); MaxRootDomainInfoSpanningTree path(tv, pos, &selector); TransformPropagator propagator(tv, pos); path.traverse(&propagator); ComputeAtPosPropagator ca_propagator(tv, pos); path.traverse(&ca_propagator); } ``` * Revert scheduling changes. Cleanup only. * Start drafting grid persistent kernels. * Extend mma dimension and layout checking to support strided batched matmul and tensor contractions (#1761) Co-authored-by: Christian Sarofeen <csarofeen@nvidia.com> * Fix FusionMaxRootDomainInfoSpanningTreePrintTwice_CUDA (#1781) * Fix div(Val, TensorView) (#1778) * Fix div(scalar, tensor) * lintrunner: clang-format * Adding sibling path for MaxInfoSpanningTree (#1776) The sibling path is required to generate consistent replay for some cases where `MaxInfoSpanningTree` is used with a selector. For example, when the producer of a Welford is excluded from the propagation section. See test `FusionTransformPropagateSelectorSibling_CUDA` for a detailed example. Besides, since we know that siblings should be transformed exactly the same, the sibling path is a perfect next hop for preserving information. If you want a spanning tree without a sibling path, you can override `allowSibling` as `return false` in your selector; * Save. * Disable register reuse across serial broadcast ops (#1787) Disable memory aliasing for inner sharing across serial broadcast. * Fix isIntegralType error msg (#1789) * Transform propagator skip replay when possible (#1782) This comment in the code describes what this PR is doing: ```C++ // Note: [Using multiple TransformPropagators] // There are cases that we use multiple TransformPropagators along different // spanning trees with different references in the same fusion. Some of these // spanning trees could overlap. In cases when there are overlapping nodes, // TransformPropagator needs to respect the replay of others, because the // current TransformPropagator might not contain the most amount of // information on how to do the correct transformation. The logic below tells // TransformPropagator to skip the replay when not necessary. ``` * Output allocate patch (#1790) Caching strides along with sizes. This is to support current expand, which introduces non-contiguous output tensor * Add SpanningTreePrinter (#1786) * New compute at interface (#1743) Rewrite of the compute at pass to rely on the new propagation mechanisms. * Fix TransformReplay::getMatchedLeafPosWithoutReplay* (#1791) * Some further cleanup for the new computeAt interface (#1793) Revert MaxProducerPosUpdater to old algo. * Use TransformPropagatorWithCheck in many tests (#1795) * validateDomain in TransformPropagator (#1796) * InlinePropagator please don't replay (#1797) This PR makes `InlinePropagator` just set compute-at positions. It will not replay any tensor. If you want to replay, please use `TransformPropagator` and friends to do so. Currently, `InlinePropagator` is already asserting no replay for standard and best effort compute at. So this PR is mostly about making most inlined compute at works as well. This PR also does a lot of cleanups to remove the word "replay" from comments and variable and function names from `InlinePropagator`. I also cleaned up `recordReplayedPos` and `retrieveReplayedPos`, now the logic is much easier to understand. * Coding style cleanups (#1798) Per offline discussion with @csarofeen, this PR does many renaming for better coding style: For all propagation-related things, I am now using the names `P2C` and `C2P` instead of `CasP` and `PasC`. Because "A as B" somewhat implies we want to replay A the same as B, but "B to A" sounds more general and is a better word for this case. Also, I modified the order of function arguments to match the order in its name. For example `PasC` should have `(producer, consumer)` or `(to, from)`, but not `(consumer, producer)` or `(from, to)`, and `C2P` should have `(consumer, producer)` or `(from, to)`, but not `(producer, consumer)` or `(to, from)`. * Add parsing support for `_to_copy` to handle AMP casts. (#1756) 1. Add support for _to_copy() to support AMP casts. 2. refactored cast, accept none for dtype 3. python tests Co-authored-by: jjsjann123 <jiej@nvidia.com> * MMA Rfactor support for cross-warp and cross-CTA split on K dimension (#1554) * Indexing refactor stage 2 : Remove reference tensor in predicate indexing logic (#1784) Co-authored-by: Christian Sarofeen <csarofeen@nvidia.com> * More cleanup on InlinePropagator (#1800) I just realized that `InlinePropagator` can be further simplified because it no longer replays. Since `InlinePropagator` is no longer doing replay, it is more like a "for each" problem rather than a propagation problem: For each tensor `tv`, if we already know what is the max position of `tv` that is mapped to the reference tensor's selected outer dimensions(stored in `mapped_reference_pos_` in the code), setting the CA position is a very local operation, and is as simple as checking `tv` itself and all its consumers to determine the inline position. `InlinePropagator` is not completely a "for each" problem only because the computation of `mapped_reference_pos_` is a propagation problem. This cleanup reorganizes the code of `InlinePropagator` so it is clear that `InlinePropagator` is nothing but a two-step process: Step 1: Do a propagation to find the `mapped_reference_pos_` for all tensors. Step 2: For each tensor, check itself and its consumers to determine the CA position. Conceptually, I would like to split step 1 with step 2. Because this split makes these concepts decoupled. Especially, this PR makes `mapped_reference_pos_` only contain info about the reference tensor, and is independent of the CA position (Currently, this is not true for best effort and most inlined computeAt without this PR). Now, in my view, `InlinePropagator` is conceptually very simple and easy to understand. In terms of implementation, step 1 and step 2 can be interleaved, because when we don't need to know the `mapped_reference_pos_` for `tv`'s consumer in order to compute the CA position of `tv`. So a one-pass traverse could do both step 1 and step 2 altogether. * Temporarily disable test requring large shared memory. (#1802) * Grouping grid allreduces across iterations (#1755) * Extend the grouped grid reduction kernel The kernel itself should work with an arbitrary number of inputs, but the underlying data structure, Tuple, still explicitly needs to be specialized for the number of values, which is currently limited to 8. * Check siblings in getMaxPosAll (#1805) * remove dead indexing code (#1806) * Broadcast in dim with expand (#1794) Fixes #1788 Added expand in broadcast_in_dim to support expanding to concrete size. Note that we are not supporting dynamic shape for concrete size at this moment. * spam nvrtc options (#1783) TORCH_WARN on nvrtc debug option impacting performance. Co-authored-by: Gao, Xiang <qasdfgtyuiop@gmail.com> Co-authored-by: S. Song <41357537+shmsong@users.noreply.github.com> Co-authored-by: Ivan Yashchuk <IvanYashchuk@users.noreply.github.com> Co-authored-by: Sergey Lebedev <sergeyle@nvidia.com> Co-authored-by: jjsjann123 <jiej@nvidia.com> Co-authored-by: Kevin Stephano <kevin.stephano@gmail.com> Co-authored-by: Naoya Maruyama <naoyam@users.noreply.github.com>

Syncing nvfuser devel branch to upstream master. https://github.com/csarofeen/pytorch/ Code changes includes: - codegen improvements: 1. Indexing refactor -> Remove reference tensor in predicate indexing logic 2. MMA Rfactor support for cross-warp and cross-CTA split on K dimension 3. Grouping grid allreduces across iterations 4. Swizzle op formulation for non-affine swizzles 5. Use scheduler_utils to cache inputs and outputs in schedulePointwise - scheduler refactor 1. New compute at interface - transformation propagation refactor on MaxInfoSpanningTree 1. Added sibling path that is required to generate consistent replay for some cases where `MaxInfoSpanningTree` is used with a selector. 2. Optimization to skip Transform propagator 3. SpanningTreePrinter for debugging - parser update 1. Fixes `div` 2. Added `_to_copy` 3. Broadcast in dim with expand to support expanding to concrete size 4. Dropout prob extremal patch - executor patch on caching strides for output allocation Squashed commits to WAR github API Commits that's actually in this PR from the devel branch: ``` 3b87896 Fix allocation of work buffers and `fused_reduction::ParallelReduce` with unswitch (csarofeen#1818) 4cae122 schedulePointwise cleanup: - computeAt + InlinePropagator (csarofeen#1815) 3df9742 Use scheduler_utils to cache inputs and outputs in schedulePointwise (csarofeen#1811) 03180aa improve broadcast resolution (csarofeen#1792) bee6c69 bug fix (csarofeen#1819) 4413c8f Support PYTORCH_NVFUSER_DUMP=transform_propagator (csarofeen#1812) de6b7ca Fix negative position in InlinePropagator (csarofeen#1813) 10a996c Remove redundant check in schedulePointwise (csarofeen#1810) acd5ed4 Swizzle op formulation for non-affine swizzles (csarofeen#1441) 3ed8330 Kernel args patch to show zero_init buffer (csarofeen#1809) 037a75a Dropout prob extremal patch (csarofeen#1804) 282c429 spam nvrtc options (csarofeen#1783) 3ba6a5f Broadcast in dim with expand (csarofeen#1794) fd4be12 remove dead indexing code (csarofeen#1806) fa4e6a4 Check siblings in getMaxPosAll (csarofeen#1805) 025c840 Grouping grid allreduces across iterations (csarofeen#1755) 37c579e Temporarily disable test requring large shared memory. (csarofeen#1802) 5f375d0 More cleanup on InlinePropagator (csarofeen#1800) 8d384da Indexing refactor stage 2 : Remove reference tensor in predicate indexing logic (csarofeen#1784) f008140 MMA Rfactor support for cross-warp and cross-CTA split on K dimension (csarofeen#1554) 76b3cca Add parsing support for `_to_copy` to handle AMP casts. (csarofeen#1756) ef04f6c Coding style cleanups (csarofeen#1798) 38c7f3c InlinePropagator please don't replay (csarofeen#1797) 3f2c263 validateDomain in TransformPropagator (csarofeen#1796) c077085 Use TransformPropagatorWithCheck in many tests (csarofeen#1795) d0d0908 Some further cleanup for the new computeAt interface (csarofeen#1793) 45f5203 Fix TransformReplay::getMatchedLeafPosWithoutReplay* (csarofeen#1791) 28cbaf9 New compute at interface (csarofeen#1743) 635ebfc Add SpanningTreePrinter (csarofeen#1786) 59f3c32 Output allocate patch (csarofeen#1790) fe93bf5 Transform propagator skip replay when possible (csarofeen#1782) ebf23a5 Fix isIntegralType error msg (csarofeen#1789) 0c82ecf Disable register reuse across serial broadcast ops (csarofeen#1787) 33a824d Adding sibling path for MaxInfoSpanningTree (csarofeen#1776) 86f46aa Fix div(Val, TensorView) (csarofeen#1778) d3de227 Fix FusionMaxRootDomainInfoSpanningTreePrintTwice_CUDA (csarofeen#1781) ecc7a87 Extend mma dimension and layout checking to support strided batched matmul and tensor contractions (csarofeen#1761) ``` [ghstack-poisoned]

Syncing nvfuser devel branch to upstream master. https://github.com/csarofeen/pytorch/ Code changes includes: - codegen improvements: 1. Indexing refactor -> Remove reference tensor in predicate indexing logic 2. MMA Rfactor support for cross-warp and cross-CTA split on K dimension 3. Grouping grid allreduces across iterations 4. Swizzle op formulation for non-affine swizzles 5. Use scheduler_utils to cache inputs and outputs in schedulePointwise - scheduler refactor 1. New compute at interface - transformation propagation refactor on MaxInfoSpanningTree 1. Added sibling path that is required to generate consistent replay for some cases where `MaxInfoSpanningTree` is used with a selector. 2. Optimization to skip Transform propagator 3. SpanningTreePrinter for debugging - parser update 1. Fixes `div` 2. Added `_to_copy` 3. Broadcast in dim with expand to support expanding to concrete size 4. Dropout prob extremal patch - executor patch on caching strides for output allocation Squashed commits to WAR github API Commits that's actually in this PR from the devel branch: ``` 3b87896 Fix allocation of work buffers and `fused_reduction::ParallelReduce` with unswitch (csarofeen#1818) 4cae122 schedulePointwise cleanup: - computeAt + InlinePropagator (csarofeen#1815) 3df9742 Use scheduler_utils to cache inputs and outputs in schedulePointwise (csarofeen#1811) 03180aa improve broadcast resolution (csarofeen#1792) bee6c69 bug fix (csarofeen#1819) 4413c8f Support PYTORCH_NVFUSER_DUMP=transform_propagator (csarofeen#1812) de6b7ca Fix negative position in InlinePropagator (csarofeen#1813) 10a996c Remove redundant check in schedulePointwise (csarofeen#1810) acd5ed4 Swizzle op formulation for non-affine swizzles (csarofeen#1441) 3ed8330 Kernel args patch to show zero_init buffer (csarofeen#1809) 037a75a Dropout prob extremal patch (csarofeen#1804) 282c429 spam nvrtc options (csarofeen#1783) 3ba6a5f Broadcast in dim with expand (csarofeen#1794) fd4be12 remove dead indexing code (csarofeen#1806) fa4e6a4 Check siblings in getMaxPosAll (csarofeen#1805) 025c840 Grouping grid allreduces across iterations (csarofeen#1755) 37c579e Temporarily disable test requring large shared memory. (csarofeen#1802) 5f375d0 More cleanup on InlinePropagator (csarofeen#1800) 8d384da Indexing refactor stage 2 : Remove reference tensor in predicate indexing logic (csarofeen#1784) f008140 MMA Rfactor support for cross-warp and cross-CTA split on K dimension (csarofeen#1554) 76b3cca Add parsing support for `_to_copy` to handle AMP casts. (csarofeen#1756) ef04f6c Coding style cleanups (csarofeen#1798) 38c7f3c InlinePropagator please don't replay (csarofeen#1797) 3f2c263 validateDomain in TransformPropagator (csarofeen#1796) c077085 Use TransformPropagatorWithCheck in many tests (csarofeen#1795) d0d0908 Some further cleanup for the new computeAt interface (csarofeen#1793) 45f5203 Fix TransformReplay::getMatchedLeafPosWithoutReplay* (csarofeen#1791) 28cbaf9 New compute at interface (csarofeen#1743) 635ebfc Add SpanningTreePrinter (csarofeen#1786) 59f3c32 Output allocate patch (csarofeen#1790) fe93bf5 Transform propagator skip replay when possible (csarofeen#1782) ebf23a5 Fix isIntegralType error msg (csarofeen#1789) 0c82ecf Disable register reuse across serial broadcast ops (csarofeen#1787) 33a824d Adding sibling path for MaxInfoSpanningTree (csarofeen#1776) 86f46aa Fix div(Val, TensorView) (csarofeen#1778) d3de227 Fix FusionMaxRootDomainInfoSpanningTreePrintTwice_CUDA (csarofeen#1781) ecc7a87 Extend mma dimension and layout checking to support strided batched matmul and tensor contractions (csarofeen#1761) ``` RUN_TORCHBENCH: nvfuser Differential Revision: [D38043938](https://our.internmc.facebook.com/intern/diff/D38043938) [ghstack-poisoned]

Syncing nvfuser devel branch to upstream master. https://github.com/csarofeen/pytorch/ Code changes includes: - codegen improvements: 1. Indexing refactor -> Remove reference tensor in predicate indexing logic 2. MMA Rfactor support for cross-warp and cross-CTA split on K dimension 3. Grouping grid allreduces across iterations 4. Swizzle op formulation for non-affine swizzles 5. Use scheduler_utils to cache inputs and outputs in schedulePointwise - scheduler refactor 1. New compute at interface - transformation propagation refactor on MaxInfoSpanningTree 1. Added sibling path that is required to generate consistent replay for some cases where `MaxInfoSpanningTree` is used with a selector. 2. Optimization to skip Transform propagator 3. SpanningTreePrinter for debugging - parser update 1. Fixes `div` 2. Added `_to_copy` 3. Broadcast in dim with expand to support expanding to concrete size 4. Dropout prob extremal patch - executor patch on caching strides for output allocation Squashed commits to WAR github API Commits that's actually in this PR from the devel branch: ``` 3b87896 Fix allocation of work buffers and `fused_reduction::ParallelReduce` with unswitch (#1818) 4cae122 schedulePointwise cleanup: - computeAt + InlinePropagator (#1815) 3df9742 Use scheduler_utils to cache inputs and outputs in schedulePointwise (#1811) 03180aa improve broadcast resolution (#1792) bee6c69 bug fix (#1819) 4413c8f Support PYTORCH_NVFUSER_DUMP=transform_propagator (#1812) de6b7ca Fix negative position in InlinePropagator (#1813) 10a996c Remove redundant check in schedulePointwise (#1810) acd5ed4 Swizzle op formulation for non-affine swizzles (#1441) 3ed8330 Kernel args patch to show zero_init buffer (#1809) 037a75a Dropout prob extremal patch (#1804) 282c429 spam nvrtc options (#1783) 3ba6a5f Broadcast in dim with expand (#1794) fd4be12 remove dead indexing code (#1806) fa4e6a4 Check siblings in getMaxPosAll (#1805) 025c840 Grouping grid allreduces across iterations (#1755) 37c579e Temporarily disable test requring large shared memory. (#1802) 5f375d0 More cleanup on InlinePropagator (#1800) 8d384da Indexing refactor stage 2 : Remove reference tensor in predicate indexing logic (#1784) f008140 MMA Rfactor support for cross-warp and cross-CTA split on K dimension (#1554) 76b3cca Add parsing support for `_to_copy` to handle AMP casts. (#1756) ef04f6c Coding style cleanups (#1798) 38c7f3c InlinePropagator please don't replay (#1797) 3f2c263 validateDomain in TransformPropagator (#1796) c077085 Use TransformPropagatorWithCheck in many tests (#1795) d0d0908 Some further cleanup for the new computeAt interface (#1793) 45f5203 Fix TransformReplay::getMatchedLeafPosWithoutReplay* (#1791) 28cbaf9 New compute at interface (#1743) 635ebfc Add SpanningTreePrinter (#1786) 59f3c32 Output allocate patch (#1790) fe93bf5 Transform propagator skip replay when possible (#1782) ebf23a5 Fix isIntegralType error msg (#1789) 0c82ecf Disable register reuse across serial broadcast ops (#1787) 33a824d Adding sibling path for MaxInfoSpanningTree (#1776) 86f46aa Fix div(Val, TensorView) (#1778) d3de227 Fix FusionMaxRootDomainInfoSpanningTreePrintTwice_CUDA (#1781) ecc7a87 Extend mma dimension and layout checking to support strided batched matmul and tensor contractions (#1761) ``` RUN_TORCHBENCH: nvfuser Differential Revision: [D38043938](https://our.internmc.facebook.com/intern/diff/D38043938) Pull Request resolved: pytorch#81861 Approved by: https://github.com/davidberard98

naoyam added 5 commits June 1, 2022 23:30

Extend the grouped grid reduction kernel

288ef90

The kernel itself should work with an arbitrary number of inputs, but the underlying data structure, Tuple, still explicitly needs to be specialized for the number of values, which is currently limited to 8.

cleanup

1f5f861

cleanup

5f9a9ae

Merge branch 'devel' into grid_reduction_runtime_kernel_ext

8787578

Merge branch 'devel' into grid_reduction_runtime_kernel_ext

1d95801

naoyam force-pushed the grouped_grid_reduction_across_iterations branch 2 times, most recently from a3ac316 to d4b11c7 Compare June 9, 2022 00:50

naoyam added 4 commits June 13, 2022 11:09

Add a test of grouped allreduce with 3 different types

4670707

Add more comments

4398e31

Merge branch 'devel' into grid_reduction_runtime_kernel_ext

d5cf60d

Add ParallelType::Group

f4e3291

Base automatically changed from grid_reduction_runtime_kernel_ext to devel June 13, 2022 19:39

Merge branch 'devel' into grouped_grid_reduction_across_iterations

a1bc8f5

naoyam force-pushed the grouped_grid_reduction_across_iterations branch from d4b11c7 to a1bc8f5 Compare June 13, 2022 20:03

naoyam changed the title ~~[WIP] Add ParallelType::Group~~ Grouping grid allreduces across iterations Jun 13, 2022

naoyam added 4 commits June 13, 2022 16:19

test updates; minor fix

4c68117

computeAt support

236feb5

comments

89cee41

cleanup

d3cdec4

naoyam marked this pull request as ready for review June 14, 2022 04:59

naoyam requested a review from csarofeen June 14, 2022 05:12

csarofeen approved these changes Jun 24, 2022

View reviewed changes

PR feedback

6120c79

naoyam added 2 commits July 5, 2022 08:45

Merge branch 'devel' into grouped_grid_reduction_across_iterations

e4f47d4

comment

536a18c

naoyam merged commit 025c840 into devel Jul 5, 2022

naoyam deleted the grouped_grid_reduction_across_iterations branch July 5, 2022 22:09

naoyam mentioned this pull request Dec 1, 2022

[WIP] Enable the persistent scheduler to use grid persistence #1772

Closed

9 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Grouping grid allreduces across iterations #1755

Grouping grid allreduces across iterations #1755

naoyam commented Jun 8, 2022 •

edited

Loading

csarofeen left a comment

csarofeen Jun 24, 2022

shmsong Jun 24, 2022 •

edited

Loading

csarofeen Jun 24, 2022

naoyam Jul 3, 2022

csarofeen Jun 24, 2022

csarofeen commented Jul 1, 2022

naoyam commented Jul 1, 2022

csarofeen commented Jul 5, 2022

Grouping grid allreduces across iterations #1755

Grouping grid allreduces across iterations #1755

Conversation

naoyam commented Jun 8, 2022 • edited Loading

csarofeen left a comment

Choose a reason for hiding this comment

csarofeen Jun 24, 2022

Choose a reason for hiding this comment

shmsong Jun 24, 2022 • edited Loading

Choose a reason for hiding this comment

csarofeen Jun 24, 2022

Choose a reason for hiding this comment

naoyam Jul 3, 2022

Choose a reason for hiding this comment

csarofeen Jun 24, 2022

Choose a reason for hiding this comment

csarofeen commented Jul 1, 2022

naoyam commented Jul 1, 2022

csarofeen commented Jul 5, 2022

naoyam commented Jun 8, 2022 •

edited

Loading

shmsong Jun 24, 2022 •

edited

Loading