Skip to content

Commit

Permalink
Update tutorial about placing sharding_filter (#487)
Browse files Browse the repository at this point in the history
Summary:
See the feedback from a user: #454 (comment)

We should explicitly ask users to place `sharding_filter` as early as possible.

Pull Request resolved: #487

Reviewed By: wenleix

Differential Revision: D36812259

Pulled By: ejguan

fbshipit-source-id: 4c983f3216a80be398f85b20871e65b0e41627e0
  • Loading branch information
ejguan authored and facebook-github-bot committed Jun 2, 2022
1 parent cd38927 commit 6f1f81d
Showing 1 changed file with 5 additions and 0 deletions.
5 changes: 5 additions & 0 deletions docs/source/tutorial.rst
Original file line number Diff line number Diff line change
Expand Up @@ -176,6 +176,11 @@ When we re-run, we will get:
...
n_sample = 6
Note:

- Place ``ShardingFilter`` (``datapipe.sharding_filter``) as early as possible in the pipeline, especially before expensive
operations such as decoding, in order to avoid repeating these expensive operations across worker/distributed processes.


You can find more DataPipe implementation examples for various research domains `on this page <torchexamples.html>`_.

Expand Down

0 comments on commit 6f1f81d

Please sign in to comment.