Skip to content

Commit

Permalink
[DataPipe] Small doc improvement for S3
Browse files Browse the repository at this point in the history
ghstack-source-id: f0f7bf2c49540f65bc72ab8ceddb54418c9a99ec
Pull Request resolved: #784
  • Loading branch information
NivekT committed Sep 20, 2022
1 parent 2212cb7 commit d5bf3c7
Showing 1 changed file with 8 additions and 0 deletions.
8 changes: 8 additions & 0 deletions torchdata/datapipes/iter/load/s3io.py
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,10 @@ class S3FileListerIterDataPipe(IterDataPipe[str]):
until all files are iterated.
3. ``request_timeout_ms`` and ``region`` will overwrite settings in the configuration file or
environment variables.
4. For more details related to S3 IO DataPipe setup and AWS config, please see the `README file`_.
.. _README file:
https://github.com/pytorch/data/tree/main/torchdata/datapipes/iter/load#s3-io-datapipe-documentation
Args:
source_datapipe: a DataPipe that contains URLs/URL prefixes to s3 files
Expand Down Expand Up @@ -77,6 +81,10 @@ class S3FileLoaderIterDataPipe(IterDataPipe[Tuple[str, StreamWrapper]]):
1. ``source_datapipe`` **must** contain a list of valid S3 URLs.
2. ``request_timeout_ms`` and ``region`` will overwrite settings in the
configuration file or environment variables.
3. For more details related to S3 IO DataPipe setup and AWS config, please see the `README file`_.
.. _README file:
https://github.com/pytorch/data/tree/main/torchdata/datapipes/iter/load#s3-io-datapipe-documentation
Args:
source_datapipe: a DataPipe that contains URLs to s3 files
Expand Down

0 comments on commit d5bf3c7

Please sign in to comment.