Skip to content

Commit 571fd78

Browse files
d4l3kfacebook-github-bot
d4l3k
authored and
facebook-github-bot
committed
docs: add documentation for workspaces + general cleanup (#402)
Summary: This makes some of the changes listed in https://docs.google.com/document/d/14icG0GJG1nPHF3DW7yMreZpe3S8FwYdrI_XYIzDTSbI/edit#heading=h.6ixj5co4aocb * Adds workspaces documention * Adds coverage test to docs build to ensure all exported methods/classes are documented in the docs. * Adds previously undocumented methods/classes to docs * Moves torchconfig from experimental to beta * Splits docker/local scheduler docs * Reshuffles the sidebar to prioritize the schedulers over some of the other content Next steps: * update readme/index/quickstart guides to highlight workspaces + patching w/ ddp builtin * deprioritize/remove some of the more complex custom component examples Pull Request resolved: #402 Test Plan: ``` $ make -C docs coverage; and cat docs/build/0.1.2dev0/coverage/python.txt $ pyre $ pytest ``` CI ![Screenshot 2022-02-28 at 17-21-17 torchx workspace — PyTorch TorchX main documentation](https://user-images.githubusercontent.com/909104/156086755-8330fece-dc86-442d-a656-aaef775bd3e5.png) Reviewed By: kiukchung Differential Revision: D34540003 Pulled By: d4l3k fbshipit-source-id: a7987e35beb4566dbdc5aea8ce05aae930d694f4
1 parent ca86f97 commit 571fd78

29 files changed

+305
-89
lines changed

‎.github/workflows/doc-build.yaml‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,17 @@ jobs:
4242
run: |
4343
cd docs
4444
make papermill
45+
- name: Coverage
46+
run: |
47+
set -ex
48+
cd docs
49+
make coverage
50+
if [ "$(wc -l build/*/coverage/python.txt)" -ne 2 ]
51+
then
52+
cat build/*/coverage/python.txt
53+
echo "missing documentation coverage!"
54+
exit 1
55+
fi
4556
4657
docpush:
4758
runs-on: ubuntu-18.04

‎docs/source/app_best_practices.rst‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -111,7 +111,7 @@ model definition from a python file and then you'll load the weights and state
111111
dict from a ``.ckpt`` or ``.pt`` file.
112112

113113
This is how Pytorch Lightning's
114-
`ModelCheckpoint <https://pytorch-lightning.readthedocs.io/en/latest/extensions/generated/pytorch_lightning.callbacks.ModelCheckpoint.html>`__ hook works.
114+
`ModelCheckpoint <https://pytorch-lightning.readthedocs.io/en/latest/api/pytorch_lightning.callbacks.ModelCheckpoint.html>`__ hook works.
115115

116116
This is the most common but makes it harder to make a reusable app since your
117117
trainer app needs to include the model definition code.

‎docs/source/basics.rst‎

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -10,11 +10,12 @@ The top level modules in TorchX are:
1010

1111
1. :mod:`torchx.specs`: application spec (job definition) APIs
1212
2. :mod:`torchx.components`: predefined (builtin) app specs
13-
3. :mod:`torchx.runner`: given an app spec, submits the app as a job on a scheduler
14-
4. :mod:`torchx.schedulers`: backend job schedulers that the runner supports
15-
5. :mod:`torchx.pipelines`: adapters that convert the given app spec to a "stage" in an ML pipeline platform
16-
6. :mod:`torchx.runtime`: util and abstraction libraries you can use in authoring apps (not app spec)
17-
7. :mod:`torchx.cli`: CLI tool
13+
3. :mod:`torchx.workspace`: handles patching images for remote execution
14+
4. :mod:`torchx.cli`: CLI tool
15+
5. :mod:`torchx.runner`: given an app spec, submits the app as a job on a scheduler
16+
6. :mod:`torchx.schedulers`: backend job schedulers that the runner supports
17+
7. :mod:`torchx.pipelines`: adapters that convert the given app spec to a "stage" in an ML pipeline platform
18+
8. :mod:`torchx.runtime`: util and abstraction libraries you can use in authoring apps (not app spec)
1819

1920
Below is a UML diagram
2021

‎docs/source/components/utils.rst‎

Lines changed: 7 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -4,9 +4,10 @@ Utils
44
.. automodule:: torchx.components.utils
55
.. currentmodule:: torchx.components.utils
66

7-
.. autofunction:: torchx.components.utils.echo
8-
.. autofunction:: torchx.components.utils.touch
9-
.. autofunction:: torchx.components.utils.sh
10-
.. autofunction:: torchx.components.utils.copy
11-
.. autofunction:: torchx.components.utils.python
12-
.. autofunction:: torchx.components.utils.booth
7+
.. autofunction:: echo
8+
.. autofunction:: touch
9+
.. autofunction:: sh
10+
.. autofunction:: copy
11+
.. autofunction:: python
12+
.. autofunction:: booth
13+
.. autofunction:: binary

‎docs/source/conf.py‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -66,6 +66,12 @@
6666
"IPython.sphinxext.ipython_console_highlighting",
6767
]
6868

69+
# coverage options
70+
71+
coverage_ignore_modules = [
72+
"torchx.components.component_test_base",
73+
]
74+
6975
# katex options
7076
#
7177
#

‎docs/source/index.rst‎

Lines changed: 32 additions & 35 deletions
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@ most unique applications can be serviced without customizing the whole vertical
1616

1717

1818
**GETTING STARTED?** First learn the :ref:`basic concepts<basics:Basic Concepts>` and
19-
follow the :ref:`quickstart guide<quickstart:Quickstart>`.
19+
follow the :ref:`quickstart guide<quickstart:Quickstart - Custom Components>`.
2020

2121
.. image:: torchx_index_diag.png
2222

@@ -47,8 +47,37 @@ Documentation
4747
quickstart.md
4848
cli
4949

50+
runner.config
51+
5052
advanced
5153

54+
55+
Works With
56+
---------------
57+
58+
.. _Schedulers:
59+
.. toctree::
60+
:maxdepth: 1
61+
:caption: Schedulers
62+
63+
schedulers/local
64+
schedulers/docker
65+
schedulers/kubernetes
66+
schedulers/slurm
67+
schedulers/ray
68+
schedulers/aws_batch
69+
70+
.. _Pipelines:
71+
.. toctree::
72+
:maxdepth: 1
73+
:caption: Pipelines
74+
75+
pipelines/kfp
76+
77+
78+
Examples
79+
------------
80+
5281
.. toctree::
5382
:maxdepth: 1
5483
:caption: Examples
@@ -58,6 +87,7 @@ Documentation
5887
examples_pipelines/index
5988

6089

90+
6191
Components Library
6292
---------------------
6393
.. _Components:
@@ -85,28 +115,6 @@ Runtime Library
85115
runtime/tracking
86116

87117

88-
Works With
89-
---------------
90-
91-
.. _Schedulers:
92-
.. toctree::
93-
:maxdepth: 1
94-
:caption: Schedulers
95-
96-
schedulers/local
97-
schedulers/kubernetes
98-
schedulers/slurm
99-
schedulers/ray
100-
schedulers/aws_batch
101-
102-
.. _Pipelines:
103-
.. toctree::
104-
:maxdepth: 1
105-
:caption: Pipelines
106-
107-
pipelines/kfp
108-
109-
110118
Reference
111119
-----------
112120

@@ -118,6 +126,7 @@ Reference
118126
specs
119127
runner
120128
schedulers
129+
workspace
121130
pipelines
122131

123132
.. toctree::
@@ -126,15 +135,3 @@ Reference
126135

127136
app_best_practices
128137
component_best_practices
129-
130-
131-
Experimental
132-
---------------
133-
.. toctree::
134-
:maxdepth: 1
135-
:caption: Experimental Features
136-
137-
experimental/runner.config
138-
139-
140-

‎docs/source/quickstart.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@ jupyter:
1212
name: python3
1313
---
1414

15-
# Quickstart
15+
# Quickstart - Custom Components
1616

1717
This is a self contained guide on how to build a simple app and component spec
1818
and launch it via two different schedulers.

‎docs/source/experimental/runner.config.rst‎ renamed to ‎docs/source/runner.config.rst‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
(beta) .torchxconfig file
1+
.torchxconfig
22
-----------------------------
33

44
.. automodule:: torchx.runner.config
@@ -10,3 +10,7 @@ Config API Functions
1010
.. autofunction:: apply
1111
.. autofunction:: load
1212
.. autofunction:: dump
13+
.. autofunction:: find_configs
14+
.. autofunction:: get_configs
15+
.. autofunction:: get_config
16+
.. autofunction:: load_sections

‎docs/source/runtime/hpo.rst‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,4 +14,7 @@ Ax (Adaptive Experimentation)
1414
.. currentmodule:: torchx.runtime.hpo.ax
1515

1616
.. autoclass:: TorchXRunner
17+
:members:
18+
1719
.. autoclass:: AppMetric
20+
:members:

‎docs/source/schedulers/aws_batch.rst‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,17 @@ AWS Batch
22
=================
33

44
.. automodule:: torchx.schedulers.aws_batch_scheduler
5+
56
.. currentmodule:: torchx.schedulers.aws_batch_scheduler
67

78
.. autoclass:: AWSBatchScheduler
89
:members:
10+
:show-inheritance:
11+
12+
.. autoclass:: BatchJob
13+
:members:
14+
15+
Reference
16+
~~~~~~~~~~~~
17+
18+
.. autofunction:: create_scheduler

0 commit comments

Comments
 (0)