Skip to content

Commit 1f1a50f

Browse files
d4l3kfacebook-github-bot
d4l3k
authored and
facebook-github-bot
committed
docs: add new quickstart for job launching (#404)
Summary: This updates the README/index/quickstart to highlight the job launching as the main user entrypoint as per https://docs.google.com/document/d/14icG0GJG1nPHF3DW7yMreZpe3S8FwYdrI_XYIzDTSbI/edit Changes: * update README/index w/ new job launcher descriptions emphasized * move existing quickstart to `custom_compents.md` * update `utils.python` to have `--script` to match `dist.ddp` * update the pipelines/scheduler doc pages to list all child schedulers via glob * make pipeline doc structure match scheduler doc (piplines/*, schedulers/*) * docker_workspace * use `Dockerfile.torchx` to avoid unexpected collisions with `Dockerfile`. * docker_workspace pull images explicitly to avoid error from newer docker daemon during build * docker_scheduler: skip pulling `sha256:` images to avoid logspam Pull Request resolved: #404 Test Plan: ``` cd docs; make clean html pyre ``` CI ![Screenshot 2022-03-03 at 15-44-05 Quickstart — PyTorch TorchX main documentation](https://user-images.githubusercontent.com/909104/156671986-6bdc28bb-7a8d-471e-bf98-f2d1ae167465.png) Reviewed By: kiukchung Differential Revision: D34628895 Pulled By: d4l3k fbshipit-source-id: e5f5533a36c3e17f46b7c843b42ac921ac346f0d
1 parent 3240111 commit 1f1a50f

15 files changed

+393
-140
lines changed

‎README.md

+7-6
Original file line numberDiff line numberDiff line change
@@ -8,11 +8,16 @@
88
# TorchX
99

1010

11-
TorchX is a library containing standard DSLs for authoring and running PyTorch
12-
related components for an E2E production ML pipeline.
11+
TorchX is a universal job launcher for PyTorch applications.
12+
TorchX is designed to have fast iteration time for training/research and support
13+
for E2E production ML pipelines when you're ready.
1314

1415
For the latest documentation, please refer to our [website](https://pytorch.org/torchx).
1516

17+
## Quickstart
18+
19+
See the [quickstart guide](https://pytorch.org/torchx/latest/quickstart.html).
20+
1621

1722
## Requirements
1823
TorchX SDK (torchx):
@@ -58,10 +63,6 @@ $ pip install -e git+https://github.com/pytorch/torchx.git#egg=torchx
5863
$ pip install -e git+https://github.com/pytorch/torchx.git#egg=torchx[kubernetes]
5964
```
6065

61-
## Quickstart
62-
63-
See the [quickstart guide](https://pytorch.org/torchx/latest/quickstart.html).
64-
6566
## Contributing
6667

6768
We welcome PRs! See the [CONTRIBUTING](CONTRIBUTING.md) file.

‎docs/source/.gitignore

+3
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
.torchxconfig
2+
Dockerfile*
3+
*.py

‎docs/source/conf.py

+3-1
Original file line numberDiff line numberDiff line change
@@ -361,4 +361,6 @@ def handle_item(fieldarg, content):
361361
362362
<div id="is-nbsphinx"></div>
363363
"""
364-
# nbsphinx_execute = 'never'
364+
365+
if os.environ.get("SKIP_NB"):
366+
nbsphinx_execute = "never"

‎docs/source/custom_components.md

+149
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,149 @@
1+
---
2+
jupyter:
3+
jupytext:
4+
text_representation:
5+
extension: .md
6+
format_name: markdown
7+
format_version: '1.1'
8+
jupytext_version: 1.1.0
9+
kernelspec:
10+
display_name: Python 3
11+
language: python
12+
name: python3
13+
---
14+
15+
# Custom Components
16+
17+
This is a guide on how to build a simple app and custom component spec
18+
and launch it via two different schedulers.
19+
20+
See the [Quickstart Guide](quickstart.md) for installation and basic usage.
21+
22+
## Hello World
23+
24+
Lets start off with writing a simple "Hello World" python app. This is just a
25+
normal python program and can contain anything you'd like.
26+
27+
<div class="admonition note">
28+
<div class="admonition-title">Note</div>
29+
This example uses Jupyter Notebook `%%writefile` to create local files for
30+
example purposes. Under normal usage you would have these as standalone files.
31+
</div>
32+
33+
```python
34+
%%writefile my_app.py
35+
36+
import sys
37+
import argparse
38+
39+
def main(user: str) -> None:
40+
print(f"Hello, {user}!")
41+
42+
if __name__ == "__main__":
43+
parser = argparse.ArgumentParser(
44+
description="Hello world app"
45+
)
46+
parser.add_argument(
47+
"--user",
48+
type=str,
49+
help="the person to greet",
50+
required=True,
51+
)
52+
args = parser.parse_args(sys.argv[1:])
53+
54+
main(args.user)
55+
```
56+
57+
Now that we have an app we can write the component file for it. This
58+
function allows us to reuse and share our app in a user friendly way.
59+
60+
We can use this component from the `torchx` cli or programmatically as part of a
61+
pipeline.
62+
63+
```python
64+
%%writefile my_component.py
65+
66+
import torchx.specs as specs
67+
68+
def greet(user: str, image: str = "my_app:latest") -> specs.AppDef:
69+
return specs.AppDef(
70+
name="hello_world",
71+
roles=[
72+
specs.Role(
73+
name="greeter",
74+
image=image,
75+
entrypoint="python",
76+
args=[
77+
"-m", "my_app",
78+
"--user", user,
79+
],
80+
)
81+
],
82+
)
83+
```
84+
85+
We can execute our component via `torchx run`. The
86+
`local_cwd` scheduler executes the component relative to the current directory.
87+
88+
```sh
89+
torchx run --scheduler local_cwd my_component.py:greet --user "your name"
90+
```
91+
92+
If we want to run in other environments, we can build a Docker container so we
93+
can run our component in Docker enabled environments such as Kubernetes or via
94+
the local Docker scheduler.
95+
96+
<div class="admonition note">
97+
<div class="admonition-title">Note</div>
98+
This requires Docker installed and won't work in environments such as Google
99+
Colab. If you have not done so already follow the install instructions on:
100+
[https://docs.docker.com/get-docker/](https://docs.docker.com/get-docker/)</a>
101+
</div>
102+
103+
```python
104+
%%writefile Dockerfile.custom
105+
106+
FROM ghcr.io/pytorch/torchx:0.1.0rc1
107+
108+
ADD my_app.py .
109+
```
110+
111+
Once we have the Dockerfile created we can create our docker image.
112+
113+
```sh
114+
docker build -t my_app:latest -f Dockerfile.custom .
115+
```
116+
117+
We can then launch it on the local scheduler.
118+
119+
```sh
120+
torchx run --scheduler local_docker my_component.py:greet --image "my_app:latest" --user "your name"
121+
```
122+
123+
If you have a Kubernetes cluster you can use the [Kubernetes scheduler](schedulers/kubernetes.rst) to launch
124+
this on the cluster instead.
125+
126+
127+
<!-- #md -->
128+
```sh
129+
$ docker push my_app:latest
130+
$ torchx run --scheduler kubernetes my_component.py:greet --image "my_app:latest" --user "your name"
131+
```
132+
<!-- #endmd -->
133+
134+
135+
## Builtins
136+
137+
TorchX also provides a number of builtin components with premade images. You can discover
138+
them via:
139+
140+
```sh
141+
torchx builtins
142+
```
143+
144+
You can use these either from the CLI, from a pipeline or programmatically like
145+
you would any other component.
146+
147+
```sh
148+
torchx run utils.echo --msg "Hello :)"
149+
```

‎docs/source/index.rst

+21-26
Original file line numberDiff line numberDiff line change
@@ -3,37 +3,33 @@
33
TorchX
44
==================
55

6-
TorchX is an SDK for quickly building and deploying ML applications from R&D to production.
7-
It offers various builtin components that encode MLOps best practices and make advanced
8-
features like distributed training and hyperparameter optimization accessible to all.
9-
Users can get started with TorchX with no added setup cost since it supports popular
10-
ML schedulers and pipeline orchestrators that are already widely adopted and deployed
11-
in production.
6+
TorchX is a universal job launcher for PyTorch applications.
7+
TorchX is designed to have fast iteration time for training/research and support
8+
for E2E production ML pipelines when you're ready.
129

13-
No two production environments are the same. To comply with various use cases, TorchX's
14-
core APIs allow tons of customization at well-defined extension points so that even the
15-
most unique applications can be serviced without customizing the whole vertical stack.
10+
**GETTING STARTED?** Follow the :ref:`quickstart guide<quickstart:Quickstart>`.
1611

1712

18-
**GETTING STARTED?** First learn the :ref:`basic concepts<basics:Basic Concepts>` and
19-
follow the :ref:`quickstart guide<quickstart:Quickstart - Custom Components>`.
20-
21-
.. image:: torchx_index_diag.png
22-
2313
In 1-2-3
2414
-----------------
2515

26-
**01 DEFINE OR CHOOSE** Start by :ref:`writing a component<components/overview:Overview>` -- a python
27-
function that returns an AppDef object for your application. Or you can choose one of the
28-
:ref:`builtin components<Components>`.
16+
Step 1. Install
17+
18+
.. code-block:: shell
19+
20+
pip install torchx[dev]
21+
22+
Step 2. Run Locally
23+
24+
.. code-block:: shell
2925
30-
**02 RUN AS A JOB** Once you've defined or chosen a component, you can :ref:`run it<runner:torchx.runner>`
31-
by submitting it as a job in one of the supported :ref:`Schedulers<Schedulers>`. TorchX supports several
32-
popular ones, such as Kubernetes and SLURM out of the box.
26+
torchx run --scheduler local_cwd utils.python --script my_app.py "Hello, localhost!"
3327
34-
**03 CONVERT TO PIPELINE** In production, components are often run as a workflow (aka pipeline).
35-
TorchX components can be converted to pipeline stages by passing them through the :py:mod:`torchx.pipelines`
36-
adapter. :ref:`Pipelines<Pipelines>` lists the pipeline orchestrators supported out of the box.
28+
Step 3. Run Remotely
29+
30+
.. code-block:: shell
31+
32+
torchx run --scheduler kubernetes utils.python --script my_app.py "Hello, Kubernetes!"
3733
3834
3935
Documentation
@@ -43,13 +39,12 @@ Documentation
4339
:maxdepth: 1
4440
:caption: Usage
4541

46-
basics
4742
quickstart.md
4843
cli
49-
44+
basics
5045
runner.config
51-
5246
advanced
47+
custom_components.md
5348

5449

5550
Works With

‎docs/source/pipeline_kfp_diagram.png

-841 KB
Binary file not shown.

‎docs/source/pipelines.rst

+6-13
Original file line numberDiff line numberDiff line change
@@ -4,19 +4,12 @@ torchx.pipelines
44
.. automodule:: torchx.pipelines
55
.. currentmodule:: torchx.pipelines
66

7-
torchx.pipelines.kfp
8-
#####################
7+
All Pipelines
8+
~~~~~~~~~~~~~~~~
99

10-
.. image:: pipeline_kfp_diagram.png
10+
.. toctree::
11+
:maxdepth: 1
12+
:glob:
1113

12-
.. automodule:: torchx.pipelines.kfp
13-
.. currentmodule:: torchx.pipelines.kfp
14+
pipelines/*
1415

15-
.. currentmodule:: torchx.pipelines.kfp.adapter
16-
17-
.. autofunction:: container_from_app
18-
.. autofunction:: resource_from_app
19-
.. autofunction:: component_from_app
20-
.. autofunction:: component_spec_from_app
21-
22-
.. autoclass:: ContainerFactory

‎docs/source/pipelines/kfp.rst

+18-2
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,23 @@ Kubeflow Pipelines
22
======================
33

44
TorchX provides an adapter to run TorchX components as part of Kubeflow
5-
Pipelines. See :ref:`examples_pipelines/index:KubeFlow Pipelines Examples` and
6-
the :mod:`torchx.pipelines.kfp` for API reference.
5+
Pipelines. See :ref:`examples_pipelines/index:KubeFlow Pipelines Examples`.
76

87
.. image:: kfp_diagram.jpg
8+
9+
torchx.pipelines.kfp
10+
#####################
11+
12+
.. image:: pipeline_kfp_diagram.png
13+
14+
.. automodule:: torchx.pipelines.kfp
15+
.. currentmodule:: torchx.pipelines.kfp
16+
17+
.. currentmodule:: torchx.pipelines.kfp.adapter
18+
19+
.. autofunction:: container_from_app
20+
.. autofunction:: resource_from_app
21+
.. autofunction:: component_from_app
22+
.. autofunction:: component_spec_from_app
23+
24+
.. autoclass:: ContainerFactory

‎docs/source/pipelines/pipeline_kfp_diagram.png

Loading

0 commit comments

Comments
 (0)