iterative · casperdcl · Mar 15, 2022 · Mar 8, 2022 · Mar 8, 2022 · Mar 8, 2022
diff --git a/README.md b/README.md
@@ -1,48 +1,28 @@
-![Terraform Provider Iterative](https://static.iterative.ai/img/cml/banner-terraform.png)
+![TPI](https://static.iterative.ai/img/cml/banner-terraform.png)
 
-# Iterative Provider [![](https://img.shields.io/badge/-documentation-5c4ee5?logo=terraform)](https://registry.terraform.io/providers/iterative/iterative/latest/docs)
+# Terraform Provider Iterative
 
-The Iterative Provider is a Terraform plugin that enables full lifecycle
-management of computing resources for machine learning pipelines, including GPUs, from your favorite cloud vendors.
+[![docs](https://img.shields.io/badge/-docs-5c4ee5?logo=terraform)](https://registry.terraform.io/providers/iterative/iterative/latest/docs)
+[![tests](https://img.shields.io/github/workflow/status/iterative/terraform-provider-iterative/Test?label=tests&logo=GitHub)](https://github.com/iterative/terraform-provider-iterative/actions/workflows/test.yml)
 
-The Iterative Provider makes it easy to:
+- **Orchestrate Resources**: create cloud compute & storage resources without reading pages of documentation
+- **Sync & Execute**: move data & run code in the cloud with minimal configuration
+- **Low cost**: auto-recovery from spot/preemptible instances to vastly reduce cost
+- **No waste**: auto-cleanup unused resources
+- **No lock-in**: switch between cloud vendors with ease
 
-- Rapidly move local machine learning experiments to a cloud infrastructure
-- Take advantage of training models on spot instances without losing any progress
-- Unify configuration of various cloud compute providers
-- Automatically destroy unused cloud resources (compute instances are terminated on job completion/failure, and storage is removed when results are downloaded)
+Iterative's Provider is a [Terraform](https://terraform.io) plugin built with machine learning pipelines in mind. It enables full lifecycle management of computing resources (including GPUs) from several cloud vendors:
 
-The Iterative Provider can provision resources with the following cloud providers and orchestrators:
-
-- Amazon Web Services
+- Amazon Web Services (AWS)
 - Microsoft Azure
-- Google Cloud Platform
-- Kubernetes
-
-## Documentation
-
-See the [Getting Started](https://registry.terraform.io/providers/iterative/iterative/latest/docs/guides/getting-started) guide to learn how to use the Iterative Provider. More details on configuring and using the Iterative Provider are in the [documentation](https://registry.terraform.io/providers/iterative/iterative/latest/docs).
+- Google Cloud Platform (GCP)
+- Kubernetes (K8s)
 
-## Support
+With a minimal configuration unified across cloud vendors, the aim is to easily move local experiments to the cloud, transparently resume from interrupted low-cost spot instances, and avoid being charged for unused cloud resources (terminate compute instances upon job completion/failure, and remove storage upon download of results).
 
-Have a feature request or found a bug? Let us know via [GitHub issues](https://github.com/iterative/terraform-provider-iterative/issues). Have questions? Join our [community on Discord](https://discord.gg/bzA6uY7); we'll be happy to help you get started!
-
-## License
+## Usage
 
-Iterative Provider is released under the [Apache 2.0 License](https://github.com/iterative/terraform-provider-iterative/blob/master/LICENSE).
-
-## Development
-
-### Install Go 1.17+
-
-Refer to the [official documentation](https://golang.org/doc/install) for specific instructions.
-
-### Clone the repository
-
-```console
-git clone https://github.com/iterative/terraform-provider-iterative
-cd terraform-provider-iterative
-```
+See the [Getting Started](https://registry.terraform.io/providers/iterative/iterative/latest/docs/guides/getting-started) guide for a more detailed guide.
 
 ### Install the provider
 
@@ -61,10 +41,23 @@ terraform {
   required_providers { iterative = { source = "iterative/iterative" } }
 }
 provider "iterative" {}
-# ... other resource blocks ...
+resource "iterative_task" "example" {
+  cloud   = "aws" # or any of: gcp, az, k8s
+  machine = "m"   # medium, or any of: l, xl, m+k80, xl+v100, ...
+
+  storage {
+    workdir = "."
+    output  = "results"
+  }
+  script = <<-END
+    #!/bin/bash
+    mkdir results
+    echo "Hello World!" > results/greeting.txt
+  END
+}
 ```
 
-**Note:** to use your local build, specify `source = "github.com/iterative/iterative"` (`source = "iterative/iterative"` will download the latest stable release instead).
+See the [Documentation](https://registry.terraform.io/providers/iterative/iterative/latest/docs) for obtaining credentials for the chosen `cloud`, and the [Reference](https://registry.terraform.io/providers/iterative/iterative/latest/docs/resources/task) for the full list of options for `main.tf`.
 
 ### Initialize the provider
 
@@ -79,3 +72,32 @@ terraform init --upgrade
 ```console
 terraform apply
 ```
+
+## Help
+
+Have a feature request or found a bug? Let us know via [GitHub issues](https://github.com/iterative/terraform-provider-iterative/issues). Have questions? Join our [community on Discord](https://discord.gg/bzA6uY7); we'll be happy to help you get started!
+
+## Contributing
+
+Instead of using the latest stable release, a local copy of the repository must be used.
+
+### Install Go 1.17+
+
+Refer to the [official documentation](https://golang.org/doc/install) for specific instructions.
+
+### Clone the repository
+
+```console
+git clone https://github.com/iterative/terraform-provider-iterative
+cd terraform-provider-iterative
+```
+
+### Modify test file
+
+Specify `source = "github.com/iterative/iterative"` to use the local repository.
+
+**Note:** `source = "iterative/iterative"` will download the latest release instead.
+
+## License
+
+[Apache 2.0](https://github.com/iterative/terraform-provider-iterative/blob/master/LICENSE).
diff --git a/docs/guides/getting-started.md b/docs/guides/getting-started.md
@@ -13,25 +13,25 @@ To use the Iterative Provider you will need to:
 
 In the project root directory:
 
-1. Create a directory named `shared` to store input data and output artefacts.
-2. Create a file named `main.tf` with the following contents:
+Create a file named `main.tf` with the following contents:
 
 ```hcl
 terraform {
   required_providers { iterative = { source = "iterative/iterative" } }
 }
 provider "iterative" {}
-resource "iterative_task" "task" {
+resource "iterative_task" "example" {
   cloud   = "aws" # or any of: gcp, az, k8s
-  machine = "m"
+  machine = "m"   # medium, or any of: l, xl, m+k80, xl+v100, ...
 
-  workdir {
-    input = "${path.root}/shared"
-    output = "${path.root}/shared"
+  storage {
+    workdir = "."
+    output  = "results"
   }
   script = <<-END
     #!/bin/bash
-    echo "Hello World!" > greeting.txt
+    mkdir results
+    echo "Hello World!" > results/greeting.txt
   END
 }
 ```
@@ -45,8 +45,8 @@ The project layout should look similar to this:
 ```
 project/
 ├── main.tf
-└── shared/
-    └── ...
+└── results/
+    └── greeting.txt (created in the cloud and downloaded locally)
 ```
 
 ## Initializing Terraform
@@ -71,7 +71,7 @@ $ terraform apply
 This command will:
 
 1. Create all the required cloud resources.
-2. Upload the specified shared `input` working directory to the cloud.
+2. Upload the specified working directory (`workdir`) to the cloud.
 3. Launch the task `script`.
 
 ## Viewing Task Statuses
@@ -93,9 +93,9 @@ $ terraform destroy
 
 This command will:
 
-1. Download the specified shared working directory from the cloud.
+1. Download the specified output directory from the cloud.
 2. Delete all the cloud resources created by `terraform apply`.
 
 ## Viewing Task Results
 
-After running `terraform destroy`, the `shared` directory should contain a file named `greeting.txt` with the text `Hello, World!`
+After running `terraform destroy`, the `results` directory should contain a file named `greeting.txt` with the text `Hello, World!`
diff --git a/docs/index.md b/docs/index.md
@@ -1,6 +1,23 @@
 # Iterative Provider
 
-Use the Iterative Provider to launch resource-intensive tasks in popular cloud providers with a single Terraform file.
+![TPI](https://static.iterative.ai/img/cml/banner-terraform.png)
+
+[![tests](https://img.shields.io/github/workflow/status/iterative/terraform-provider-iterative/Test?label=tests&logo=GitHub)](https://github.com/iterative/terraform-provider-iterative/actions/workflows/test.yml)
+
+- **Orchestrate Resources**: create cloud compute & storage resources without reading pages of documentation
+- **Sync & Execute**: move data & run code in the cloud with minimal configuration
+- **Low cost**: auto-recovery from spot/preemptible instances to vastly reduce cost
+- **No waste**: auto-cleanup unused resources
+- **No lock-in**: switch between cloud vendors with ease
+
+Iterative's Provider is a [Terraform](https://terraform.io) plugin built with machine learning pipelines in mind. It enables full lifecycle management of computing resources (including GPUs) from several cloud vendors:
+
+- Amazon Web Services (AWS)
+- Microsoft Azure
+- Google Cloud Platform (GCP)
+- Kubernetes (K8s)
+
+With a minimal configuration unified across cloud vendors, the aim is to easily move local experiments to the cloud, transparently resume from interrupted low-cost spot instances, and avoid being charged for unused cloud resources (terminate compute instances upon job completion/failure, and remove storage upon download of results).
 
 ## Example Usage
 
@@ -9,12 +26,18 @@ terraform {
   required_providers { iterative = { source = "iterative/iterative" } }
 }
 provider "iterative" {}
-resource "iterative_task" "task" {
-  cloud = "aws"
-
+resource "iterative_task" "example" {
+  cloud   = "aws" # or any of: gcp, az, k8s
+  machine = "m"   # medium, or any of: l, xl, m+k80, xl+v100, ...
+
+  storage {
+    workdir = "."
+    output  = "results"
+  }
   script = <<-END
     #!/bin/bash
-    echo "hello!"
+    mkdir results
+    echo "Hello World!" > results/greeting.txt
   END
 }
 ```

diff --git a/docs/resources/task.md b/docs/resources/task.md
@@ -3,25 +3,34 @@
 This resource will:
 
 1. Create cloud resources (machines and storage) for the task.
-2. Upload the given `workdir.input` to the cloud storage.
+2. Upload the given `storage.workdir` to the cloud storage.
 3. Run the given `script` on the cloud machine until completion or `timeout`.
-4. Download results to the given `workdir.output`.
+4. Download results to the given `storage.output`.
 
 ## Example Usage
 
 ```hcl
-resource "iterative_task" "task" {
-  cloud = "aws"
+resource "iterative_task" "example" {
+  name        = "example"
+  cloud       = "aws"
+  machine     = "m"       # medium, or any of: l, xl, m+k80, xl+v100, ...
+  image       = "ubuntu"
+  region      = "us-east"
+  disk_size   = 30        # GB
+  spot        = 0         # auto-price
+  parallelism = 1
+  timeout     = 3600      # max 1h idle
 
   environment = { GREETING = "Hello, world!" }
-  workdir {
-    input  = "${path.root}/shared"
-    output = "${path.root}/results"
+  storage {
+    workdir = "."
+    output  = "results"
   }
   script = <<-END
     #!/bin/bash
-    echo "$GREETING" | tee $(uuidgen)
+    echo "$GREETING" | tee results/$(uuidgen)
   END
+  # or: script = file("example.sh")
 }
 ```
 
@@ -30,22 +39,24 @@ resource "iterative_task" "task" {
 ### Required
 
 - `cloud` - (Required) Cloud provider to run the task on; valid values are `aws`, `gcp`, `az` and `k8s`.
-- `script` - (Required) Script to run (relative to `workdir.input`); must begin with a valid [shebang](<https://en.wikipedia.org/wiki/Shebang_(Unix)>). Can use a string, including a [heredoc](https://www.terraform.io/docs/language/expressions/strings.html#heredoc-strings), or the contents of a file returned by the [`file`](https://www.terraform.io/docs/language/functions/file.html) function.
+- `script` - (Required) Script to run (relative to `storage.workdir`); must begin with a valid [shebang](<https://en.wikipedia.org/wiki/Shebang_(Unix)>). Can use a string, including a [heredoc](https://www.terraform.io/docs/language/expressions/strings.html#heredoc-strings), or the contents of a file returned by the [`file`](https://www.terraform.io/docs/language/functions/file.html) function.
 
 ### Optional
 
 - `name` - (Optional) Deterministic task name.
 - `region` - (Optional) [Cloud region/zone](#cloud-regions) to run the task on.
 - `machine` - (Optional) See [Machine Types](#machine-types) below.
-- `disk_size` - (Optional) Size of the ephemeral machine storage.
+- `disk_size` - (Optional) Size of the ephemeral machine storage in GB.
 - `spot` - (Optional) Spot instance price. `-1`: disabled, `0`: automatic price, any other positive number: fixed price.
 - `image` - (Optional) [Machine image](#machine-images) to run the task with.
 - `parallelism` - (Optional) Number of machines to be launched in parallel.
-- `workdir.input` - (Optional) Local working directory to upload.
-- `workdir.output` - (Optional) Local directory to download results to (default: no download).
+- `storage.workdir` - (Optional) Local working directory to upload and use as the `script` working directory.
+- `storage.output` - (Optional) Results directory (**relative to `workdir`**) to download (default: no download).
 - `environment` - (Optional) Map of environment variable names and values for the task script. Empty string values are replaced with local environment values. Empty values may also be combined with a [glob](<https://en.wikipedia.org/wiki/Glob_(programming)>) name to import all matching variables.
 - `timeout` - (Optional) Maximum number of seconds to run before termination.
 
+~> **Note:** `output` is relative to `workdir`, so `storage { workdir = "foo", output = "bar" }` means "upload `./foo/`, change working directory to the uploaded folder, run `script`, and download `bar` (i.e. `./foo/bar`)".
+
 ## Attribute Reference
 
 In addition to all arguments above, the following attributes are exported:
@@ -209,7 +220,7 @@ Setting the `region` attribute results in undefined behaviour.
 
 #### Directory storage
 
-Unlike public cloud providers, Kubernetes does not offer any portable way of persisting and sharing storage between pods. When specified, the `workdir.input` attribute will create a `PersistentVolumeClaim` of the default `StorageClass`, with the same lifecycle as the task and the specified `disk_size`.
+Unlike public cloud providers, Kubernetes does not offer any portable way of persisting and sharing storage between pods. When specified, the `storage.workdir` attribute will create a `PersistentVolumeClaim` of the default `StorageClass`, with the same lifecycle as the task and the specified `disk_size`.
 
 ~> **Warning:** Access mode will be `ReadWriteOnce` if `parallelism=1` or `ReadWriteMany` otherwise.