Support reusing existing storage containers across task providers #687

tasdomas · 2022-10-05T14:18:37Z

This closes #299.

The storage type is inferred by provider.

The storage config section is extended with additional options:

 storage {
    container = <container-id>
    container_path = <subdirectory in container>
    container_opts = {
    }
  }

* Update config schema. * AWS support for existing S3 buckets. * Existing bucket support for gcp. * Add mocks and tests to existing bucket resource in aws. * Update docs with new pre-allocated container fields.

* AWS support for existing S3 buckets. * Existing bucket support for gcp. * Fix subdirectory in rclone remote. * Blob container generates the rclone connection string. * Introduce a type for generating rclone connection strings. * Azure support for reusable blob containers. * Update docs. * Fix path prefix. * Initialize s3 existing bucket with RemoteStorage struct. * Update gcp and aws existing bucket data sources to align with the azure data source. Use common.RemoteStorage to initialize the data sources. Using rclone to verify storage during Read. Remove aws s3 client mocks and tests that rely on them. * Fix comment.

* K8s support for specifying an existing persistent volume claim. Co-authored-by: Helio Machado <0x2b3bfa0+git@googlemail.com>

iterative/resource_task.go

tasdomas · 2022-10-19T12:14:55Z

How does leo list work when using this feature? Worth shifting from storage enumeration to e.g. compute enumeration and creating the latter first?

leo list will need to be updated. But I think we should rethink the approach instead of switching to listing compute instances. Unless I'm mistaken, resource groups are supported by all major cloud providers, we could use (and list) those.

0x2b3bfa0 · 2022-10-19T12:21:05Z

Unless I'm mistaken, resource groups are supported by all major cloud providers

It depends on your definition of resource group.¹ 😅

Tangentially related to https://github.com/iterative/terraform-provider-iterative/issues/289#issue-1062601339 ↩

0x2b3bfa0 · 2022-10-23T03:06:06Z

@tasdomas, would you mind opening an issue to track #687 (comment)?

tasdomas · 2022-10-24T13:58:55Z

Unless I'm mistaken, resource groups are supported by all major cloud providers

It depends on your definition of resource group.1 sweat_smile

Footnotes
1. Tangentially related to [Allow tasks to destroy themselves #289 (comment)](https://github.com/iterative/terraform-provider-iterative/issues/289#issue-1062601339) [leftwards_arrow_with_hook](#user-content-fnref-1-0d2c7bd068cff5611d0349f0c7a4d25e)

Done. Thanks for reminding me!

casperdcl · 2022-10-28T22:30:48Z

docs/resources/task.md

+### Generic
+
 A set of "permissions" assigned to the `task` instance, format depends on the cloud provider

-#### Amazon Web Services
+### Cloud-specific
+
+#### Kubernetes
+
+The name of a service account in the current namespace.
+
+### Amazon Web Services

 An [instance profile `arn`](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html), e.g.:
 `permission_set = "arn:aws:iam:1234567890:instance-profile/rolename"`

-#### Google Cloud Platform
+### Google Cloud Platform

 A service account email and a [list of scopes](https://cloud.google.com/sdk/gcloud/reference/alpha/compute/instances/set-scopes#--scopes), e.g.:
 `permission_set = "sa-name@project_id.iam.gserviceaccount.com,scopes=storage-rw"`

-#### Microsoft Azure
+### Microsoft Azure


I don't get this title reworking. There are no generic options for permission_set, so there shouldn't be a ### Generic section?

casperdcl · 2022-10-28T22:33:02Z

docs/resources/task.md

 A comma-separated list of [user-assigned identity](https://docs.microsoft.com/en-us/azure/active-directory/managed-identities-azure-resources/overview) ARM resource ids, e.g.:
 `permission_set = "/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.ManagedIdentity/userAssignedIdentities/{identityName}"`

+## Pre-allocated blob container
+
+### Generic


again, would remove "### Generic" here since there are no generic options for container, right? (for example, machine="m" is generic, machine="t2.micro" is not).

casperdcl · 2022-10-28T22:35:38Z

docs/resources/task.md

@@ -65,6 +65,8 @@ resource "iterative_task" "example" {
 - `storage.workdir` - (Optional) Local working directory to upload and use as the `script` working directory.
 - `storage.output` - (Optional) Results directory (**relative to `workdir`**) to download (default: no download).
 - `storage.exclude` - (Optional) List of files and globs to exclude from transfering. Excluded files are neither uploaded to cloud storage nor downloaded from it. Exclusions are defined relative to `storage.workdir`.
+- `storage.container` - (Optional) Pre-allocated container to use for storage of task data, results and status.


slightly wary of clashing with "docker containers" but willing to accept owing to "storage" namespace.

casperdcl · 2022-10-28T22:36:47Z

docs/resources/task.md

+- `storage.container` - (Optional) Pre-allocated container to use for storage of task data, results and status.
+- `storage.container_opts` - (Optional) Block of cloud-specific container settings.


what about storage.container_(options|conf(ig)|settings)?

tasdomas and others added 5 commits September 8, 2022 10:30

Support for existing storage containers in aws and gcp. (#651)

a731724

* Update config schema. * AWS support for existing S3 buckets. * Existing bucket support for gcp. * Add mocks and tests to existing bucket resource in aws. * Update docs with new pre-allocated container fields.

K8s support for specifying an existing persistent volume claim (#661)

ce0b92e

* K8s support for specifying an existing persistent volume claim. Co-authored-by: Helio Machado <0x2b3bfa0+git@googlemail.com>

Merge remote-tracking branch 'origin/master' into feature-bucket-reuse

4e4166a

Update use of Identifier struct.

fb271e5

tasdomas temporarily deployed to automatic October 5, 2022 14:18 Inactive

0x2b3bfa0 self-requested a review October 5, 2022 14:46

0x2b3bfa0 changed the title ~~Support reusing existing storage containers across task providers.~~ Support reusing existing storage containers across task providers Oct 6, 2022

Merge branch 'master' into feature-bucket-reuse

610374d

0x2b3bfa0 temporarily deployed to automatic October 10, 2022 05:40 Inactive

0x2b3bfa0 reviewed Oct 10, 2022

View reviewed changes

iterative/resource_task.go Outdated Show resolved Hide resolved

0x2b3bfa0 assigned tasdomas Oct 10, 2022

0x2b3bfa0 added enhancement New feature or request storage cloud-common Applies to every cloud vendor labels Oct 10, 2022

tasdomas added 2 commits October 12, 2022 14:52

Combine container and container_path config keys.

158568b

Update docs.

5163911

tasdomas temporarily deployed to automatic October 12, 2022 12:11 Inactive

tasdomas temporarily deployed to automatic October 18, 2022 16:43 Inactive

Use split function from rclone.

670f214

tasdomas force-pushed the feature-bucket-reuse branch from b900f4b to 670f214 Compare October 19, 2022 07:58

tasdomas temporarily deployed to automatic October 19, 2022 07:58 Inactive

tasdomas requested a review from 0x2b3bfa0 October 20, 2022 12:19

Merge remote-tracking branch 'origin/master' into feature-bucket-reuse

69da848

tasdomas temporarily deployed to automatic October 21, 2022 08:40 Inactive

tasdomas requested a review from a team October 21, 2022 08:40

tasdomas temporarily deployed to automatic October 21, 2022 08:40 Inactive

tasdomas merged commit 110a740 into master Oct 21, 2022

tasdomas deleted the feature-bucket-reuse branch October 21, 2022 10:03

tasdomas mentioned this pull request Oct 24, 2022

leo: list won't list tasks that use pre-allocated storage #702

Open

casperdcl reviewed Oct 28, 2022

View reviewed changes

0x2b3bfa0 mentioned this pull request Aug 24, 2023

Unexpected folder naming when providing s3 bucket as storage container #756

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support reusing existing storage containers across task providers #687

Support reusing existing storage containers across task providers #687

tasdomas commented Oct 5, 2022 •

edited

Loading

tasdomas commented Oct 19, 2022 •

edited

Loading

0x2b3bfa0 commented Oct 19, 2022

0x2b3bfa0 commented Oct 23, 2022

tasdomas commented Oct 24, 2022

Footnotes

casperdcl Oct 28, 2022

casperdcl Oct 28, 2022

casperdcl Oct 28, 2022

casperdcl Oct 28, 2022

		- `storage.container` - (Optional) Pre-allocated container to use for storage of task data, results and status.
		- `storage.container_opts` - (Optional) Block of cloud-specific container settings.

Support reusing existing storage containers across task providers #687

Support reusing existing storage containers across task providers #687

Conversation

tasdomas commented Oct 5, 2022 • edited Loading

tasdomas commented Oct 19, 2022 • edited Loading

0x2b3bfa0 commented Oct 19, 2022

Footnotes

0x2b3bfa0 commented Oct 23, 2022

tasdomas commented Oct 24, 2022

Footnotes

casperdcl Oct 28, 2022

Choose a reason for hiding this comment

casperdcl Oct 28, 2022

Choose a reason for hiding this comment

casperdcl Oct 28, 2022

Choose a reason for hiding this comment

casperdcl Oct 28, 2022

Choose a reason for hiding this comment

tasdomas commented Oct 5, 2022 •

edited

Loading

tasdomas commented Oct 19, 2022 •

edited

Loading