-
Notifications
You must be signed in to change notification settings - Fork 123
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
**Description of your changes:** Moves current s3 kustomize files into s3-static folder, andmakes changes necessary for irsa in s3. Installation script changes coming in separate pr **Testing:** - [ ] Unit tests pass - [ ] e2e tests pass - Details about new tests (If this PR adds a new feature) - Details about any manual tests performed By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
- Loading branch information
1 parent
e33b90e
commit 9fcd434
Showing
17 changed files
with
411 additions
and
18 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,14 @@ | ||
| apiVersion: kustomize.config.k8s.io/v1beta1 | ||
| kind: Kustomization | ||
| namespace: kubeflow | ||
| bases: | ||
| - ../pipeline/ | ||
| configMapGenerator: | ||
| - name: workflow-controller-configmap | ||
| behavior: replace | ||
| files: | ||
| - ./s3/config | ||
| generatorOptions: | ||
| disableNameSuffixHash: true | ||
| patchesStrategicMerge: | ||
| - ./s3/disable-default-secret.yaml |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,20 @@ | ||
| { | ||
| artifactRepository: | ||
| { | ||
| s3: { | ||
| bucket: $(kfp-artifact-bucket-name), | ||
| keyPrefix: artifacts, | ||
| endpoint: $(kfp-artifact-storage-endpoint), | ||
| insecure: false, | ||
| accessKeySecret: { | ||
| name: mlpipeline-minio-artifact, | ||
| key: accesskey | ||
| }, | ||
| secretKeySecret: { | ||
| name: mlpipeline-minio-artifact, | ||
| key: secretkey | ||
| } | ||
| }, | ||
| archiveLogs: true | ||
| } | ||
| } |
6 changes: 6 additions & 0 deletions
6
awsconfigs/apps/pipeline-static/s3/disable-default-secret.yaml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| apiVersion: v1 | ||
| kind: Secret | ||
| metadata: | ||
| name: mlpipeline-minio-artifact | ||
| $patch: delete | ||
|
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,14 @@ | ||
| apiVersion: kustomize.config.k8s.io/v1beta1 | ||
| kind: Kustomization | ||
| namespace: kubeflow | ||
| bases: | ||
| - ../../pipeline/s3/ | ||
| configMapGenerator: | ||
| - name: workflow-controller-configmap | ||
| behavior: replace | ||
| files: | ||
| - config | ||
| generatorOptions: | ||
| disableNameSuffixHash: true | ||
| patchesStrategicMerge: | ||
| - disable-default-secret.yaml |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,5 +1,10 @@ | ||
| apiVersion: v1 | ||
| kind: Secret | ||
| metadata: | ||
| labels: | ||
| application-crd-id: kubeflow-pipelines | ||
| name: mlpipeline-minio-artifact | ||
| $patch: delete | ||
| namespace: kubeflow | ||
| stringData: | ||
| accesskey: "" | ||
| secretkey: "" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
| apiVersion: v1 | ||
| kind: ServiceAccount | ||
| metadata: | ||
| name: ml-pipeline | ||
| namespace: kubeflow | ||
| annotations: | ||
| eks.amazonaws.com/role-arn: '{{ .Values.irsa.roleName }}' |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Empty file.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,54 @@ | ||
| # Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. | ||
| # SPDX-License-Identifier: Apache-2.0 | ||
|
|
||
| import logging | ||
| import json | ||
| import boto3 | ||
|
|
||
| from e2e.utils.utils import ( | ||
| load_json_file, | ||
| get_iam_client, | ||
| get_eks_client, | ||
| ) | ||
| from e2e.fixtures.cluster import ( | ||
| associate_iam_oidc_provider, | ||
| ) | ||
| from e2e.utils.aws.iam import IAMPolicy | ||
| from e2e.utils.pipelines import common | ||
| from e2e.utils.config import configure_env_file | ||
| from e2e.utils.utils import print_banner, load_yaml_file | ||
|
|
||
|
|
||
| logging.basicConfig(level=logging.INFO) | ||
| logger = logging.getLogger(__name__) | ||
|
|
||
|
|
||
| def get_account_id(): | ||
| return boto3.client("sts").get_caller_identity().get("Account") | ||
|
|
||
| def delete_iam_role(role_name, region): | ||
| iam_client = get_iam_client(region=region) | ||
| try: | ||
| iam_client.detach_role_policy( | ||
| RoleName=role_name, PolicyArn="arn:aws:iam::aws:policy/AmazonS3FullAccess" | ||
| ) | ||
| except: | ||
| logger.log("Failed to detach role policy, it may not exist anymore.") | ||
|
|
||
| iam_client.delete_role(RoleName=role_name) | ||
| print(f"Deleted IAM Role : {role_name}") | ||
|
|
||
|
|
||
| if __name__ == "__main__": | ||
| print_banner("Reading Config") | ||
| config_file_path = common.CONFIG_FILE_PATH | ||
| cfg = load_yaml_file(file_path=config_file_path) | ||
| cluster_region = cfg["cluster"]["region"] | ||
| cluster_name = cfg["cluster"]["name"] | ||
|
|
||
| print_banner("Deleting all resources created for Pipeline IRSA") | ||
| role_name = f"{common.PIPELINE_OIDC_ROLE_NAME_PREFIX}-{cluster_name}" | ||
| delete_iam_role(role_name, cluster_region) | ||
|
|
||
| print_banner("CLEANUP SUCCESSFUL") | ||
|
|
Oops, something went wrong.