Destroy AWS EFS volumes #5092
Destroy AWS EFS volumes #5092
Conversation
Destroy AWS EFS volumes that have owner tag when destroying a cluster. Destroy all access points and mount targets used by the volume, even when they are not tagged. AWS EFS CSI driver is an optional OCP component (installed via OLM). Right now, an EFS must be created manually by user. Since VPC and subnets cannot be deleted if they are used by an EFS volume, allow user to tag the EFS volume so cluster destruction deletes EFS and thus allows deletion of VPC + subnets.
|
/approve |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: patrickdillon The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci
bot
added
the
approved
|
|
||
| _, err = client.DeleteFileSystemWithContext(ctx, &efs.DeleteFileSystemInput{FileSystemId: aws.String(fsid)}) | ||
| if err != nil { | ||
| return err |
There was a problem hiding this comment.
Is there a case here where the error is that the file system doesn't exist (as it may have been previously deleted)? If you look elsewhere in the code, we handle this differently from other errors.
|
I added checks for various NotFound errors and tested in a real cluster (with some hacks around to use invalid IDs) |
|
/lgtm |
|
/retest |
|
@jsafrane: The following tests failed, say
Full PR test history. Your PR dashboard. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here. |
Like part of 4d66658 (Use 4.8 cli/installer in CI-related content, 2021-04-19, openshift#17834). Among other fixes, this should pull in deletion of AWS EFS volumes [1], which is occasionally causing leaked resources like [2]: $ curl -s https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/logs/periodic-ipi-deprovision-aws/1478982622368501760/artifacts/deprovision/ci-ln-lwnyy5b-/.openshift_install.log | tail -6 time="2022-01-06T07:52:51Z" level=debug msg="unrecognized ARN service elasticfilesystem (arn:aws:elasticfilesystem:us-west-2:460538899914:access-point/fsap-012a52e782c7ed0ac)" arn="arn:aws:elasticfilesystem:us-west-2:460538899914:access-point/fsap-012a52e782c7ed0ac" time="2022-01-06T07:52:51Z" level=debug msg="DependencyViolation: The subnet 'subnet-02146773c4035ad4e' has dependencies and cannot be deleted.\n\tstatus code: 400, request id: 3c014d42-3538-4422-86f5-d6d740e15669" arn="arn:aws:ec2:us-west-2:460538899914:subnet/subnet-02146773c4035ad4e" time="2022-01-06T07:52:51Z" level=debug msg="search for matching resources by tag in us-west-2 matching aws.Filter{\"kubernetes.io/cluster/ci-ln-lwnyy5b-76ef8-mr89d\":\"owned\"}" time="2022-01-06T07:52:52Z" level=debug msg="search for IAM roles" time="2022-01-06T07:52:52Z" level=debug msg="search for IAM users" time="2022-01-06T07:52:52Z" level=debug msg="search for IAM instance profiles" [1]: openshift/installer#5092 [2]: https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ipi-deprovision-aws/1478982622368501760
Destroy AWS EFS volumes that have owner tag when destroying a cluster. Recursively destroy all access points and mount targets used by the volume, even when they are not tagged, otherwise EFS cannot be deleted. Not sure what's common practice for recursive deletion on
cluster destroy- access points could be tagged (and thus we may require users to tag them), but mount targets do not have any tags.AWS EFS CSI driver is an optional OCP component (installed via OLM). Right now, an EFS must be created manually by user. Since VPC and subnets cannot be deleted if they are used by an EFS volume, allow user to tag the EFS volume so cluster destruction deletes EFS, VPC + subnets (with two iterations to get the ordering right...)
KEP for whole AWS EFS support: openshift/enhancements#687