Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

service/s3: Add sweepers #7398

Merged
merged 1 commit into from
Jan 31, 2019
Merged

service/s3: Add sweepers #7398

merged 1 commit into from
Jan 31, 2019

Conversation

bflad
Copy link
Contributor

@bflad bflad commented Jan 31, 2019

To prevent TooManyBuckets errors from failed acceptance testing.

Output from test sweepers:

$ go test ./aws -v -sweep=us-west-2 -sweep-run=aws_s3_bucket -timeout 10h
2019/01/31 11:28:57 [DEBUG] Running Sweepers for region (us-west-2):
...
2019/01/31 11:28:59 [INFO] Skipping S3 Bucket: 187416307283-awsmacietrail-dataevent
2019/01/31 11:28:59 [INFO] Skipping S3 Bucket: aws-athena-query-results-187416307283-us-east-1
...
2019/01/31 11:28:59 [INFO] Skipping S3 Bucket (tf-acc-test-6417188599110687269destination) Objects: BucketRegionError: incorrect region, the bucket is not in 'us-west-2' region
  status code: 301, request id: , host id:
2019/01/31 11:28:59 [INFO] Skipping S3 Bucket: tf-alex-preview
2019/01/31 11:28:59 [INFO] Skipping S3 Bucket: tf-alex-preview2
2019/01/31 11:28:59 [INFO] Skipping S3 Bucket: tf-deploy-alex
2019/01/31 11:28:59 [INFO] Skipping S3 Bucket: tf-emr-bootstrap-5850549931442788494
2019/01/31 11:28:59 [INFO] Skipping S3 Bucket: tf-emr-bootstrap-8212862313789127315
2019/01/31 11:28:59 [INFO] Skipping S3 Bucket: tf-io-dev
2019/01/31 11:28:59 [INFO] Skipping S3 Bucket: tf-k8s-acc-test
2019/01/31 11:28:59 [INFO] Skipping S3 Bucket: tf-macie-test-bucket-921738264640544542
2019/01/31 11:29:00 [INFO] Deleting S3 Bucket (tf-object-test-bucket-272994333124655711) Object (tf-testing-obj-272994333124655711-all-params) Delete Marker: YECgZ632K8z.xlufooF.aeFvafBA6kuz
2019/01/31 11:29:02 [INFO] Deleting S3 Bucket (tf-object-test-bucket-3609874408477041358) Object (tf-testing-obj-3609874408477041358-all-params) Delete Marker: hQ4eUTYHGqhZ6lV9ATtCbu2F3HNuqgwa
2019/01/31 11:29:03 [INFO] Deleting S3 Bucket (tf-object-test-bucket-3660475575348874992) Object (updateable-key) Delete Marker: JxNsmwLQGbCqmtpMgt4n.d60NeCtC3V_
2019/01/31 11:29:04 [INFO] Deleting S3 Bucket (tf-object-test-bucket-3900350678782068183) Object (test-key) Delete Marker: auhlvd6dgHaEc4jv2sfi1yUKM_lkvh..
2019/01/31 11:29:05 [INFO] Deleting S3 Bucket (tf-object-test-bucket-4138169415932139850) Object (updateable-key) Delete Marker: nA74rE0HYETVIrXvq2zSu7Z4AQGc7jR9
2019/01/31 11:29:07 [INFO] Deleting S3 Bucket (tf-object-test-bucket-6391031696293038021) Object (updateable-key) Delete Marker: Cc0CRHZZ6P5Z3RIGSolfZk1KfNqs4n1l
2019/01/31 11:29:08 [INFO] Skipping S3 Bucket: tf-redshift-logging-5085425770880693968
2019/01/31 11:29:08 [INFO] Skipping S3 Bucket: tf-redshift-logging-9209707503695989768
2019/01/31 11:29:08 [INFO] Skipping S3 Bucket: tf-spot-datafeed-3152131621798213916
2019/01/31 11:29:08 [INFO] Skipping S3 Bucket (tf-test-bucket-1173093638223895558) Objects: BucketRegionError: incorrect region, the bucket is not in 'us-west-2' region
  status code: 301, request id: , host id:
2019/01/31 11:29:09 [INFO] Skipping S3 Bucket (tf-test-bucket-6879147095700184482) Objects: BucketRegionError: incorrect region, the bucket is not in 'us-west-2' region
  status code: 301, request id: , host id:
2019/01/31 11:29:09 [DEBUG] Sweeper (aws_s3_bucket) has dependency (aws_s3_bucket_object), running..
2019/01/31 11:29:09 [DEBUG] Sweeper (aws_s3_bucket_object) already ran in region (us-west-2)
...
2019/01/31 11:29:09 [INFO] Skipping S3 Bucket: 187416307283-awsmacietrail-dataevent
2019/01/31 11:29:09 [INFO] Skipping S3 Bucket: aws-athena-query-results-187416307283-us-east-1
...
2019/01/31 11:29:09 [INFO] Deleting S3 Bucket: tf-acc-test-6417188599110687269destination
2019/01/31 11:29:09 [DEBUG] Waiting for state to become: [success]
2019/01/31 11:29:10 [INFO] Skipping S3 Bucket (tf-acc-test-6417188599110687269destination): BucketRegionError: incorrect region, the bucket is not in 'us-west-2' region
  status code: 301, request id: , host id:
2019/01/31 11:29:10 [INFO] Skipping S3 Bucket: tf-alex-preview
2019/01/31 11:29:10 [INFO] Skipping S3 Bucket: tf-alex-preview2
2019/01/31 11:29:10 [INFO] Skipping S3 Bucket: tf-deploy-alex
2019/01/31 11:29:10 [INFO] Skipping S3 Bucket: tf-emr-bootstrap-5850549931442788494
2019/01/31 11:29:10 [INFO] Skipping S3 Bucket: tf-emr-bootstrap-8212862313789127315
2019/01/31 11:29:10 [INFO] Skipping S3 Bucket: tf-io-dev
2019/01/31 11:29:10 [INFO] Skipping S3 Bucket: tf-k8s-acc-test
2019/01/31 11:29:10 [INFO] Skipping S3 Bucket: tf-macie-test-bucket-921738264640544542
2019/01/31 11:29:10 [INFO] Deleting S3 Bucket: tf-object-test-bucket-272994333124655711
2019/01/31 11:29:10 [DEBUG] Waiting for state to become: [success]
2019/01/31 11:29:10 [INFO] Deleting S3 Bucket: tf-object-test-bucket-3609874408477041358
2019/01/31 11:29:10 [DEBUG] Waiting for state to become: [success]
2019/01/31 11:29:11 [INFO] Deleting S3 Bucket: tf-object-test-bucket-3660475575348874992
2019/01/31 11:29:11 [DEBUG] Waiting for state to become: [success]
2019/01/31 11:29:11 [INFO] Deleting S3 Bucket: tf-object-test-bucket-3900350678782068183
2019/01/31 11:29:11 [DEBUG] Waiting for state to become: [success]
2019/01/31 11:29:12 [INFO] Deleting S3 Bucket: tf-object-test-bucket-4138169415932139850
2019/01/31 11:29:12 [DEBUG] Waiting for state to become: [success]
2019/01/31 11:29:13 [INFO] Deleting S3 Bucket: tf-object-test-bucket-6391031696293038021
2019/01/31 11:29:13 [DEBUG] Waiting for state to become: [success]
2019/01/31 11:29:13 [INFO] Deleting S3 Bucket: tf-object-test-bucket-7215878632290092620
2019/01/31 11:29:13 [DEBUG] Waiting for state to become: [success]
2019/01/31 11:29:14 [INFO] Skipping S3 Bucket: tf-redshift-logging-5085425770880693968
2019/01/31 11:29:14 [INFO] Skipping S3 Bucket: tf-redshift-logging-9209707503695989768
2019/01/31 11:29:14 [INFO] Skipping S3 Bucket: tf-spot-datafeed-3152131621798213916
2019/01/31 11:29:14 [INFO] Deleting S3 Bucket: tf-test-bucket-1173093638223895558
2019/01/31 11:29:14 [DEBUG] Waiting for state to become: [success]
2019/01/31 11:29:14 [INFO] Skipping S3 Bucket (tf-test-bucket-1173093638223895558): BucketRegionError: incorrect region, the bucket is not in 'us-west-2' region
  status code: 301, request id: , host id:
2019/01/31 11:29:14 [INFO] Deleting S3 Bucket: tf-test-bucket-6879147095700184482
2019/01/31 11:29:14 [DEBUG] Waiting for state to become: [success]
2019/01/31 11:29:14 [INFO] Skipping S3 Bucket (tf-test-bucket-6879147095700184482): BucketRegionError: incorrect region, the bucket is not in 'us-west-2' region
  status code: 301, request id: , host id:
2019/01/31 11:29:14 Sweeper Tests ran:
  - aws_s3_bucket_object
  - aws_s3_bucket
ok    github.com/terraform-providers/terraform-provider-aws/aws 19.735s

@bflad bflad requested a review from a team January 31, 2019 16:34
@ghost ghost added size/L Managed by automation to categorize the size of a PR. service/s3 Issues and PRs that pertain to the s3 service. tests PRs: expanded test coverage. Issues: expanded coverage, enhancements to test infrastructure. labels Jan 31, 2019
@bflad
service/s3: Add sweepers
f9562cd 
To prevent TooManyBuckets errors from failed acceptance testing.

Output from test sweepers:

```
$ go test ./aws -v -sweep=us-west-2 -sweep-run=aws_s3_bucket -timeout 10h
2019/01/31 11:28:57 [DEBUG] Running Sweepers for region (us-west-2):
...
2019/01/31 11:28:59 [INFO] Skipping S3 Bucket: 187416307283-awsmacietrail-dataevent
2019/01/31 11:28:59 [INFO] Skipping S3 Bucket: aws-athena-query-results-187416307283-us-east-1
...
2019/01/31 11:28:59 [INFO] Skipping S3 Bucket (tf-acc-test-6417188599110687269destination) Objects: BucketRegionError: incorrect region, the bucket is not in 'us-west-2' region
  status code: 301, request id: , host id:
2019/01/31 11:28:59 [INFO] Skipping S3 Bucket: tf-alex-preview
2019/01/31 11:28:59 [INFO] Skipping S3 Bucket: tf-alex-preview2
2019/01/31 11:28:59 [INFO] Skipping S3 Bucket: tf-deploy-alex
2019/01/31 11:28:59 [INFO] Skipping S3 Bucket: tf-emr-bootstrap-5850549931442788494
2019/01/31 11:28:59 [INFO] Skipping S3 Bucket: tf-emr-bootstrap-8212862313789127315
2019/01/31 11:28:59 [INFO] Skipping S3 Bucket: tf-io-dev
2019/01/31 11:28:59 [INFO] Skipping S3 Bucket: tf-k8s-acc-test
2019/01/31 11:28:59 [INFO] Skipping S3 Bucket: tf-macie-test-bucket-921738264640544542
2019/01/31 11:29:00 [INFO] Deleting S3 Bucket (tf-object-test-bucket-272994333124655711) Object (tf-testing-obj-272994333124655711-all-params) Delete Marker: YECgZ632K8z.xlufooF.aeFvafBA6kuz
2019/01/31 11:29:02 [INFO] Deleting S3 Bucket (tf-object-test-bucket-3609874408477041358) Object (tf-testing-obj-3609874408477041358-all-params) Delete Marker: hQ4eUTYHGqhZ6lV9ATtCbu2F3HNuqgwa
2019/01/31 11:29:03 [INFO] Deleting S3 Bucket (tf-object-test-bucket-3660475575348874992) Object (updateable-key) Delete Marker: JxNsmwLQGbCqmtpMgt4n.d60NeCtC3V_
2019/01/31 11:29:04 [INFO] Deleting S3 Bucket (tf-object-test-bucket-3900350678782068183) Object (test-key) Delete Marker: auhlvd6dgHaEc4jv2sfi1yUKM_lkvh..
2019/01/31 11:29:05 [INFO] Deleting S3 Bucket (tf-object-test-bucket-4138169415932139850) Object (updateable-key) Delete Marker: nA74rE0HYETVIrXvq2zSu7Z4AQGc7jR9
2019/01/31 11:29:07 [INFO] Deleting S3 Bucket (tf-object-test-bucket-6391031696293038021) Object (updateable-key) Delete Marker: Cc0CRHZZ6P5Z3RIGSolfZk1KfNqs4n1l
2019/01/31 11:29:08 [INFO] Skipping S3 Bucket: tf-redshift-logging-5085425770880693968
2019/01/31 11:29:08 [INFO] Skipping S3 Bucket: tf-redshift-logging-9209707503695989768
2019/01/31 11:29:08 [INFO] Skipping S3 Bucket: tf-spot-datafeed-3152131621798213916
2019/01/31 11:29:08 [INFO] Skipping S3 Bucket (tf-test-bucket-1173093638223895558) Objects: BucketRegionError: incorrect region, the bucket is not in 'us-west-2' region
  status code: 301, request id: , host id:
2019/01/31 11:29:09 [INFO] Skipping S3 Bucket (tf-test-bucket-6879147095700184482) Objects: BucketRegionError: incorrect region, the bucket is not in 'us-west-2' region
  status code: 301, request id: , host id:
2019/01/31 11:29:09 [DEBUG] Sweeper (aws_s3_bucket) has dependency (aws_s3_bucket_object), running..
2019/01/31 11:29:09 [DEBUG] Sweeper (aws_s3_bucket_object) already ran in region (us-west-2)
...
2019/01/31 11:29:09 [INFO] Skipping S3 Bucket: 187416307283-awsmacietrail-dataevent
2019/01/31 11:29:09 [INFO] Skipping S3 Bucket: aws-athena-query-results-187416307283-us-east-1
...
2019/01/31 11:29:09 [INFO] Deleting S3 Bucket: tf-acc-test-6417188599110687269destination
2019/01/31 11:29:09 [DEBUG] Waiting for state to become: [success]
2019/01/31 11:29:10 [INFO] Skipping S3 Bucket (tf-acc-test-6417188599110687269destination): BucketRegionError: incorrect region, the bucket is not in 'us-west-2' region
  status code: 301, request id: , host id:
2019/01/31 11:29:10 [INFO] Skipping S3 Bucket: tf-alex-preview
2019/01/31 11:29:10 [INFO] Skipping S3 Bucket: tf-alex-preview2
2019/01/31 11:29:10 [INFO] Skipping S3 Bucket: tf-deploy-alex
2019/01/31 11:29:10 [INFO] Skipping S3 Bucket: tf-emr-bootstrap-5850549931442788494
2019/01/31 11:29:10 [INFO] Skipping S3 Bucket: tf-emr-bootstrap-8212862313789127315
2019/01/31 11:29:10 [INFO] Skipping S3 Bucket: tf-io-dev
2019/01/31 11:29:10 [INFO] Skipping S3 Bucket: tf-k8s-acc-test
2019/01/31 11:29:10 [INFO] Skipping S3 Bucket: tf-macie-test-bucket-921738264640544542
2019/01/31 11:29:10 [INFO] Deleting S3 Bucket: tf-object-test-bucket-272994333124655711
2019/01/31 11:29:10 [DEBUG] Waiting for state to become: [success]
2019/01/31 11:29:10 [INFO] Deleting S3 Bucket: tf-object-test-bucket-3609874408477041358
2019/01/31 11:29:10 [DEBUG] Waiting for state to become: [success]
2019/01/31 11:29:11 [INFO] Deleting S3 Bucket: tf-object-test-bucket-3660475575348874992
2019/01/31 11:29:11 [DEBUG] Waiting for state to become: [success]
2019/01/31 11:29:11 [INFO] Deleting S3 Bucket: tf-object-test-bucket-3900350678782068183
2019/01/31 11:29:11 [DEBUG] Waiting for state to become: [success]
2019/01/31 11:29:12 [INFO] Deleting S3 Bucket: tf-object-test-bucket-4138169415932139850
2019/01/31 11:29:12 [DEBUG] Waiting for state to become: [success]
2019/01/31 11:29:13 [INFO] Deleting S3 Bucket: tf-object-test-bucket-6391031696293038021
2019/01/31 11:29:13 [DEBUG] Waiting for state to become: [success]
2019/01/31 11:29:13 [INFO] Deleting S3 Bucket: tf-object-test-bucket-7215878632290092620
2019/01/31 11:29:13 [DEBUG] Waiting for state to become: [success]
2019/01/31 11:29:14 [INFO] Skipping S3 Bucket: tf-redshift-logging-5085425770880693968
2019/01/31 11:29:14 [INFO] Skipping S3 Bucket: tf-redshift-logging-9209707503695989768
2019/01/31 11:29:14 [INFO] Skipping S3 Bucket: tf-spot-datafeed-3152131621798213916
2019/01/31 11:29:14 [INFO] Deleting S3 Bucket: tf-test-bucket-1173093638223895558
2019/01/31 11:29:14 [DEBUG] Waiting for state to become: [success]
2019/01/31 11:29:14 [INFO] Skipping S3 Bucket (tf-test-bucket-1173093638223895558): BucketRegionError: incorrect region, the bucket is not in 'us-west-2' region
  status code: 301, request id: , host id:
2019/01/31 11:29:14 [INFO] Deleting S3 Bucket: tf-test-bucket-6879147095700184482
2019/01/31 11:29:14 [DEBUG] Waiting for state to become: [success]
2019/01/31 11:29:14 [INFO] Skipping S3 Bucket (tf-test-bucket-6879147095700184482): BucketRegionError: incorrect region, the bucket is not in 'us-west-2' region
  status code: 301, request id: , host id:
2019/01/31 11:29:14 Sweeper Tests ran:
  - aws_s3_bucket_object
  - aws_s3_bucket
ok    github.com/terraform-providers/terraform-provider-aws/aws 19.735s
```
@bflad bflad added this to the v1.58.0 milestone Jan 31, 2019
@bflad bflad merged commit c00504c into master Jan 31, 2019
@bflad bflad deleted the t-s3-sweepers branch January 31, 2019 20:09
@ewbankkit ewbankkit mentioned this pull request Feb 1, 2019
Closed
@ghost
Copy link

ghost commented Apr 1, 2020

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!

@ghost ghost locked and limited conversation to collaborators Apr 1, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@paddycarver paddycarver paddycarver approved these changes

Assignees
No one assigned
Labels
service/s3 Issues and PRs that pertain to the s3 service. size/L Managed by automation to categorize the size of a PR. tests PRs: expanded test coverage. Issues: expanded coverage, enhancements to test infrastructure.
Projects
None yet
Milestone
v1.58.0
Development

Successfully merging this pull request may close these issues.
2 participants
@bflad @paddycarver