Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

service/s3control: DeleteBucketInput Modified After Use #3583

Closed
3 tasks done
bflad opened this issue Oct 8, 2020 · 2 comments · Fixed by #3597
Closed
3 tasks done

service/s3control: DeleteBucketInput Modified After Use #3583

bflad opened this issue Oct 8, 2020 · 2 comments · Fixed by #3597
Labels
bug This issue is a bug.

Comments

@bflad
Copy link
Contributor

bflad commented Oct 8, 2020

Confirm by changing [ ] to [x] below to ensure that it's a bug:

Describe the bug

The new S3 on Outposts functionality released in version 1.35.0 of the AWS Go SDK requires customizations to handle routing S3 Control requests to the correct endpoint. These customization seem to be modifying the operation inputs (at least the DeleteBucketInput type), which is unexpected behavior compared to other AWS Go SDK service handling and especially causes errors with the s3control service since it modifies the Bucket parameter in a way that breaks downstream usage such as manually retrying on error.

Version of AWS SDK for Go?

v1.35.5

Version of Go (go version)?

go version go1.15.2 darwin/amd64

To Reproduce (observed behavior)

package main

import (
	"log"

	"github.com/aws/aws-sdk-go/aws"
	"github.com/aws/aws-sdk-go/aws/session"
	"github.com/aws/aws-sdk-go/service/s3control"
)

func main() {
	config := aws.NewConfig().WithLogLevel(aws.LogDebugWithHTTPBody).WithRegion("us-east-1")
	sess := session.Must(session.NewSession(config))
	svc := s3control.New(sess)

	input := &s3control.DeleteBucketInput{
		AccountId: aws.String("123456789012"),
		Bucket:    aws.String("arn:aws:s3-outposts:us-east-1:123456789012:outpost/op-1234567890abcdef/bucket/does-not-matter"),
	}

	log.Printf("[INFO] DeleteBucketInput: %v", input)

	_, err := svc.DeleteBucket(input)

	if err != nil {
		log.Printf("[ERROR] DeleteBucket error: %s", err)
	}

	log.Printf("[INFO] DeleteBucketInput: %v", input)

	_, err = svc.DeleteBucket(input)

	if err != nil {
		log.Printf("[ERROR] DeleteBucket error: %s", err)
	}
}

Output:

2020/10/07 21:32:03 [INFO] DeleteBucketInput: {
  AccountId: "123456789012",
  Bucket: "arn:aws:s3-outposts:us-east-1:123456789012:outpost/op-1234567890abcdef/bucket/does-not-matter"
}
2020/10/07 21:32:03 DEBUG: Request S3 Control/DeleteBucket Details:
---[ REQUEST POST-SIGN ]-----------------------------
DELETE /v20180820/bucket/does-not-matter HTTP/1.1
Host: s3-outposts.us-east-1.amazonaws.com
User-Agent: aws-sdk-go/1.35.5 (go1.15.2; darwin; amd64)
Authorization: AWS4-HMAC-SHA256 Credential=AKIASXIXB4ZJSOXTHGOI/20201008/us-east-1/s3-outposts/aws4_request, SignedHeaders=host;x-amz-account-id;x-amz-date;x-amz-outpost-id, Signature=3bc91e0ff7dbcd45782ead9f4e61650266cfa82538c74b8fdc7ce114c5e0537e
X-Amz-Account-Id: 123456789012
X-Amz-Account-Id: 123456789012
X-Amz-Date: 20201008T013203Z
X-Amz-Outpost-Id: op-1234567890abcdef
Accept-Encoding: gzip


-----------------------------------------------------
2020/10/07 21:32:04 DEBUG: Response S3 Control/DeleteBucket Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 404 Not Found
Content-Length: 348
Content-Md5: yYXYFLBs4gQ1+sRMOkiflg==
Content-Type: application/xml
Date: Thu, 8 Oct 2020 01:32:04 GMT
X-Amz-Id-2: YpuES4k3dxaO/ByGzsMa514g9FxSaNGsgY4ARcUN48b8OnLcXDjZlx/Ukqh/RToX
X-Amzn-Requestid: ee49a321-3df9-4354-a039-3c82c1979ce8


-----------------------------------------------------
2020/10/07 21:32:04 <?xml version="1.0" encoding="UTF-8"?>
<ErrorResponse><Error><Code>NoSuchOutpost</Code><Message>The specified outpost does not exist</Message><OutpostId>op-1234567890abcdef</OutpostId></Error><RequestId>ee49a321-3df9-4354-a039-3c82c1979ce8</RequestId><HostId>YpuES4k3dxaO/ByGzsMa514g9FxSaNGsgY4ARcUN48b8OnLcXDjZlx/Ukqh/RToX</HostId></ErrorResponse>
2020/10/07 21:32:04 [ERROR] DeleteBucket error: NoSuchOutpost: The specified outpost does not exist
	status code: 404, request id: ee49a321-3df9-4354-a039-3c82c1979ce8, host id: YpuES4k3dxaO/ByGzsMa514g9FxSaNGsgY4ARcUN48b8OnLcXDjZlx/Ukqh/RToX
2020/10/07 21:32:04 [INFO] DeleteBucketInput: {
  AccountId: "123456789012",
  Bucket: "does-not-matter"
}
2020/10/07 21:32:04 DEBUG: Request S3 Control/DeleteBucket Details:
---[ REQUEST POST-SIGN ]-----------------------------
DELETE /v20180820/bucket/does-not-matter HTTP/1.1
Host: 123456789012.s3-control.us-east-1.amazonaws.com
User-Agent: aws-sdk-go/1.35.5 (go1.15.2; darwin; amd64)
Authorization: AWS4-HMAC-SHA256 Credential=AKIASXIXB4ZJSOXTHGOI/20201008/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-account-id;x-amz-content-sha256;x-amz-date, Signature=7ddd5ef0dd31406497b91fefd70b8a03bd86b20fbfcba09a707cdbb2cd0fb130
X-Amz-Account-Id: 123456789012
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20201008T013204Z
Accept-Encoding: gzip


-----------------------------------------------------
2020/10/07 21:32:04 DEBUG: Response S3 Control/DeleteBucket Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 403 Forbidden
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Thu, 08 Oct 2020 01:32:04 GMT
Server: AmazonS3
X-Amz-Id-2: U4vxQl07F4F+1Pbl8BUkGj8W+KjOQJUubCmSPDjVh/Y2Gs/FwDdMIM9G4B2v1ZSh8TppfMfwdwA=
X-Amz-Request-Id: ACD15EC4E6E82E24


-----------------------------------------------------
2020/10/07 21:32:04 <?xml version="1.0" encoding="UTF-8"?>
<ErrorResponse><Error><Code>AccessDenied</Code><Message>Access Denied</Message><AccountId>123456789012</AccountId></Error><RequestId>ACD15EC4E6E82E24</RequestId><HostId>U4vxQl07F4F+1Pbl8BUkGj8W+KjOQJUubCmSPDjVh/Y2Gs/FwDdMIM9G4B2v1ZSh8TppfMfwdwA=</HostId></ErrorResponse>
2020/10/07 21:32:04 [ERROR] DeleteBucket error: AccessDenied: Access Denied
	status code: 403, request id: ACD15EC4E6E82E24, host id: U4vxQl07F4F+1Pbl8BUkGj8W+KjOQJUubCmSPDjVh/Y2Gs/FwDdMIM9G4B2v1ZSh8TppfMfwdwA=

Expected behavior

2020/10/07 21:32:03 [INFO] DeleteBucketInput: {
  AccountId: "123456789012",
  Bucket: "arn:aws:s3-outposts:us-east-1:123456789012:outpost/op-1234567890abcdef/bucket/does-not-matter"
}
2020/10/07 21:32:03 DEBUG: Request S3 Control/DeleteBucket Details:
---[ REQUEST POST-SIGN ]-----------------------------
DELETE /v20180820/bucket/does-not-matter HTTP/1.1
Host: s3-outposts.us-east-1.amazonaws.com
...
X-Amz-Account-Id: 123456789012
X-Amz-Outpost-Id: op-1234567890abcdef
...
2020/10/07 21:32:04 [ERROR] DeleteBucket error: NoSuchOutpost: The specified outpost does not exist
...
2020/10/07 21:32:04 [INFO] DeleteBucketInput: {
  AccountId: "123456789012",
  Bucket: "arn:aws:s3-outposts:us-east-1:123456789012:outpost/op-1234567890abcdef/bucket/does-not-matter"
}
---[ REQUEST POST-SIGN ]-----------------------------
DELETE /v20180820/bucket/does-not-matter HTTP/1.1
Host: s3-outposts.us-east-1.amazonaws.com
...
X-Amz-Account-Id: 123456789012
X-Amz-Outpost-Id: op-1234567890abcdef
...
2020/10/07 21:32:04 [ERROR] DeleteBucket error: NoSuchOutpost: The specified outpost does not exist

Additional context

The Bucket being changed from the ARN to the name causes the HTTP headers and endpoint routing to break in real world usage on second use. Real world logs:

2020/10/07 21:04:53 [INFO] DeleteBucketInput: {
  AccountId: "--OMITTED--",
  Bucket: "arn:aws:s3-outposts:us-east-1:--OMITTED--:outpost/op-0e532e26b9a150b8d/bucket/tf-acc-test-5912182846350777500"
}
2020/10/07 21:04:53 [DEBUG] [aws-sdk-go] DEBUG: Request S3 Control/DeleteBucket Details:
---[ REQUEST POST-SIGN ]-----------------------------
DELETE /v20180820/bucket/tf-acc-test-5912182846350777500 HTTP/1.1
Host: s3-outposts.us-east-1.amazonaws.com
User-Agent: aws-sdk-go/1.35.2 (go1.15.2; darwin; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.12.29 (+https://www.terraform.io)
Authorization: AWS4-HMAC-SHA256 Credential=ASIATTIORZ32USXPHOHU/20201008/us-east-1/s3-outposts/aws4_request, SignedHeaders=host;x-amz-account-id;x-amz-date;x-amz-outpost-id;x-amz-security-token, Signature=87d083d6ed1b4f20710b97e5e5a635f9f4e9d61eb7880d2cbfa6bcc160ea9d78
X-Amz-Account-Id: --OMITTED--
X-Amz-Account-Id: --OMITTED--
X-Amz-Date: 20201008T010453Z
X-Amz-Outpost-Id: op-0e532e26b9a150b8d
X-Amz-Security-Token: FwoGZXIvYXdzEPv//////////wEaDHD4rWm6KGtyWAN9vCK3ASKjoFjRu6/CBh90XwAJwKfrhIc6eh8jwlU1PjzFub1Ujus21E016AZaZiF61lbzOawQbRQp+FmXSfQ71cnOl2FP3/kLAQ3NiXDNUNek3OrwC83a9jePcqz7RH9tsSBiq9jlobomtbwj2ZtuApRhRPyNUN3ujKq3gsA3xtgH0TXGjCJb6yRyPLpJsLJkLSmnkEIPY5DkpYDHj2vETSKBtc/n/xap3nNNGDR7jQGju7FCQzycodoxNSi0y/n7BTItEU24OdBJRSzm+O9bBXBYubIeYfZnppanFRfr5et3OglR8FItHYvWYOGhpUdx
Accept-Encoding: gzip


-----------------------------------------------------
2020/10/07 21:04:53 [DEBUG] [aws-sdk-go] DEBUG: Response S3 Control/DeleteBucket Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 409 Conflict
Content-Length: 304
Content-Md5: yYXYFLBs4gQ1+sRMOkiflg==
Content-Type: application/xml
Date: Thu, 8 Oct 2020 01:04:53 GMT
X-Amz-Id-2: OOTqifq0i71wXCCINPTkPe95Gv3LCQ1j8d15p0VtUTFB9cJWtYudoAPTn+vYvNwB
X-Amzn-Requestid: 10b05d18-e080-4d71-ab1f-f61dc8b03835


-----------------------------------------------------
2020/10/07 21:04:53 [DEBUG] [aws-sdk-go] <?xml version="1.0" encoding="UTF-8"?>
<ErrorResponse><Error><Code>InvalidBucketState</Code><Message>Bucket is in an invalid state</Message></Error><RequestId>10b05d18-e080-4d71-ab1f-f61dc8b03835</RequestId><HostId>OOTqifq0i71wXCCINPTkPe95Gv3LCQ1j8d15p0VtUTFB9cJWtYudoAPTn+vYvNwB</HostId></ErrorResponse>
2020/10/07 21:04:53 [DEBUG] [aws-sdk-go] DEBUG: Validate Response S3 Control/DeleteBucket failed, attempt 0/25, error InvalidBucketState: Bucket is in an invalid state
	status code: 409, request id: 10b05d18-e080-4d71-ab1f-f61dc8b03835, host id: OOTqifq0i71wXCCINPTkPe95Gv3LCQ1j8d15p0VtUTFB9cJWtYudoAPTn+vYvNwB
2020/10/07 21:04:53 [BFLAD] DeleteBucketInput: {
  AccountId: "--OMITTED--",
  Bucket: "tf-acc-test-5912182846350777500"
}
2020/10/07 21:04:53 [DEBUG] [aws-sdk-go] DEBUG: Request S3 Control/DeleteBucket Details:
---[ REQUEST POST-SIGN ]-----------------------------
DELETE /v20180820/bucket/tf-acc-test-5912182846350777500 HTTP/1.1
Host: --OMITTED--.s3-control.us-east-1.amazonaws.com
User-Agent: aws-sdk-go/1.35.2 (go1.15.2; darwin; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.12.29 (+https://www.terraform.io)
Authorization: AWS4-HMAC-SHA256 Credential=ASIATTIORZ32USXPHOHU/20201008/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-account-id;x-amz-content-sha256;x-amz-date;x-amz-security-token, Signature=f2282f3b4b71a7c64a2653c13db6b74aba08449734b89a132e27e03212af8ca0
X-Amz-Account-Id: --OMITTED--
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20201008T010453Z
X-Amz-Security-Token: FwoGZXIvYXdzEPv//////////wEaDHD4rWm6KGtyWAN9vCK3ASKjoFjRu6/CBh90XwAJwKfrhIc6eh8jwlU1PjzFub1Ujus21E016AZaZiF61lbzOawQbRQp+FmXSfQ71cnOl2FP3/kLAQ3NiXDNUNek3OrwC83a9jePcqz7RH9tsSBiq9jlobomtbwj2ZtuApRhRPyNUN3ujKq3gsA3xtgH0TXGjCJb6yRyPLpJsLJkLSmnkEIPY5DkpYDHj2vETSKBtc/n/xap3nNNGDR7jQGju7FCQzycodoxNSi0y/n7BTItEU24OdBJRSzm+O9bBXBYubIeYfZnppanFRfr5et3OglR8FItHYvWYOGhpUdx
Accept-Encoding: gzip


-----------------------------------------------------
2020/10/07 21:04:54 [DEBUG] [aws-sdk-go] DEBUG: Response S3 Control/DeleteBucket Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 400 Bad Request
Connection: close
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Thu, 08 Oct 2020 01:04:54 GMT
Server: AmazonS3
X-Amz-Id-2: YcPvoAO9Najfa2AEzZGncoX82rsW31dONAXjXs8KLSwnj4L9k7Q6dZTtPYX8fC5R55guP5f8R9U=
X-Amz-Request-Id: DD7F8F01B812FC1C


-----------------------------------------------------
2020/10/07 21:04:54 [DEBUG] [aws-sdk-go] <?xml version="1.0" encoding="UTF-8"?>
<ErrorResponse><Error><Code>InvalidURI</Code><Message>Couldn't parse the specified URI.</Message><URI>bucket/tf-acc-test-5912182846350777500</URI></Error><RequestId>DD7F8F01B812FC1C</RequestId><HostId>YcPvoAO9Najfa2AEzZGncoX82rsW31dONAXjXs8KLSwnj4L9k7Q6dZTtPYX8fC5R55guP5f8R9U=</HostId></ErrorResponse>
2020/10/07 21:04:54 [DEBUG] [aws-sdk-go] DEBUG: Validate Response S3 Control/DeleteBucket failed, attempt 0/25, error InvalidURI: Couldn't parse the specified URI.
	status code: 400, request id: DD7F8F01B812FC1C, host id: YcPvoAO9Najfa2AEzZGncoX82rsW31dONAXjXs8KLSwnj4L9k7Q6dZTtPYX8fC5R55guP5f8R9U=
@bflad bflad added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Oct 8, 2020
@KaibaLopez
Copy link
Contributor

Hi @bflad ,
Thanks for pointing this out to us, we'll be looking into this.

@KaibaLopez KaibaLopez removed the needs-triage This issue or PR still needs to be triaged. label Oct 8, 2020
Merged
aws-sdk-go-automation pushed a commit that referenced this issue Oct 15, 2020
Release v1.35.8 (2020-10-15)
c441669 
===

### Service Client Updates
* `service/accessanalyzer`: Updates service API and documentation
* `service/budgets`: Updates service API, documentation, and paginators
  * This release introduces AWS Budgets Actions, allowing you to define an explicit response(or set of responses)  to take when your budget exceeds it's action threshold.
* `service/ce`: Updates service API and documentation
* `service/dms`: Updates service API and documentation
  * When creating Endpoints, Replication Instances, and Replication Tasks, the feature provides you the option to specify friendly name to the resources.
* `service/glue`: Updates service documentation
  * API Documentation updates for Glue Get-Plan API
* `service/groundstation`: Updates service API and documentation
* `service/iot`: Updates service API and documentation
  * Add new variable, lastStatusChangeDate, to DescribeDomainConfiguration  API
* `service/macie2`: Updates service API and documentation
* `service/rds`: Updates service API and documentation
  * Return tags for all resources in the output of DescribeDBInstances, DescribeDBSnapshots, DescribeDBClusters, and DescribeDBClusterSnapshots API operations.
* `service/rekognition`: Updates service API and documentation
  * This SDK Release introduces new API (DetectProtectiveEquipment) for Amazon Rekognition. This release also adds ServiceQuotaExceeded exception to Amazon Rekognition IndexFaces API.
* `service/ssm`: Updates service API and documentation
  * This Patch Manager release now supports searching for available packages from Amazon Linux and Amazon Linux 2 via the DescribeAvailablePatches API.
* `service/transfer`: Updates service API and documentation
  * Add support to associate VPC Security Groups at server creation.
* `service/workmail`: Updates service API and documentation
  * Add CreateOrganization and DeleteOrganization API operations.
* `service/workspaces`: Updates service documentation
  * Documentation updates for WorkSpaces
* `service/xray`: Updates service API, documentation, and paginators
  * Enhancing CreateGroup, UpdateGroup, GetGroup and GetGroups APIs to support configuring X-Ray Insights Notifications. Adding TraceLimit information into X-Ray BatchGetTraces API response.

### SDK Bugs
* `s3control`: Fixes bug in SDK that caused input for certain s3control operation to be modified, when using ARNs. ([#3595](#3595))
  * Fixes [#3583](#3583).
aws-sdk-go-automation added a commit that referenced this issue Oct 15, 2020
@aws-sdk-go-automation
Release v1.35.8 (2020-10-15) (#3597)
32ff04d 
Release v1.35.8 (2020-10-15)
===

### Service Client Updates
* `service/accessanalyzer`: Updates service API and documentation
* `service/budgets`: Updates service API, documentation, and paginators
  * This release introduces AWS Budgets Actions, allowing you to define an explicit response(or set of responses)  to take when your budget exceeds it's action threshold.
* `service/ce`: Updates service API and documentation
* `service/dms`: Updates service API and documentation
  * When creating Endpoints, Replication Instances, and Replication Tasks, the feature provides you the option to specify friendly name to the resources.
* `service/glue`: Updates service documentation
  * API Documentation updates for Glue Get-Plan API
* `service/groundstation`: Updates service API and documentation
* `service/iot`: Updates service API and documentation
  * Add new variable, lastStatusChangeDate, to DescribeDomainConfiguration  API
* `service/macie2`: Updates service API and documentation
* `service/rds`: Updates service API and documentation
  * Return tags for all resources in the output of DescribeDBInstances, DescribeDBSnapshots, DescribeDBClusters, and DescribeDBClusterSnapshots API operations.
* `service/rekognition`: Updates service API and documentation
  * This SDK Release introduces new API (DetectProtectiveEquipment) for Amazon Rekognition. This release also adds ServiceQuotaExceeded exception to Amazon Rekognition IndexFaces API.
* `service/ssm`: Updates service API and documentation
  * This Patch Manager release now supports searching for available packages from Amazon Linux and Amazon Linux 2 via the DescribeAvailablePatches API.
* `service/transfer`: Updates service API and documentation
  * Add support to associate VPC Security Groups at server creation.
* `service/workmail`: Updates service API and documentation
  * Add CreateOrganization and DeleteOrganization API operations.
* `service/workspaces`: Updates service documentation
  * Documentation updates for WorkSpaces
* `service/xray`: Updates service API, documentation, and paginators
  * Enhancing CreateGroup, UpdateGroup, GetGroup and GetGroups APIs to support configuring X-Ray Insights Notifications. Adding TraceLimit information into X-Ray BatchGetTraces API response.

### SDK Bugs
* `s3control`: Fixes bug in SDK that caused input for certain s3control operation to be modified, when using ARNs. ([#3595](#3595))
  * Fixes [#3583](#3583).
@github-actions
Copy link

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

@bflad bflad mentioned this issue Oct 15, 2020
Merged
bflad added a commit to hashicorp/terraform-provider-aws that referenced this issue Oct 16, 2020
@bflad
resource/aws_s3control_bucket: Remove workaround fixed in AWS Go SDK
c58091d 
Reference: aws/aws-sdk-go#3583
bflad added a commit to hashicorp/terraform-provider-aws that referenced this issue Oct 16, 2020
@bflad
resource/aws_s3control_bucket: Remove workaround fixed in AWS Go SDK
2d6f9bf 
Reference: aws/aws-sdk-go#3583
bflad added a commit to hashicorp/terraform-provider-aws that referenced this issue Oct 27, 2020
@bflad
New Resource: aws_s3control_bucket (#15510)
af69dda 
* New Resource: aws_s3control_bucket

Reference: #15413

Output from acceptance testing:

```
--- PASS: TestAccAWSS3ControlBucket_basic (58.52s)
--- PASS: TestAccAWSS3ControlBucket_disappears (63.94s)
--- SKIP: TestAccAWSS3ControlBucket_Tags (0.00s)
```

* resource/aws_s3control_bucket: Remove workaround fixed in AWS Go SDK

Reference: aws/aws-sdk-go#3583

* Update aws/resource_aws_s3control_bucket_test.go
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug This issue is a bug.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Release v1.35.8 aws/aws-sdk-go
2 participants
@bflad @KaibaLopez