Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Terraform AWS Provider Version 2.45.0: Missing Error Messages and Not Retrying #11682

Closed
bflad opened this issue Jan 20, 2020 · 6 comments · Fixed by #11727
Closed

Terraform AWS Provider Version 2.45.0: Missing Error Messages and Not Retrying #11682

bflad opened this issue Jan 20, 2020 · 6 comments · Fixed by #11727
Labels
regression Pertains to a degraded workflow resulting from an upstream patch or internal enhancement. service/cloud9 Issues and PRs that pertain to the cloud9 service. service/cloudtrail Issues and PRs that pertain to the cloudtrail service. service/cloudwatch Issues and PRs that pertain to the cloudwatch service. service/codebuild Issues and PRs that pertain to the codebuild service. service/configservice Issues and PRs that pertain to the configservice service. service/ecs Issues and PRs that pertain to the ecs service. service/elasticsearch Issues and PRs that pertain to the elasticsearch service. upstream Addresses functionality related to the cloud provider.
Milestone
v2.46.0

Comments

@bflad
Copy link
Contributor

bflad commented Jan 20, 2020 •
edited
Loading

Community Note

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
  • Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

Terraform AWS Provider version 2.45.0

Affected Resource(s)

  • aws_cloud9_environment_ec2
  • aws_cloudtrail
  • aws_cloudwatch_log_destination
  • aws_cloudwatch_log_subscription
  • aws_codebuild_project
  • aws_codedeploy_deployment_group
  • aws_config_delivery_channel
  • aws_ecs_service
  • aws_elasticsearch_domain

Additional resources and errors may be missing error messaging as well, however these are directly affected by resource logic attempting to retry on error messages that are no longer present in the AWS Go SDK as of version 1.28.0.

Terraform Configuration Files

# Resource dependent

Debug Output

Example debug output (note lack of error message after error code on last line):

2020/01/20 12:59:15 [DEBUG] [aws-sdk-go] DEBUG: Response codebuild/CreateProject Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 400 Bad Request
Connection: close
Content-Length: 167
Content-Type: application/x-amz-json-1.1
Date: Mon, 20 Jan 2020 17:59:14 GMT
X-Amzn-Requestid: 2f589026-d531-458d-b851-86bd4afb5cd0


-----------------------------------------------------
2020/01/20 12:59:15 [DEBUG] [aws-sdk-go] {"__type":"InvalidInputException","message":"CodeBuild is not authorized to perform: sts:AssumeRole on arn:aws:iam::--OMITTED--:role/tf-acc-test-4387558838057379294"}
2020/01/20 12:59:15 [DEBUG] [aws-sdk-go] DEBUG: Validate Response codebuild/CreateProject failed, attempt 0/25, error InvalidInputException:

Expected Behavior

In affected resources, Terraform AWS Provider outputs error messages and retries on certain errors. e.g. those resources affected by new lack of ability to retry properly:

Historically, we have preferred to check specific error messaging from the service API responses since the error code itself is not descriptive enough to determine if the error could/should be retried. While the service APIs should likely introduce more granular error codes and fix their API models, the unexpected removal of the error messaging represents a fairly problematic change in the AWS Go SDK since it was previously supported by many services.

Actual Behavior

Error messages (after the error code and colon when printed) are missing (an empty string). Retries in certain resources are not performed as expected since the error message conditions are no longer true.

Steps to Reproduce

  1. terraform apply

Important Factoids

Changes in v1.28.0 of the AWS Go SDK (imported and released by Terraform AWS Provider version 2.45.0) introduced code generated types for certain service error shapes, which can be used to extract additional information provided by the API response. Prior to these code generated types, most (if not all) of these errors returned the correct error messaging via Message() via the message protocol response field. However after this change, since some of these error shapes in the service API models are incorrectly missing the Message structure member, this causes the Message_ (Go struct type) to be missing, which causes the error message via Message() to always be empty.

An example of this missing error messaging can be seen with codebuild service InvalidInputException shape:

https://github.com/aws/aws-sdk-go/blob/6627e8938cdef5476dec0510b524258d3bf3983e/models/apis/codebuild/2016-10-06/api-2.json#L998-L1003

https://github.com/aws/aws-sdk-go/blob/6627e8938cdef5476dec0510b524258d3bf3983e/service/codebuild/api.go#L5333-L5336

Full SDK analysis of this new behavior was performed here: #11642 (comment)

References
@bflad bflad added upstream Addresses functionality related to the cloud provider. service/ecs Issues and PRs that pertain to the ecs service. service/configservice Issues and PRs that pertain to the configservice service. service/cloudwatchlogs service/codebuild Issues and PRs that pertain to the codebuild service. service/cloudtrail Issues and PRs that pertain to the cloudtrail service. service/elasticsearch Issues and PRs that pertain to the elasticsearch service. service/cloud9 Issues and PRs that pertain to the cloud9 service. labels Jan 20, 2020
@ghost ghost added the service/cloudwatch Issues and PRs that pertain to the cloudwatch service. label Jan 20, 2020
@github-actions github-actions bot added the needs-triage Waiting for first response or review from a maintainer. label Jan 20, 2020
@bflad bflad mentioned this issue Jan 20, 2020
Closed
@bflad bflad removed the needs-triage Waiting for first response or review from a maintainer. label Jan 20, 2020
@bflad
Copy link
Contributor Author

bflad commented Jan 20, 2020

Created upstream AWS Go SDK issue: aws/aws-sdk-go#3088

This was referenced Jan 20, 2020
Closed
Closed
@DrFaust92 DrFaust92 mentioned this issue Jan 20, 2020
Closed
@bflad bflad added the regression Pertains to a degraded workflow resulting from an upstream patch or internal enhancement. label Jan 20, 2020
@bflad
Copy link
Contributor Author

bflad commented Jan 21, 2020

Upstream fix aws/aws-sdk-go#3089 appears to resolve this issue -- we'll get this dependency update merged as soon as its available upstream.

@bflad
Copy link
Contributor Author

bflad commented Jan 22, 2020

The upstream fix for this has been merged and should go out with today's AWS Go SDK release.

@bflad bflad mentioned this issue Jan 22, 2020
Closed
@ghost ghost mentioned this issue Jan 22, 2020
Closed
@bflad bflad added this to the v2.46.0 milestone Jan 22, 2020
This was referenced Jan 22, 2020
Closed
Closed
Closed
bflad added a commit that referenced this issue Jan 23, 2020
@bflad
deps: github.com/aws/aws-sdk-go@6ca8a5496cb4
f8c65f5 
Reference: #11682
Reference: aws/aws-sdk-go#3089

The fix for widespread error handling regression was merged upstream into the AWS Go SDK, however there was not a release today on the normal daily release cadence. This pulls in the commit after AWS Go SDK v1.28.7 containing the fix for the missing error messages that was merged yesterday so we can allow the full acceptance test suite to run overnight.

Updated via:

```
go get github.com/aws/aws-sdk-go@6ca8a5496cb4723ad4ef1e6709ed593ea81ce000
go mod tidy
go mod vendor
```

Output from acceptance testing (as a smoke test of the fixed behavior):

```
--- PASS: TestAccAWSCodeBuildProject_basic (37.35s)
```
@bflad bflad mentioned this issue Jan 23, 2020
Merged
bflad added a commit that referenced this issue Jan 23, 2020
@bflad
deps: github.com/aws/aws-sdk-go@6ca8a5496cb4 (#11727)
1860e72 
Reference: #11682
Reference: aws/aws-sdk-go#3089

The fix for widespread error handling regression was merged upstream into the AWS Go SDK, however there was not a release today on the normal daily release cadence. This pulls in the commit after AWS Go SDK v1.28.7 containing the fix for the missing error messages that was merged yesterday so we can allow the full acceptance test suite to run overnight.

Updated via:

```
go get github.com/aws/aws-sdk-go@6ca8a5496cb4723ad4ef1e6709ed593ea81ce000
go mod tidy
go mod vendor
```

Output from acceptance testing (as a smoke test of the fixed behavior):

```
--- PASS: TestAccAWSCodeBuildProject_basic (37.35s)
```
@bflad
Copy link
Contributor Author

bflad commented Jan 23, 2020

Oddly enough, there was not an AWS Go SDK release on its normal daily release cadence today. To ensure we have fully run the provider test suite (occurs nightly), we have merged in the expected fix directly. If there are no additional regressions, this fix will release tomorrow with version 2.46.0 of the Terraform AWS Provider.

@bflad bflad mentioned this issue Jan 23, 2020
Closed
@ghost
Copy link

ghost commented Jan 23, 2020

This has been released in version 2.46.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template for triage. Thanks!

This was referenced Jan 24, 2020
Closed
Closed
Closed
Closed
Merged
@ghost
Copy link

ghost commented Mar 27, 2020

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!

@ghost ghost locked and limited conversation to collaborators Mar 27, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
regression Pertains to a degraded workflow resulting from an upstream patch or internal enhancement. service/cloud9 Issues and PRs that pertain to the cloud9 service. service/cloudtrail Issues and PRs that pertain to the cloudtrail service. service/cloudwatch Issues and PRs that pertain to the cloudwatch service. service/codebuild Issues and PRs that pertain to the codebuild service. service/configservice Issues and PRs that pertain to the configservice service. service/ecs Issues and PRs that pertain to the ecs service. service/elasticsearch Issues and PRs that pertain to the elasticsearch service. upstream Addresses functionality related to the cloud provider.
Projects
None yet
Milestone
v2.46.0
Development

Successfully merging a pull request may close this issue.

deps: github.com/aws/aws-sdk-go@6ca8a5496cb4 hashicorp/terraform-provider-aws
1 participant
@bflad