Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

resource/aws_api_gateway_rest_api: Remove extraneous timeout for resource deletion #7554

Merged
merged 1 commit into from
Feb 14, 2019

Conversation

bflad
Copy link
Contributor

@bflad bflad commented Feb 14, 2019

AWS API Gateway currently has a 1 DeleteRestApi/minute rate limit and when exceeding that returns 429 TooManyRequests errors which are automatically retried by the AWS Go SDK. The resource was artificially timeboxing the DeleteRestApi API call to 10 minutes previously with no RetryableError condition.

When running with parallelism over 10 (e.g. the acceptance testing runs at 20), there was a very high probability that one or more resource deletions would fail with the following error since the AWS Go SDK does not return the 429 error back to the calling code while retrying:

Error: errors during apply: timeout while waiting for state to become 'success' (timeout: 10m0s)

Removing the Retry() wrapper means the resource will retry the DeleteRestApi request without time limitations up to the configured provider max_retries (AWS Go SDK MaxRetries), which defaults to 25 and its exponential backoff generally lasts up to an hour.

A previous full TestAccAWSAPIGateway (76 tests currently) acceptance test run resulted in 16 failures due to the timeout. Rerunning with this change yields no timeout related failures where anything over ~600 seconds would have failed previously. The TestAccAWSAPIGatewayUsagePlan failures are unrelated and will require separate addressing.

--- PASS: TestAccAWSAPIGatewayAccount_importBasic (9.38s)
--- PASS: TestAccAWSAPIGatewayClientCertificate_basic (15.14s)
--- PASS: TestAccAWSAPIGatewayDocumentationPart_basic (21.81s)
--- PASS: TestAccAWSAPIGatewayDomainName_CertificateName (57.40s)
--- PASS: TestAccAWSAPIGatewayDocumentationVersion_importBasic (64.03s)
--- PASS: TestAccAWSAPIGatewayIntegrationResponse_basic (93.82s)
--- PASS: TestAccAWSAPIGatewayAuthorizer_cognito (31.27s)
--- PASS: TestAccAWSAPIGatewayDocumentationPart_importBasic (115.89s)
--- PASS: TestAccAWSAPIGatewayGatewayResponse_basic (187.30s)
--- PASS: TestAccAWSAPIGatewayBasePathMapping_basic (30.50s)
--- PASS: TestAccAWSAPIGatewayClientCertificate_importBasic (9.61s)
--- PASS: TestAccAWSAPIGatewayIntegration_basic (245.51s)
--- PASS: TestAccAWSAPIGatewayDeployment_StageDescription (31.08s)
--- PASS: TestAccAWSAPIGatewayDeployment_createBeforeDestoryUpdate (32.20s)
--- PASS: TestAccAWSAPIGatewayMethod_customauthorizer (31.91s)
--- PASS: TestAccAWSAPIGatewayIntegration_contentHandling (370.99s)
--- PASS: TestAccAWSAPIGatewayAuthorizer_switchAuthType (58.16s)
--- PASS: TestAccAWSAPIGatewayApiKey_basic (67.22s)
--- PASS: TestAccAWSAPIGatewayDeployment_Description (25.98s)
--- PASS: TestAccAWSAPIGatewayDocumentationVersion_importAllFields (517.75s)
--- PASS: TestAccAWSAPIGatewayAuthorizer_basic (45.30s)
--- PASS: TestAccAWSAPIGatewayResource_update (33.81s)
--- PASS: TestAccAWSAPIGatewayDocumentationVersion_basic (600.43s)
--- PASS: TestAccAWSAPIGatewayMethodSettings_Settings_RequireAuthorizationForCacheControl (43.25s)
--- PASS: TestAccAWSAPIGatewayRequestValidator_basic (33.05s)
--- PASS: TestAccAWSAPIGatewayMethod_basic (24.61s)
--- PASS: TestAccAWSAPIGatewayApiKey_importBasic (142.18s)
--- PASS: TestAccAWSAPIGatewayMethodSettings_Settings_UnauthorizedCacheControlHeaderStrategy (31.66s)
--- PASS: TestAccAWSAPIGatewayMethodSettings_basic (773.05s)
--- PASS: TestAccAWSAPIGatewayMethodSettings_Settings_CachingEnabled (817.51s)
--- PASS: TestAccAWSAPIGatewayDeployment_basic (729.53s)
--- PASS: TestAccAWSAPIGatewayMethodSettings_Settings_ThrottlingBurstLimit (30.70s)
--- PASS: TestAccAWSAPIGatewayRestApi_EndpointConfiguration_Private (905.15s)
--- PASS: TestAccAWSAPIGatewayDocumentationVersion_allFields (949.55s)
--- PASS: TestAccAWSAPIGatewayMethodSettings_Settings_ThrottlingRateLimit (132.41s)
--- FAIL: TestAccAWSAPIGatewayUsagePlan_importBasic (63.25s)
    testing.go:538: Step 0 error: Error applying: 1 error occurred:
          * aws_api_gateway_deployment.foo: 1 error occurred:
          * aws_api_gateway_deployment.foo: Error creating API Gateway Deployment: ConflictException: Another Deployment is in progress for rest api with id b4e9g4hos5 . Please try again later.
          status code: 409, request id: 63e82cc4-305e-11e9-925e-d98b99325ebe

--- FAIL: TestAccAWSAPIGatewayUsagePlanKey_basic (39.18s)
    testing.go:538: Step 0 error: Error applying: 1 error occurred:
          * aws_api_gateway_deployment.test: 1 error occurred:
          * aws_api_gateway_deployment.test: Error creating API Gateway Deployment: ConflictException: Another Deployment is in progress for rest api with id 2hi2taru0i . Please try again later.
          status code: 409, request id: 7dd99050-305e-11e9-9b22-11a3b27a8062

--- FAIL: TestAccAWSAPIGatewayUsagePlan_description (54.72s)
    testing.go:538: Step 0 error: Error applying: 1 error occurred:
          * aws_api_gateway_deployment.test: 1 error occurred:
          * aws_api_gateway_deployment.test: Error creating API Gateway Deployment: ConflictException: Another Deployment is in progress for rest api with id 5fkt8pa7g7 . Please try again later.
          status code: 409, request id: 93e8d7f7-305e-11e9-82a4-e7660a766498

--- PASS: TestAccAWSAPIGatewayMethodSettings_Settings_LoggingLevel (34.52s)
--- PASS: TestAccAWSAPIGatewayRestApi_openapi (19.54s)
--- PASS: TestAccAWSAPIGatewayStage_accessLogSettings (250.56s)
--- PASS: TestAccAWSAPIGatewayRestApi_policy (35.38s)
--- PASS: TestAccAWSAPIGatewayMethodSettings_Settings_Multiple (230.89s)
--- PASS: TestAccAWSAPIGatewayMethod_cognitoauthorizer (452.46s)
--- PASS: TestAccAWSAPIGatewayMethodSettings_Settings_CacheDataEncrypted (1269.68s)
--- PASS: TestAccAWSAPIGatewayAccount_basic (73.03s)
--- PASS: TestAccAWSAPIGatewayMethodSettings_Settings_DataTraceEnabled (115.51s)
--- PASS: TestAccAWSAPIGatewayUsagePlan_quota (47.29s)
--- PASS: TestAccAWSAPIGatewayRestApi_api_key_source (125.12s)
--- PASS: TestAccAWSAPIGatewayMethodResponse_basic (1374.18s)
--- PASS: TestAccAWSAPIGatewayMethodSettings_Settings_MetricsEnabled (455.52s)
--- FAIL: TestAccAWSAPIGatewayUsagePlan_productCode (62.26s)
    testing.go:538: Step 0 error: Error applying: 1 error occurred:
          * aws_api_gateway_deployment.test: 1 error occurred:
          * aws_api_gateway_deployment.test: Error creating API Gateway Deployment: ConflictException: Another Deployment is in progress for rest api with id ehysmxxa8d . Please try again later.
          status code: 409, request id: 7a57ee18-305f-11e9-b5ad-518807ce17fb

--- PASS: TestAccAWSAPIGatewayUsagePlan_apiStages (63.55s)
--- PASS: TestAccAWSAPIGatewayBasePathMapping_BasePath_Empty (1372.28s)
--- PASS: TestAccAWSAPIGatewayUsagePlan_basic (61.01s)
--- PASS: TestAccAWSAPIGatewayRestApi_EndpointConfiguration (1191.02s)
--- PASS: TestAccAWSAPIGatewayDeployment_Variables (1566.24s)
--- PASS: TestAccAWSAPIGatewayIntegration_integrationType (1637.29s)
--- PASS: TestAccAWSAPIGatewayStage_basic (575.05s)
--- PASS: TestAccAWSAPIGatewayRestApi_basic (1161.13s)
--- PASS: TestAccAWSAPIGatewayMethod_customrequestvalidator (996.26s)
--- PASS: TestAccAWSAPIGatewayUsagePlan_throttlingInitialRateLimit (390.33s)
--- PASS: TestAccAWSAPIGatewayUsagePlan_throttling (480.69s)
--- PASS: TestAccAWSAPIGatewayModel_basic (1131.61s)
--- PASS: TestAccAWSAPIGatewayDocumentationPart_method (1892.44s)
--- PASS: TestAccAWSAPIGatewayDeployment_StageName_Empty (1877.46s)
--- PASS: TestAccAWSAPIGatewayVpcLink_importBasic (657.93s)
--- PASS: TestAccAWSAPIGatewayAuthorizer_authTypeValidation (1752.95s)
--- PASS: TestAccAWSAPIGatewayVpcLink_basic (684.31s)
--- PASS: TestAccAWSAPIGatewayIntegration_cache_key_parameters (2028.42s)
--- PASS: TestAccAWSAPIGatewayMethodSettings_Settings_CacheTtlInSeconds (2110.39s)
--- PASS: TestAccAWSAPIGatewayResource_basic (881.84s)
--- PASS: TestAccAWSAPIGatewayDocumentationPart_responseHeader (2282.80s)

…urce deletion

AWS API Gateway currently has a 1 `DeleteRestApi`/minute rate limit and when exceeding that returns `429 TooManyRequests` errors which are automatically retried by the AWS Go SDK. The resource was artificially timeboxing the `DeleteRestApi` API call to 10 minutes previously with no `RetryableError` condition.

When running with parallelism over 10 (e.g. the acceptance testing runs at 20), there was a very high probability that one or more resource deletions would fail with the following error since the AWS Go SDK does not return the 429 error back to the calling code while retrying:

```
Error: errors during apply: timeout while waiting for state to become 'success' (timeout: 10m0s)
```

Removing the `Retry()` wrapper means the resource will retry the `DeleteRestApi` request without time limitations up to the configured provider `max_retries` (AWS Go SDK `MaxRetries`), which defaults to 25 and its exponential backoff generally lasts up to an hour.

A previous full `TestAccAWSAPIGateway` (76 tests currently) acceptance test run resulted in 16 failures due to the timeout. Rerunning with this change yields no timeout related failures where anything over ~600 seconds would have failed previously. The `TestAccAWSAPIGatewayUsagePlan` failures are unrelated and will require separate addressing.

```
--- PASS: TestAccAWSAPIGatewayAccount_importBasic (9.38s)
--- PASS: TestAccAWSAPIGatewayClientCertificate_basic (15.14s)
--- PASS: TestAccAWSAPIGatewayDocumentationPart_basic (21.81s)
--- PASS: TestAccAWSAPIGatewayDomainName_CertificateName (57.40s)
--- PASS: TestAccAWSAPIGatewayDocumentationVersion_importBasic (64.03s)
--- PASS: TestAccAWSAPIGatewayIntegrationResponse_basic (93.82s)
--- PASS: TestAccAWSAPIGatewayAuthorizer_cognito (31.27s)
--- PASS: TestAccAWSAPIGatewayDocumentationPart_importBasic (115.89s)
--- PASS: TestAccAWSAPIGatewayGatewayResponse_basic (187.30s)
--- PASS: TestAccAWSAPIGatewayBasePathMapping_basic (30.50s)
--- PASS: TestAccAWSAPIGatewayClientCertificate_importBasic (9.61s)
--- PASS: TestAccAWSAPIGatewayIntegration_basic (245.51s)
--- PASS: TestAccAWSAPIGatewayDeployment_StageDescription (31.08s)
--- PASS: TestAccAWSAPIGatewayDeployment_createBeforeDestoryUpdate (32.20s)
--- PASS: TestAccAWSAPIGatewayMethod_customauthorizer (31.91s)
--- PASS: TestAccAWSAPIGatewayIntegration_contentHandling (370.99s)
--- PASS: TestAccAWSAPIGatewayAuthorizer_switchAuthType (58.16s)
--- PASS: TestAccAWSAPIGatewayApiKey_basic (67.22s)
--- PASS: TestAccAWSAPIGatewayDeployment_Description (25.98s)
--- PASS: TestAccAWSAPIGatewayDocumentationVersion_importAllFields (517.75s)
--- PASS: TestAccAWSAPIGatewayAuthorizer_basic (45.30s)
--- PASS: TestAccAWSAPIGatewayResource_update (33.81s)
--- PASS: TestAccAWSAPIGatewayDocumentationVersion_basic (600.43s)
--- PASS: TestAccAWSAPIGatewayMethodSettings_Settings_RequireAuthorizationForCacheControl (43.25s)
--- PASS: TestAccAWSAPIGatewayRequestValidator_basic (33.05s)
--- PASS: TestAccAWSAPIGatewayMethod_basic (24.61s)
--- PASS: TestAccAWSAPIGatewayApiKey_importBasic (142.18s)
--- PASS: TestAccAWSAPIGatewayMethodSettings_Settings_UnauthorizedCacheControlHeaderStrategy (31.66s)
--- PASS: TestAccAWSAPIGatewayMethodSettings_basic (773.05s)
--- PASS: TestAccAWSAPIGatewayMethodSettings_Settings_CachingEnabled (817.51s)
--- PASS: TestAccAWSAPIGatewayDeployment_basic (729.53s)
--- PASS: TestAccAWSAPIGatewayMethodSettings_Settings_ThrottlingBurstLimit (30.70s)
--- PASS: TestAccAWSAPIGatewayRestApi_EndpointConfiguration_Private (905.15s)
--- PASS: TestAccAWSAPIGatewayDocumentationVersion_allFields (949.55s)
--- PASS: TestAccAWSAPIGatewayMethodSettings_Settings_ThrottlingRateLimit (132.41s)
--- FAIL: TestAccAWSAPIGatewayUsagePlan_importBasic (63.25s)
    testing.go:538: Step 0 error: Error applying: 1 error occurred:
          * aws_api_gateway_deployment.foo: 1 error occurred:
          * aws_api_gateway_deployment.foo: Error creating API Gateway Deployment: ConflictException: Another Deployment is in progress for rest api with id b4e9g4hos5 . Please try again later.
          status code: 409, request id: 63e82cc4-305e-11e9-925e-d98b99325ebe

--- FAIL: TestAccAWSAPIGatewayUsagePlanKey_basic (39.18s)
    testing.go:538: Step 0 error: Error applying: 1 error occurred:
          * aws_api_gateway_deployment.test: 1 error occurred:
          * aws_api_gateway_deployment.test: Error creating API Gateway Deployment: ConflictException: Another Deployment is in progress for rest api with id 2hi2taru0i . Please try again later.
          status code: 409, request id: 7dd99050-305e-11e9-9b22-11a3b27a8062

--- FAIL: TestAccAWSAPIGatewayUsagePlan_description (54.72s)
    testing.go:538: Step 0 error: Error applying: 1 error occurred:
          * aws_api_gateway_deployment.test: 1 error occurred:
          * aws_api_gateway_deployment.test: Error creating API Gateway Deployment: ConflictException: Another Deployment is in progress for rest api with id 5fkt8pa7g7 . Please try again later.
          status code: 409, request id: 93e8d7f7-305e-11e9-82a4-e7660a766498

--- PASS: TestAccAWSAPIGatewayMethodSettings_Settings_LoggingLevel (34.52s)
--- PASS: TestAccAWSAPIGatewayRestApi_openapi (19.54s)
--- PASS: TestAccAWSAPIGatewayStage_accessLogSettings (250.56s)
--- PASS: TestAccAWSAPIGatewayRestApi_policy (35.38s)
--- PASS: TestAccAWSAPIGatewayMethodSettings_Settings_Multiple (230.89s)
--- PASS: TestAccAWSAPIGatewayMethod_cognitoauthorizer (452.46s)
--- PASS: TestAccAWSAPIGatewayMethodSettings_Settings_CacheDataEncrypted (1269.68s)
--- PASS: TestAccAWSAPIGatewayAccount_basic (73.03s)
--- PASS: TestAccAWSAPIGatewayMethodSettings_Settings_DataTraceEnabled (115.51s)
--- PASS: TestAccAWSAPIGatewayUsagePlan_quota (47.29s)
--- PASS: TestAccAWSAPIGatewayRestApi_api_key_source (125.12s)
--- PASS: TestAccAWSAPIGatewayMethodResponse_basic (1374.18s)
--- PASS: TestAccAWSAPIGatewayMethodSettings_Settings_MetricsEnabled (455.52s)
--- FAIL: TestAccAWSAPIGatewayUsagePlan_productCode (62.26s)
    testing.go:538: Step 0 error: Error applying: 1 error occurred:
          * aws_api_gateway_deployment.test: 1 error occurred:
          * aws_api_gateway_deployment.test: Error creating API Gateway Deployment: ConflictException: Another Deployment is in progress for rest api with id ehysmxxa8d . Please try again later.
          status code: 409, request id: 7a57ee18-305f-11e9-b5ad-518807ce17fb

--- PASS: TestAccAWSAPIGatewayUsagePlan_apiStages (63.55s)
--- PASS: TestAccAWSAPIGatewayBasePathMapping_BasePath_Empty (1372.28s)
--- PASS: TestAccAWSAPIGatewayUsagePlan_basic (61.01s)
--- PASS: TestAccAWSAPIGatewayRestApi_EndpointConfiguration (1191.02s)
--- PASS: TestAccAWSAPIGatewayDeployment_Variables (1566.24s)
--- PASS: TestAccAWSAPIGatewayIntegration_integrationType (1637.29s)
--- PASS: TestAccAWSAPIGatewayStage_basic (575.05s)
--- PASS: TestAccAWSAPIGatewayRestApi_basic (1161.13s)
--- PASS: TestAccAWSAPIGatewayMethod_customrequestvalidator (996.26s)
--- PASS: TestAccAWSAPIGatewayUsagePlan_throttlingInitialRateLimit (390.33s)
--- PASS: TestAccAWSAPIGatewayUsagePlan_throttling (480.69s)
--- PASS: TestAccAWSAPIGatewayModel_basic (1131.61s)
--- PASS: TestAccAWSAPIGatewayDocumentationPart_method (1892.44s)
--- PASS: TestAccAWSAPIGatewayDeployment_StageName_Empty (1877.46s)
--- PASS: TestAccAWSAPIGatewayVpcLink_importBasic (657.93s)
--- PASS: TestAccAWSAPIGatewayAuthorizer_authTypeValidation (1752.95s)
--- PASS: TestAccAWSAPIGatewayVpcLink_basic (684.31s)
--- PASS: TestAccAWSAPIGatewayIntegration_cache_key_parameters (2028.42s)
--- PASS: TestAccAWSAPIGatewayMethodSettings_Settings_CacheTtlInSeconds (2110.39s)
--- PASS: TestAccAWSAPIGatewayResource_basic (881.84s)
--- PASS: TestAccAWSAPIGatewayDocumentationPart_responseHeader (2282.80s)
```
@bflad bflad added the service/apigateway Issues and PRs that pertain to the apigateway service. label Feb 14, 2019
@bflad bflad requested a review from a team February 14, 2019 14:08
@ghost ghost added the size/XS Managed by automation to categorize the size of a PR. label Feb 14, 2019
@bflad bflad added this to the v1.59.0 milestone Feb 14, 2019
@bflad bflad merged commit fb42c0d into master Feb 14, 2019
@bflad bflad deleted the t-apigateway-timeout branch February 14, 2019 22:01
bflad added a commit that referenced this pull request Feb 14, 2019
@bflad
Copy link
Contributor Author

bflad commented Feb 14, 2019

This has been released in version 1.59.0 of the AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

@ghost
Copy link

ghost commented Mar 31, 2020

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!

@ghost ghost locked and limited conversation to collaborators Mar 31, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
service/apigateway Issues and PRs that pertain to the apigateway service. size/XS Managed by automation to categorize the size of a PR.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants