Add configurable timeouts to aws_route #3639
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
bflad
added
enhancement
bflad
approved these changes
Mar 6, 2018
There was a problem hiding this comment.
LGTM! Thanks for this. 🚀
9 tests passed (all tests)
=== RUN TestAccAWSRoute_basic
--- PASS: TestAccAWSRoute_basic (103.12s)
=== RUN TestAccAWSRoute_doesNotCrashWithVPCEndpoint
--- PASS: TestAccAWSRoute_doesNotCrashWithVPCEndpoint (103.82s)
=== RUN TestAccAWSRoute_ipv6ToPeeringConnection
--- PASS: TestAccAWSRoute_ipv6ToPeeringConnection (109.02s)
=== RUN TestAccAWSRoute_ipv6ToInternetGateway
--- PASS: TestAccAWSRoute_ipv6ToInternetGateway (117.20s)
=== RUN TestAccAWSRoute_changeCidr
--- PASS: TestAccAWSRoute_changeCidr (122.23s)
=== RUN TestAccAWSRoute_ipv6Support
--- PASS: TestAccAWSRoute_ipv6Support (154.06s)
=== RUN TestAccAWSRoute_ipv6ToInstance
--- PASS: TestAccAWSRoute_ipv6ToInstance (197.54s)
=== RUN TestAccAWSRoute_noopdiff
--- PASS: TestAccAWSRoute_noopdiff (204.66s)
=== RUN TestAccAWSRoute_ipv6ToNetworkInterface
--- PASS: TestAccAWSRoute_ipv6ToNetworkInterface (248.34s)
bflad
added a commit
that referenced
this pull request
Mar 6, 2018
|
This has been released in version 1.11.0 of the AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading. |
wking
added a commit
to wking/openshift-installer
that referenced
this pull request
Apr 26, 2019
Using [1,2,3,4,5], both of which were added in v1.11, so we have them
in our v2.2 AWS provider. This should mitigate some of the issues
we've been having in our busy CI account, where out of ~1150 jobs in
the last 24 hours, we've had the following failures [6]:
$ curl -s 'http://localhost:8000/search?name=-e2e-aws&.&q=level%3Derror.*timeout+while+waiting+for+state' | jq -r '. | to_entries[].value[] | to_entries[].value[]' | sed 's/(i-[^)]*/(i-.../;s/(igw-[^)]*/(igw-.../;s/\(master\|nat_gw\|private_routing\|route_net\)\.[0-9]/\1.../' | sort | uniq -c | sort -n
2 level=error msg="\t* aws_instance.master...: Error waiting for instance (i-...) to become ready: timeout while waiting for state
10 level=error msg="\t* aws_security_group.bootstrap: timeout while waiting for state
38 level=error msg="\t* aws_route.igw_route: Error creating route: timeout while waiting for state
58 level=error msg="\t* aws_internet_gateway.igw: error attaching EC2 Internet Gateway (igw-...): timeout while waiting for state
76 level=error msg="\t* aws_route_table_association.private_routing...: timeout while waiting for state
90 level=error msg="\t* aws_route_table_association.route_net...: timeout while waiting for state
164 level=error msg="\t* aws_route.to_nat_gw...: Error creating route: timeout while waiting for state
The 20 minute timeout is much higher than the two-minute route default
[2], so that should help a lot with our leading error. The security
group default is 10 minutes [4], so this is less of change there, and
we only see that error rarely anyway. I went with 20 minutes (instead
of a higher number), because a single resource (or parallel resources)
coming in just under that range will keep the full Terraform step
under the 30 minutes that we've chosen as a timeout for our other
steps (waiting for the Kubernetes API, bootstrap completion, and
install completion. But obviously we can tune more later if
necessary.
[1]: https://www.terraform.io/docs/configuration/resources.html#operation-timeouts
[2]: https://www.terraform.io/docs/providers/aws/r/route.html#timeouts
[3]: hashicorp/terraform-provider-aws#3639 (v1.11.0)
[4]: https://www.terraform.io/docs/providers/aws/r/security_group.html#timeouts
[5]: hashicorp/terraform-provider-aws#3599 (v1.11.0)
[6]: https://github.com/wking/openshift-release/tree/debug-scripts/d3
|
I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks! |
ghost
locked and limited conversation to collaborators
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Turns out the 2 minutes timeout for route creation is pretty tight as we are hitting it quite a lot lately.
Instead of bumping it I just made it configurable for others.
Cheers!