Unable to find matching route for Route Table (rtb-xxxxx) and destination CIDR block (x.x.x.x/x) #13138
Comments
github-actions
bot
added
the
needs-triage
|
New findings: Create route request was sent at 14:16:18 GMT. After that terraform performs this operation: AWS API response time is 1 second behind then request before (ec2/CreateRoute): 14:16:17 GMT Also It's strange that retry call is not executed in this case |
|
There have been other reports of this issue going back to 2016, but, it looks like this very intermittent issue can, as per my colleague's initial report above, still occur. We have seen it happen for a few hrs in a row, e.g., 2 in 10 runs, with repeated apply & destroy cycles, and then no errors for hours or days afterward without any changes to the TF code. Older references to issue can be found here: hashicorp/terraform#8542 Some of the previous issues, as per the above link (and related links in that issue) stated that the problem was mitigated by adding a create timeout for the route, but, we have > 5m timeout set, and this does not stop the issue from intermittently occurring. We are now testing with updated TF code that explicitly delays the creation of the routes, until all the other VPC resources are created, e.g., separate TF run to create routes after VPC and related resource creation is completed, e.g., after VPC, NAT, IG and subnets etc are created. |
|
Update: aws_route resource sometimes crashes while describing route tables with error (due to aws api latency): Unable to find matching route for Route Table (rtb-08f8b12546e9e1ed8) and destination CIDR block (0.0.0.0/0). However most of the cases, it's crashing due to latency in aws api, route is not visible in aws yet and it's trying to read route tables already and crashes without finding it on the first try. If retry would be implemented on this case, this could allow to recover after this error and it won't stop terraform run with an error. (now it just gives an error and doesn't retry) |
|
Re: previous comment by my colleague, to be very clear, we are not seeing a "crash", we are seeing the following error:
This in turn causes our apply to fail As indicated previously in this thread, we can see a successful response after the route is created, but, the immediate call to describe route tables, produces a response which does not have the newly created route in the RouteSet section of the response, this in turn causes the error referenced above. See request/response snippet below (Route Set should have two entries, not one). Also noted previously, we’ve have seen cases where the request to describe route tables is out of sync with the route creation, e.g., describe route tables before create route call, though, it usually appears to be in proper order/sequence, but the latency causes an error due to lack of retries of the describe route table call Again, this issue is very intermittent. Running continuous automated tests over the last 48 hrs, e.g., with repeated, apply/destroy cycles, we have seen no issues for 8-10+ hrs, but then the error will resurface and it can take up to 1.5+ hrs before we can get a successful apply. We've also seen cases, where we recover after a few cycles. It would appear that a Terraform AWS Provider fix to allow retries for the "describe route tables" call, immediately following the route creation, may resolve this issue. |
|
We created a fork, and resolved the issue. The fix was affected by adding a retry of resourceAwsRouteFindRoute -after- the route creation to confirm the route is in the route table. We've been running automated testing that repeatedly applies and destroys our infrastructure, and have seen no further errors since implementing this fix. Here is the code snippet from resource_aws_route.go. |
|
Has this fix been submitted as a PR yet? |
|
No. |
|
We've been increasingly running into this issue. We also have a CI that repeatedly applies/destroys infrastructure, and it's been failing about 1 out of 3 times due to this exact error. Worst part is that simply retrying to deploy doesn't work as Terraform tries to create the route again on the second go, but fails as it already exists. First go: On retry: |
|
We posted a solution above. As per previous comment, this required forking the code and building a custom plugin. As also noted, above, we have not yet submitted a PR. |
|
After looking at the code in question, I saw what the issue is. When |
bflad
added
bug
|
The fix for this has been merged and will release with version 2.67.0 of the Terraform AWS Provider, later this week. Thanks to @alewando for the implementation. 👍 |
|
This has been released in version 2.67.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading. For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template for triage. Thanks! |
|
I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks! |
ghost
locked and limited conversation to collaborators
Community Note
Terraform Version
Terraform v0.12.24
Affected Resource(s)
Terraform Configuration Files
Debug Output
Panic Output
Expected Behavior
Route was found and added to route table. Retry for Attempting to find route in route table is not happening like it should.
Actual Behavior
Route was not found:
Unable to find matching route for Route Table (rtb-0c6b2875dd3a2331b) and destination CIDR block (0.0.0.0/0).
Steps to Reproduce
Issue is not reproducible easily, it comes sometimes only. I think there are some issues with AWS API. Increasing timeout for aws_route does not solve the problem. It goes away by itself after few hours.
terraform applyImportant Factoids
Retry for Attempting to find route in route table is not happening like it should.
References
The text was updated successfully, but these errors were encountered: