Failure while trying to add more routes to a route table.

[feanil]

Hi there,

Thank you for opening an issue. Please note that we try to keep the Terraform issue tracker reserved for bug reports and feature requests. For general usage questions, please see: https://www.terraform.io/community.html.

Terraform Version

Terraform v0.6.16

Affected Resource(s)

Please list the resources as a list, for example:

• aws_route_table
• aws_route

If this issue appears to affect multiple resources, it may be an issue with Terraform's core, so please mention this.

Terraform Configuration Files

# Copy-paste your Terraform configurations here - for large Terraform configs,
# please use a service like Dropbox and share a link to the ZIP file. For
# security, you can also encrypt the files using our GPG public key.

Debug Output

Error applying plan:

3 error(s) occurred:

* aws_route.route_to_peers.0: diffs didn't match during apply. This is a bug with Terraform and should be reported as a GitHub Issue.

Please include the following information in your report:

    Terraform Version: 0.6.16
    Resource ID: aws_route.route_to_peers.0
    Mismatch reason: attribute mismatch: vpc_peering_connection_id
    Diff One (usually from plan): *terraform.InstanceDiff{Attributes:map[string]*terraform.ResourceAttrDiff{"vpc_peering_connection_id":*terraform.ResourceAttrDiff{Old:"pcx-8a873ee3", New:"${element(aws_vpc_peering_connection.peering_connections.*.id, count.index)}", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Type:0x0}}, Destroy:false, DestroyTainted:false}
    Diff Two (usually from apply): *terraform.InstanceDiff{Attributes:map[string]*terraform.ResourceAttrDiff(nil), Destroy:false, DestroyTainted:false}

Also include as much context as you can about your config, state, and the steps you performed to trigger this error.

* aws_route.route_to_peers.1: diffs didn't match during apply. This is a bug with Terraform and should be reported as a GitHub Issue.

Please include the following information in your report:

    Terraform Version: 0.6.16
    Resource ID: aws_route.route_to_peers.1
    Mismatch reason: attribute mismatch: vpc_peering_connection_id
    Diff One (usually from plan): *terraform.InstanceDiff{Attributes:map[string]*terraform.ResourceAttrDiff{"vpc_peering_connection_id":*terraform.ResourceAttrDiff{Old:"pcx-495de520", New:"${element(aws_vpc_peering_connection.peering_connections.*.id, count.index)}", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Type:0x0}}, Destroy:false, DestroyTainted:false}
    Diff Two (usually from apply): *terraform.InstanceDiff{Attributes:map[string]*terraform.ResourceAttrDiff(nil), Destroy:false, DestroyTainted:false}

Also include as much context as you can about your config, state, and the steps you performed to trigger this error.

* aws_route.route_to_peers.2: 
error finding matching route for Route table (rtb-924fe6f5) and destination CIDR block (10.3.0.0/16)

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

Panic Output

If Terraform produced a panic, please provide a link to a GitHub Gist containing the output of the crash.log.

Expected Behavior

New routes created.

Actual Behavior

All routes got created but one route was created but not added to the state file.

[jniesen]

I just ran into a similar issue it looks like.

My plan shows that I have to routes left to create.

+ module.peer_foo_to_bar_prod.aws_route.private_peer_routes
    destination_cidr_block:     "" => "[bar_prod_cidr]"
    destination_prefix_list_id: "" => "<computed>"
    instance_owner_id:          "" => "<computed>"
    origin:                     "" => "<computed>"
    route_table_id:             "" => "rtb-r1xxxxxx"
    state:                      "" => "<computed>"
    vpc_peering_connection_id:  "" => "pcx-cfxxxxxx"

+ module.peer_foo_to_bar_dev.aws_route.private_peer_routes
    destination_cidr_block:     "" => "[bar_dev_cidr]"
    destination_prefix_list_id: "" => "<computed>"
    instance_owner_id:          "" => "<computed>"
    origin:                     "" => "<computed>"
    route_table_id:             "" => "rtb-r1xxxxxx"
    state:                      "" => "<computed>"
    vpc_peering_connection_id:  "" => "pcx-caxxxxxx"

Our apply error output shows that we cannot create these routes because they already exist.

2 error(s) occurred:

* aws_route.private_peer_routes: Error creating route: RouteAlreadyExists: The route identified by [bar_dev_cidr] already exists.
    status code: 400, request id: 
* aws_route.private_peer_routes: Error creating route: RouteAlreadyExists: The route identified by [bar_prod_cidr] already exists.
    status code: 400, request id: 

I scrubbed out statefile and left the relevant parts:
https://gist.github.com/jniesen/1d31d61db2e440bd404863d482a63693

I left the "peer_foo_to_bop" state in the statefile as an example of the expected behavior.

[catsby]

@jniesen that error indicates routes already exist on that Route Table for those cidr blocks, can you view them in the AWS web console? They would need to be removed there first, then applied via Terraform if you want Terraform to track them.

If you hit something where you attempted to add them, it failed for some reason(s) , and then it failed again because they were already there, that would be different.

[catsby]

@feanil hrm, this issue is odd... do you have a configuration that reproduces this? Looking at the diff contents:

Diff One (usually from plan): *terraform.InstanceDiff{Attributes:map[string]*terraform.ResourceAttrDiff{"vpc_peering_connection_id":*terraform.ResourceAttrDiff{Old:"pcx-495de520",
 New:"${element(aws_vpc_peering_connection.peering_connections.*.id,
 count.index)}",
 NewComputed:false,
 NewRemoved:false,
 NewExtra:interface {}(nil),
 RequiresNew:false,
 Type:0x0}},
 Destroy:false,
 DestroyTainted:false}


Diff Two (usually from apply): *terraform.InstanceDiff{Attributes:map[string]*terraform.ResourceAttrDiff(nil),
 Destroy:false,
 DestroyTainted:false}

There's practically nothing in the apply diff. I'd need more information to dig into this, can you shed some light on any steps that went into this? Any config you can share (minus secrets of course) would help!

[jniesen]

@catsby That's the weird thing. They didn't exist until I applied the Terraform a first time. Terraform created them, but didn't record them in the statefile.

I have a module called vpc_peers that creates a vpc_peering_connection, and entries in a route_table that point to the peer connection.

This is the main.tf from the module. It's hosted on a GHE instance so I can't provide a link.

resource "aws_vpc_peering_connection" "peer" {
  peer_owner_id = "${var.peer_aws_account}"
  peer_vpc_id   = "${var.peer_vpc_id}"
  vpc_id        = "${var.vpc_id}"
}

resource "aws_route" "private_peer_routes" {
  count = "${length(compact(split(",", var.peer_private_routes)))}"

  route_table_id            = "${var.private_route_table_id}"
  destination_cidr_block    = "${element(split(",", var.peer_private_routes), count.index)}"
  vpc_peering_connection_id = "${aws_vpc_peering_connection.peer.id}"
}

resource "aws_route" "public_peer_routes" {
  count = "${length(compact(split(",", var.peer_public_routes)))}"

  route_table_id            = "${var.public_route_table_id}"
  destination_cidr_block    = "${element(split(",", var.peer_public_routes), count.index)}"
  vpc_peering_connection_id = "${aws_vpc_peering_connection.peer.id}"
}

If you look at the statefile, you can see one of these modules being applied and recorded in the statefile correctly.

Then you can see another one of these modules being applied, but not recorded in the statefile correctly. I say it was applied be the the route table does show a record of the route being recorded.

[catsby]

Are you by chance using any in-line routes declared in an aws_route_table resource, as well as aws_route resources for the same Route Table? (those are incompatible)

[jniesen]

@catsby no. I saw in the documentation that it recommended against using inline and aws_route resources at the same time. Also, since the VPC and route tables are being created with the use of the tf_aws_vpc community module, I didn't have a choice but to use aws_route resources.

All of these things are being created by the community module.

Sorry all of these.

[wr0ngway]

I'm having a similar issue with 0.6.16
Setting up vpc peering routes on the receiver side of a cross account peering. Adding a peering route to 4 route-tables, 2 succeeded, 2 failed.

For the first apply, I had forgotten to accept the peering request, so that may have caused the first failure, which was:

* aws_route.vpc-peering-route.1: 
error finding matching route for Route table (rtb-cabxxxx) and destination CIDR block (10.10.0.0/16)
* aws_route.vpc-peering-route.2: 
error finding matching route for Route table (rtb-cbbxxxx) and destination CIDR block (10.10.0.0/16)

I accepted the peering, then tried again, but got:

* aws_route.vpc-peering-route.2: Error creating route: RouteAlreadyExists: The route identified by 10.10.0.0/16 already exists.
status code: 400, request id: f4123f2a-0c93-42ae-a85f-7b4601179951
* aws_route.vpc-peering-route.1: Error creating route: RouteAlreadyExists: The route identified by 10.10.0.0/16 already exists.
status code: 400, request id: 2470dd4c-1870-40da-b8f4-1a0510609394

I checked the aws console, the routes did end up in the route table, so I removed them again. Applyed again and saw the first failure again (and routes were back in aws console)

* aws_route.vpc-peering-route.1: 
error finding matching route for Route table (rtb-cabxxxx) and destination CIDR block (10.10.0.0/16)
* aws_route.vpc-peering-route.2: 
error finding matching route for Route table (rtb-cbbxxxx) and destination CIDR block (10.10.0.0/16)

[wr0ngway]

Not sure if it matters, but I'm passing the route-table IDs down into a module, which then adds the peering routes to those tables

[bkc1]

I am having this issue as well which is intermittent. I have VPC peering and several aws_route blocks adding routes to the main route table. Like @wr0ngway, I am passing the ID of an existing route-table into my terraform project. I verified via the AWS console that terraform creates the routes in the existing table but errors with:

* aws_route.from_PM_TSM2:
error finding matching route for Route table (rtb-7b00xxxx) and destination CIDR block (10.0.2.0/24)
* aws_route.from_PM_TSM1:
error finding matching route for Route table (rtb-7b00xxxx) and destination CIDR block (10.0.1.0/24)

If I run a terraform apply a second time, it errors with the following due to the resource already being created:

* aws_route.from_PM_TSM1: Error creating route: RouteAlreadyExists: The route identified by 10.0.1.0/24 already exists.
        status code: 400, request id: d6a74485-29e7-4085-955a-xxxxxxxxx
* aws_route.from_PM_TSM2: Error creating route: RouteAlreadyExists: The route identified by 10.0.2.0/24 already exists.
        status code: 400, request id: 9a931d74-8ef9-4c74-89aa-xxxxxxxx

[jrnt30]

Yes, the intermittent issue is most likely due to the issue that I fixed in the open PR #6912

[brikis98]

Same problem here with route tables and peering connections. Unlike other Terraform bugs that appear intermittently, once this one appears, re-running apply does NOT fix it. You basically have to delete the routes manually or run destroy, which is much worse.

[mitchellh]

Hello! This should be fixed in later versions of Terraform (0.7.11 at time of writing).

Its hard to know 100% for a couple reasons:

• This is on a now much-older version of Terraform. This is our fault, not yours, for not trying sooner, but we've fixed many diff mismatch issues since then.
• We've fixed issues with exact error messages as this in those versions.

Please try again! If the issue persists please just open a new issue. Thanks!

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.