plan is in a loop changing my "main" route table

[hashibot]

This issue was originally opened by @egarbi as hashicorp/terraform#2879. It was migrated here as part of the provider split. The original body of the issue is below.

---

$ terraform --version
 Terraform v0.6.1

# Main routing table private to nat
resource "aws_route_table" "main" {
   route {
       cidr_block = "0.0.0.0/0"
       network_interface_id = "eni-7aXXXX"
   }
   route {
       cidr_block = "${var.vpn_cidr}"
       instance_id = "${module.vpn.minion_id}"
   }
   route {
       cidr_block = "${lookup(var.tokio_vpc_cidr,var.env)}"
       instance_id = "${module.vpn-gw.id}"
   }
   route {
       cidr_block = "${lookup(var.usa_vpc_cidr,var.env)}"
       instance_id = "${module.gw-free.id}"
   }
   tags {
       Name = "${lookup(var.domain,var.env)}-vpc-to-nat"
   }
   vpc_id = "${aws_vpc.testing.id}"
}

terraform plan...

~ aws_route_table.main
    route.1714403534.cidr_block:                "10.32.0.0/16" => "10.32.0.0/16"
    route.1714403534.gateway_id:                "" => ""
    route.1714403534.instance_id:               "i-64XXXX" => "i-64XXXX"
    route.1714403534.network_interface_id:      "eni-2aXXXX" => ""
    route.1714403534.vpc_peering_connection_id: "" => ""
    route.2010688411.cidr_block:                "10.52.0.0/16" => "10.52.0.0/16"
    route.2010688411.gateway_id:                "" => ""
    route.2010688411.instance_id:               "i-36XXXX" => "i-36XXXX"
    route.2010688411.network_interface_id:      "eni-a7XXXX" => ""
    route.2010688411.vpc_peering_connection_id: "" => ""
    route.3093398733.cidr_block:                "192.168.240.0/24" => "192.168.240.0/24"
    route.3093398733.gateway_id:                "" => ""
    route.3093398733.instance_id:               "i-d5XXXX" => "i-d52XXXX"
    route.3093398733.network_interface_id:      "eni-69XXXX" => ""
    route.3093398733.vpc_peering_connection_id: "" => ""
    route.3891333365.cidr_block:                "0.0.0.0/0" => ""
    route.3891333365.gateway_id:                "" => ""
    route.3891333365.instance_id:               "i-0fXXXX" => ""
    route.3891333365.network_interface_id:      "eni-7aXXXX" => ""
    route.3891333365.vpc_peering_connection_id: "" => ""
    route.3991494531.cidr_block:                "" => "0.0.0.0/0"
    route.3991494531.gateway_id:                "" => ""
    route.3991494531.instance_id:               "" => ""
    route.3991494531.network_interface_id:      "" => "eni-7aXXXX"
    route.3991494531.vpc_peering_connection_id: "" => ""

[hashibot]

This comment was originally opened by @jszwedko as hashicorp/terraform#2879 (comment). It was migrated here as part of the provider split. The original comment is below.

---

@catsby we are seeing this as well -- basically if you associate a route table with an ENI that is associated with an instance, AWS also returns the instance ID on subsequently queries for the route table which causes terraform to think that there is a diff. We took a crack at fixing this today, but the nuances around schema.TypeSet and how the diffs are determined prevented us from fixing it.

I think the correct behavior is to have the instance id be persisted to the state file, but not be considered a diff.

Here is a diff with a failing test case:

diff --git a/builtin/providers/aws/resource_aws_route_table_test.go b/builtin/providers/aws/resource_aws_route_table_test.go
index 6eb8951..060ea23 100644
--- a/builtin/providers/aws/resource_aws_route_table_test.go
+++ b/builtin/providers/aws/resource_aws_route_table_test.go
@@ -123,6 +123,46 @@ func TestAccAWSRouteTable_instance(t *testing.T) {
        })
 }

+func TestAccAWSRouteTable_networkInterface(t *testing.T) {
+       var v ec2.RouteTable
+
+       testCheck := func(*terraform.State) error {
+               if len(v.Routes) != 2 {
+                       return fmt.Errorf("bad routes: %#v", v.Routes)
+               }
+
+               routes := make(map[string]*ec2.Route)
+               for _, r := range v.Routes {
+                       routes[*r.DestinationCidrBlock] = r
+               }
+
+               if _, ok := routes["10.1.0.0/16"]; !ok {
+                       return fmt.Errorf("bad routes: %#v", v.Routes)
+               }
+               if _, ok := routes["10.2.0.0/16"]; !ok {
+                       return fmt.Errorf("bad routes: %#v", v.Routes)
+               }
+
+               return nil
+       }
+
+       resource.Test(t, resource.TestCase{
+               PreCheck:     func() { testAccPreCheck(t) },
+               Providers:    testAccProviders,
+               CheckDestroy: testAccCheckRouteTableDestroy,
+               Steps: []resource.TestStep{
+                       resource.TestStep{
+                               Config: testAccRouteTableConfigNetworkInterface,
+                               Check: resource.ComposeTestCheckFunc(
+                                       testAccCheckRouteTableExists(
+                                               "aws_route_table.foo", &v),
+                                       testCheck,
+                               ),
+                       },
+               },
+       })
+}
+
 func TestAccAWSRouteTable_tags(t *testing.T) {
        var route_table ec2.RouteTable

@@ -316,6 +356,41 @@ resource "aws_route_table" "foo" {
 }
 `

+const testAccRouteTableConfigNetworkInterface = `
+resource "aws_vpc" "foo" {
+       cidr_block = "10.1.0.0/16"
+}
+
+resource "aws_route_table" "foo" {
+       vpc_id = "${aws_vpc.foo.id}"
+
+       route {
+               cidr_block = "10.2.0.0/16"
+               network_interface_id = "${aws_network_interface.foo.id}"
+       }
+}
+
+resource "aws_network_interface" "foo" {
+  subnet_id = "${aws_subnet.foo.id}"
+  attachment {
+    instance = "${aws_instance.foo.id}"
+    device_index = 1
+  }
+}
+
+resource "aws_subnet" "foo" {
+       cidr_block = "10.1.1.0/24"
+       vpc_id = "${aws_vpc.foo.id}"
+}
+
+resource "aws_instance" "foo" {
+       # us-west-2
+       ami = "ami-4fccb37f"
+       instance_type = "m1.small"
+       subnet_id = "${aws_subnet.foo.id}"
+}
+`
+

[hashibot]

This comment was originally opened by @pidah as hashicorp/terraform#2879 (comment). It was migrated here as part of the provider split. The original comment is below.

---

I am running into this issue with eni's ->

~ aws_eip.nat_elastic_ip
instance: "i-358be594" => ""

~ aws_route_table.app-subnet
route.1972381572.cidr_block: "" => "0.0.0.0/0"
route.1972381572.gateway_id: "" => ""
route.1972381572.instance_id: "" => ""
route.1972381572.network_interface_id: "" => "eni-340f9c7d"
route.1972381572.vpc_peering_connection_id: "" => ""
route.3507476028.cidr_block: "0.0.0.0/0" => ""
route.3507476028.gateway_id: "" => ""
route.3507476028.instance_id: "i-358be594" => ""
route.3507476028.network_interface_id: "eni-340f9c7d" => ""
route.3507476028.vpc_peering_connection_id: "" => ""

~ aws_route_table.databases-subnet
route.1972381572.cidr_block: "" => "0.0.0.0/0"
route.1972381572.gateway_id: "" => ""
route.1972381572.instance_id: "" => ""
route.1972381572.network_interface_id: "" => "eni-340f9c7d"
route.1972381572.vpc_peering_connection_id: "" => ""
route.3507476028.cidr_block: "0.0.0.0/0" => ""
route.3507476028.gateway_id: "" => ""
route.3507476028.instance_id: "i-358be594" => ""
route.3507476028.network_interface_id: "eni-340f9c7d" => ""
route.3507476028.vpc_peering_connection_id: "" => ""

[hashibot]

This comment was originally opened by @gtmtech as hashicorp/terraform#2879 (comment). It was migrated here as part of the provider split. The original comment is below.

---

+1 Did you find a workaround @jszwedko @catsby ? We just ran into this too and we cant roll out an ENI now unless there is a workaround or this is solved.

[hashibot]

This comment was originally opened by @jszwedko as hashicorp/terraform#2879 (comment). It was migrated here as part of the provider split. The original comment is below.

---

@gtmtech unfortunately, no, in our case we just permit the route table update to go through.

[hashibot]

This comment was originally opened by @dsalazar-carecloud as hashicorp/terraform#2879 (comment). It was migrated here as part of the provider split. The original comment is below.

---

Just ran into this issue myself. Same scenario with using the eni parameter.

$ terraform --version
Terraform v0.6.3

[hashibot]

This comment was originally opened by @TimAtLassian as hashicorp/terraform#2879 (comment). It was migrated here as part of the provider split. The original comment is below.

---

Still getting this behaviour in v0.6.6

[hashibot]

This comment was originally opened by @alexsapran as hashicorp/terraform#2879 (comment). It was migrated here as part of the provider split. The original comment is below.

---

+1 to find a solution. We are facing the same issue.

[hashibot]

This comment was originally opened by @icflournoy as hashicorp/terraform#2879 (comment). It was migrated here as part of the provider split. The original comment is below.

---

+1 I believe I'm facing the same issue.

$ terraform version
Terraform v0.6.9

Part of the output I get every time I run plan or apply. This is for the default route table as well as two secondary route tables.

~ aws_route_table.private-b
    route.1851933015.cidr_block:                "" => "10.10.0.0/16"
    route.1851933015.gateway_id:                "" => "pcx-7f6bcf16"
    route.1851933015.instance_id:               "" => ""
    route.1851933015.nat_gateway_id:            "" => ""
    route.1851933015.network_interface_id:      "" => ""
    route.1851933015.vpc_peering_connection_id: "" => ""
    route.2195099901.cidr_block:                "0.0.0.0/0" => ""
    route.2195099901.gateway_id:                "" => ""
    route.2195099901.instance_id:               "" => ""
    route.2195099901.nat_gateway_id:            "nat-051241cf3fbd466e5" => ""
    route.2195099901.network_interface_id:      "" => ""
    route.2195099901.vpc_peering_connection_id: "" => ""
    route.2675475309.cidr_block:                "10.10.0.0/16" => ""
    route.2675475309.gateway_id:                "" => ""
    route.2675475309.instance_id:               "" => ""
    route.2675475309.nat_gateway_id:            "" => ""
    route.2675475309.network_interface_id:      "" => ""
    route.2675475309.vpc_peering_connection_id: "pcx-7f6bcf16" => ""
    route.2966001641.cidr_block:                "" => "0.0.0.0/0"
    route.2966001641.gateway_id:                "" => "nat-051241cf3fbd466e5"
    route.2966001641.instance_id:               "" => ""
    route.2966001641.nat_gateway_id:            "" => ""
    route.2966001641.network_interface_id:      "" => ""
    route.2966001641.vpc_peering_connection_id: "" => ""

[hashibot]

This comment was originally opened by @SpencerBrown as hashicorp/terraform#2879 (comment). It was migrated here as part of the provider split. The original comment is below.

---

Related issue #4097 which is the other side of the coin. If you create a route table entry with an instance id, then do a subsequent plan, Terraform wants to erase the network interface ID that AWS assigns.

When Terraform creates a route, then refreshes the state, it needs to expect that AWS is filling in some of the "missing pieces" and simply add those to the state without wanting to change the route resource back to "what it thought it should be".

[hashibot]

This comment was originally opened by @ShadoMagi as hashicorp/terraform#2879 (comment). It was migrated here as part of the provider split. The original comment is below.

---

Following up on this for @jszwedko (I work with him). We were able to get a workaround by declaring the route entries as separate resources and not inline in the route_table. The diff will show if you declare the routes inline in the route_table but not when declared as separate resources. I verified this behavior on version 0.9.2 and version 0.8.8

[koooge]

I got same issue

Terraform v0.11.7
+ provider.aws v1.25.0

[github-actions]

Marking this issue as stale due to inactivity. This helps our maintainers find and focus on the active issues. If this issue receives no comments in the next 30 days it will automatically be closed. Maintainers can also remove the stale label.

If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thank you!

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!