plan is in a loop changing my "main" route table

[egarbi]

$ terraform --version
 Terraform v0.6.1

# Main routing table private to nat
resource "aws_route_table" "main" {
   route {
       cidr_block = "0.0.0.0/0"
       network_interface_id = "eni-7aXXXX"
   }
   route {
       cidr_block = "${var.vpn_cidr}"
       instance_id = "${module.vpn.minion_id}"
   }
   route {
       cidr_block = "${lookup(var.tokio_vpc_cidr,var.env)}"
       instance_id = "${module.vpn-gw.id}"
   }
   route {
       cidr_block = "${lookup(var.usa_vpc_cidr,var.env)}"
       instance_id = "${module.gw-free.id}"
   }
   tags {
       Name = "${lookup(var.domain,var.env)}-vpc-to-nat"
   }
   vpc_id = "${aws_vpc.testing.id}"
}

terraform plan...

~ aws_route_table.main
    route.1714403534.cidr_block:                "10.32.0.0/16" => "10.32.0.0/16"
    route.1714403534.gateway_id:                "" => ""
    route.1714403534.instance_id:               "i-64XXXX" => "i-64XXXX"
    route.1714403534.network_interface_id:      "eni-2aXXXX" => ""
    route.1714403534.vpc_peering_connection_id: "" => ""
    route.2010688411.cidr_block:                "10.52.0.0/16" => "10.52.0.0/16"
    route.2010688411.gateway_id:                "" => ""
    route.2010688411.instance_id:               "i-36XXXX" => "i-36XXXX"
    route.2010688411.network_interface_id:      "eni-a7XXXX" => ""
    route.2010688411.vpc_peering_connection_id: "" => ""
    route.3093398733.cidr_block:                "192.168.240.0/24" => "192.168.240.0/24"
    route.3093398733.gateway_id:                "" => ""
    route.3093398733.instance_id:               "i-d5XXXX" => "i-d52XXXX"
    route.3093398733.network_interface_id:      "eni-69XXXX" => ""
    route.3093398733.vpc_peering_connection_id: "" => ""
    route.3891333365.cidr_block:                "0.0.0.0/0" => ""
    route.3891333365.gateway_id:                "" => ""
    route.3891333365.instance_id:               "i-0fXXXX" => ""
    route.3891333365.network_interface_id:      "eni-7aXXXX" => ""
    route.3891333365.vpc_peering_connection_id: "" => ""
    route.3991494531.cidr_block:                "" => "0.0.0.0/0"
    route.3991494531.gateway_id:                "" => ""
    route.3991494531.instance_id:               "" => ""
    route.3991494531.network_interface_id:      "" => "eni-7aXXXX"
    route.3991494531.vpc_peering_connection_id: "" => ""

[jszwedko]

@catsby we are seeing this as well -- basically if you associate a route table with an ENI that is associated with an instance, AWS also returns the instance ID on subsequently queries for the route table which causes terraform to think that there is a diff. We took a crack at fixing this today, but the nuances around schema.TypeSet and how the diffs are determined prevented us from fixing it.

I think the correct behavior is to have the instance id be persisted to the state file, but not be considered a diff.

Here is a diff with a failing test case:

diff --git a/builtin/providers/aws/resource_aws_route_table_test.go b/builtin/providers/aws/resource_aws_route_table_test.go
index 6eb8951..060ea23 100644
--- a/builtin/providers/aws/resource_aws_route_table_test.go
+++ b/builtin/providers/aws/resource_aws_route_table_test.go
@@ -123,6 +123,46 @@ func TestAccAWSRouteTable_instance(t *testing.T) {
        })
 }

+func TestAccAWSRouteTable_networkInterface(t *testing.T) {
+       var v ec2.RouteTable
+
+       testCheck := func(*terraform.State) error {
+               if len(v.Routes) != 2 {
+                       return fmt.Errorf("bad routes: %#v", v.Routes)
+               }
+
+               routes := make(map[string]*ec2.Route)
+               for _, r := range v.Routes {
+                       routes[*r.DestinationCidrBlock] = r
+               }
+
+               if _, ok := routes["10.1.0.0/16"]; !ok {
+                       return fmt.Errorf("bad routes: %#v", v.Routes)
+               }
+               if _, ok := routes["10.2.0.0/16"]; !ok {
+                       return fmt.Errorf("bad routes: %#v", v.Routes)
+               }
+
+               return nil
+       }
+
+       resource.Test(t, resource.TestCase{
+               PreCheck:     func() { testAccPreCheck(t) },
+               Providers:    testAccProviders,
+               CheckDestroy: testAccCheckRouteTableDestroy,
+               Steps: []resource.TestStep{
+                       resource.TestStep{
+                               Config: testAccRouteTableConfigNetworkInterface,
+                               Check: resource.ComposeTestCheckFunc(
+                                       testAccCheckRouteTableExists(
+                                               "aws_route_table.foo", &v),
+                                       testCheck,
+                               ),
+                       },
+               },
+       })
+}
+
 func TestAccAWSRouteTable_tags(t *testing.T) {
        var route_table ec2.RouteTable

@@ -316,6 +356,41 @@ resource "aws_route_table" "foo" {
 }
 `

+const testAccRouteTableConfigNetworkInterface = `
+resource "aws_vpc" "foo" {
+       cidr_block = "10.1.0.0/16"
+}
+
+resource "aws_route_table" "foo" {
+       vpc_id = "${aws_vpc.foo.id}"
+
+       route {
+               cidr_block = "10.2.0.0/16"
+               network_interface_id = "${aws_network_interface.foo.id}"
+       }
+}
+
+resource "aws_network_interface" "foo" {
+  subnet_id = "${aws_subnet.foo.id}"
+  attachment {
+    instance = "${aws_instance.foo.id}"
+    device_index = 1
+  }
+}
+
+resource "aws_subnet" "foo" {
+       cidr_block = "10.1.1.0/24"
+       vpc_id = "${aws_vpc.foo.id}"
+}
+
+resource "aws_instance" "foo" {
+       # us-west-2
+       ami = "ami-4fccb37f"
+       instance_type = "m1.small"
+       subnet_id = "${aws_subnet.foo.id}"
+}
+`
+

[pidah]

I am running into this issue with eni's ->

~ aws_eip.nat_elastic_ip
instance: "i-358be594" => ""

~ aws_route_table.app-subnet
route.1972381572.cidr_block: "" => "0.0.0.0/0"
route.1972381572.gateway_id: "" => ""
route.1972381572.instance_id: "" => ""
route.1972381572.network_interface_id: "" => "eni-340f9c7d"
route.1972381572.vpc_peering_connection_id: "" => ""
route.3507476028.cidr_block: "0.0.0.0/0" => ""
route.3507476028.gateway_id: "" => ""
route.3507476028.instance_id: "i-358be594" => ""
route.3507476028.network_interface_id: "eni-340f9c7d" => ""
route.3507476028.vpc_peering_connection_id: "" => ""

~ aws_route_table.databases-subnet
route.1972381572.cidr_block: "" => "0.0.0.0/0"
route.1972381572.gateway_id: "" => ""
route.1972381572.instance_id: "" => ""
route.1972381572.network_interface_id: "" => "eni-340f9c7d"
route.1972381572.vpc_peering_connection_id: "" => ""
route.3507476028.cidr_block: "0.0.0.0/0" => ""
route.3507476028.gateway_id: "" => ""
route.3507476028.instance_id: "i-358be594" => ""
route.3507476028.network_interface_id: "eni-340f9c7d" => ""
route.3507476028.vpc_peering_connection_id: "" => ""

[gtmtech]

+1 Did you find a workaround @jszwedko @catsby ? We just ran into this too and we cant roll out an ENI now unless there is a workaround or this is solved.

[jszwedko]

@gtmtech unfortunately, no, in our case we just permit the route table update to go through.

[dsalazar-carecloud]

Just ran into this issue myself. Same scenario with using the eni parameter.

$ terraform --version
Terraform v0.6.3

[TimAtLassian]

Still getting this behaviour in v0.6.6

[alexsapran]

+1 to find a solution. We are facing the same issue.

[icflournoy]

+1 I believe I'm facing the same issue.

$ terraform version
Terraform v0.6.9

Part of the output I get every time I run plan or apply. This is for the default route table as well as two secondary route tables.

~ aws_route_table.private-b
    route.1851933015.cidr_block:                "" => "10.10.0.0/16"
    route.1851933015.gateway_id:                "" => "pcx-7f6bcf16"
    route.1851933015.instance_id:               "" => ""
    route.1851933015.nat_gateway_id:            "" => ""
    route.1851933015.network_interface_id:      "" => ""
    route.1851933015.vpc_peering_connection_id: "" => ""
    route.2195099901.cidr_block:                "0.0.0.0/0" => ""
    route.2195099901.gateway_id:                "" => ""
    route.2195099901.instance_id:               "" => ""
    route.2195099901.nat_gateway_id:            "nat-051241cf3fbd466e5" => ""
    route.2195099901.network_interface_id:      "" => ""
    route.2195099901.vpc_peering_connection_id: "" => ""
    route.2675475309.cidr_block:                "10.10.0.0/16" => ""
    route.2675475309.gateway_id:                "" => ""
    route.2675475309.instance_id:               "" => ""
    route.2675475309.nat_gateway_id:            "" => ""
    route.2675475309.network_interface_id:      "" => ""
    route.2675475309.vpc_peering_connection_id: "pcx-7f6bcf16" => ""
    route.2966001641.cidr_block:                "" => "0.0.0.0/0"
    route.2966001641.gateway_id:                "" => "nat-051241cf3fbd466e5"
    route.2966001641.instance_id:               "" => ""
    route.2966001641.nat_gateway_id:            "" => ""
    route.2966001641.network_interface_id:      "" => ""
    route.2966001641.vpc_peering_connection_id: "" => ""

[SpencerBrown]

Related issue #4097 which is the other side of the coin. If you create a route table entry with an instance id, then do a subsequent plan, Terraform wants to erase the network interface ID that AWS assigns.

When Terraform creates a route, then refreshes the state, it needs to expect that AWS is filling in some of the "missing pieces" and simply add those to the state without wanting to change the route resource back to "what it thought it should be".

[ShadoMagi]

Following up on this for @jszwedko (I work with him). We were able to get a workaround by declaring the route entries as separate resources and not inline in the route_table. The diff will show if you declare the routes inline in the route_table but not when declared as separate resources. I verified this behavior on version 0.9.2 and version 0.8.8

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.