feat: Add a lifecycle rule to allow vpc replacement

[relsqui]

Feature Request

Describe the Feature Request

Add a lifecycle rule to aws_internet_gateway.agentless_scan_gateway such that replacing aws_vpc.agentless_scan_vpc forces it to be recreated.

Is your feature request related to a problem? Please describe

When I tried to re-apply this module after some changes, Terraform created a plan that included replacing the VPC and updating the internet gateway attached to it. The plan failed with an error from the provider (that I forgot to copy, sorry) about not being able to destroy the VPC because other resources still depended on it.

By looking in the console I found that the dependent resource was the internet gateway. Terraform knows the gateway depends on the VPC, but it thinks it can just update the ID. In practice, it can't delete the VPC at all (and get the new ID) without destroying the gateway first.

Describe Preferred Solution

Adding an explicit lifecycle rule to the gateway fixed it for us:

resource "aws_internet_gateway" "agentless_scan_gateway" {
  count  = var.regional ? 1 : 0
  vpc_id = aws_vpc.agentless_scan_vpc[0].id
  lifecycle {
    replace_triggered_by = [
      aws_vpc.agentless_scan_vpc[0]
    ]
  }

  tags = {
    Name                     = "${local.prefix}-gw"
    LWTAG_SIDEKICK           = "1"
    LWTAG_LACEWORK_AGENTLESS = "1"
  }
}

Specifically, with this change, the gateway was destroyed first, then the VPC, and then they were both recreated without problems.

It's worth noting that this will also mean that the gateway gets replaced if the VPC updates, not just when it's being destroyed or recreated. I haven't thought through the side effects or edge cases of that at all, which is why this is an issue and not a pull request.

Additional Context

This is really a bug in Terraform, which is why I called this workaround a feature request. The chain of issues marked as duplicate of an upstream bug starts here and ends at this one, which is still open.

AWS's docs for troubleshooting the initial error are here, but their suggested script didn't actually find the gateway that was causing the problem.

[theopolis]

Thanks for the thorough notes @relsqui!

The only downside to replacing the IG when VPC attributes are updated is that current scans will be interrupted if they are trying to download new configuration for the Lacework API or send metrics. In this case the configuration is cached from the last scan and the metrics are "best effort" so it's OK if these are interrupted.

That said, I am curious if this lifecycle rule would replace the IG even when innocuous changes to the VPC occur, for example adding a new tag. This may cause the IG to be replaced often, which would be annoying.

[relsqui]

Yeah, that's my concern as well -- from the documentation for the lifecycle rule I believe that's the case. It's possible there's a cleverer way to do this that expresses the nature of the dependency more precisely, but I didn't find it in the little time I was digging through docs.

[theopolis]

Ok cool, I will look (and ask a few others too)

[theopolis]

So reading the replace_triggered_by docs, and how it is expected to work for attributes: https://developer.hashicorp.com/terraform/language/meta-arguments/lifecycle#replace_triggered_by

References trigger replacement in the following conditions:

• If the reference is to a resource with multiple instances, a plan to update or replace any instance will trigger replacement.
• If the reference is to a single resource instance, a plan to update or replace that instance will trigger replacement.
• If the reference is to a single attribute of a resource instance, any change to the attribute value will trigger replacement.

Then I assume:

resource "aws_internet_gateway" "agentless_scan_gateway" {
  count  = var.regional ? 1 : 0
  vpc_id = aws_vpc.agentless_scan_vpc[0].id
  lifecycle {
    replace_triggered_by = [
      aws_vpc.agentless_scan_vpc[0].id
    ]
  }

  tags = {
    Name                     = "${local.prefix}-gw"
    LWTAG_SIDEKICK           = "1"
    LWTAG_LACEWORK_AGENTLESS = "1"
  }
}

Would accomplish what we want:

• Replacing the IG if the VPC is replaced.
• Limiting the replace trigger to only VPC replacement (and not including innocuous changes like Tags updates).