provider/google incorrect update order between instance template and group manager

[lobeck]

Terraform Version

Terraform v0.8.6

Affected Resource(s)

Please list the resources as a list, for example:

• google_compute_instance_template
• google_compute_instance_group_manager

Terraform Configuration Files

resource "google_compute_instance_template" "consul" {
  count = "${length(var.regions)}" # one per region
  name_prefix = "${var.name}-${element(var.regions, count.index)}-"
  tags = "${concat(var.tags, list(join("-", list(var.nat_tag, element(var.regions, count.index)))), list("consul"))}"

  region = "${element(var.regions, count.index)}"

  description = "${var.description}"
  machine_type = "${var.machine_type}"

  # Create a new boot disk from an image
  disk {
    source_image = "$foo-ubuntu-1204-precise" # it's a family name
    disk_type = "${var.disk_type}"
  }

  network_interface {
    network = "${var.network}"
  }

  service_account {
    scopes = ["storage-ro", "compute-ro"]
  }

  metadata {
    startup-script-url = "${var.default_startup_script}"
  }

  lifecycle {
    create_before_destroy = true
  }

resource "google_compute_instance_group_manager" "consul" {
  count = "${length(var.regions)}" # one per region
  zone = "${element(var.regions, count.index)}-${element(var.zones[element(var.regions, count.index)], (count.index) / length(var.regions))}"
  name = "${var.name}-${element(var.regions, count.index)}"
  update_strategy    = "NONE"

  target_size  = "${var.count}"

  base_instance_name = "${var.name}-${element(var.regions, count.index)}"

  instance_template  = "${element(google_compute_instance_template.consul.*.self_link, count.index)}"

}

Terraform plan

~ module.infrastructure.consul.google_compute_instance_group_manager.consul.0
    instance_template: "https://www.googleapis.com/compute/v1/projects/$project/global/instanceTemplates/consul-europe-west1" => "https://www.googleapis.com/compute/v1/projects/$project/global/instanceTemplates/consul-europe-west1-001e39d3e89692fb74c75a96e8"

~ module.infrastructure.consul.google_compute_instance_group_manager.consul.1
    instance_template: "https://www.googleapis.com/compute/v1/projects/$project/global/instanceTemplates/consul-us-east1" => "https://www.googleapis.com/compute/v1/projects/$project/global/instanceTemplates/consul-us-east1-001e39d3e89692fb74c75a96e7"

- module.infrastructure.consul.google_compute_instance_template.consul.0 (deposed)

- module.infrastructure.consul.google_compute_instance_template.consul.1 (deposed)

Expected Behavior

The group_manager gets updated to use the new template, then the old template gets deleted

Actual Behavior

It tries to delete the template first, which subsequently fails with google_compute_instance_template.consul.0 (deposed #0): Error deleting instance template: googleapi: Error 400: The instance_template resource 'consul-europe-west1-001e39d3e89692fb74c75a96e8' is already being used by 'consul-europe-west1', resourceInUseByAnotherResource

Steps to Reproduce

• Setup everything
• Update the image/add a new image (family lookup has no effect, so i'm specifying the exact versions)
• plan works but apply fails due to the incorrect order.

[lobeck]

It seems as this is an issue with instance_template = "${element(google_compute_instance_template.consul.*.self_link, count.index)}", as it works when self_link is directly used (without the element() call)

Any ideas how to fix this?

[andyshinn]

I think it is still an issue even without element(). I am getting the error with a regular self_link attribute on Terraform v0.11.2.

[OmarDarwish]

I am also still running into this issue with v0.11.10

[OmarDarwish]

Resolved by hashicorp/terraform-provider-google#1601:

• Add to google_compute_instance_template

lifecycle {
  create_before_destroy = true
}

• Change name attribute to something other than what the existing template is named
• terraform apply
• (Optional) Change name back to what it was originally
• terraform apply

[universalvishwa]

The required debug outputs for hashicorp/terraform-provider-google#1448

This issue is still there in Terraform v0.11.10. I tried all the possible options stated above. Additionally What I want to add is, After the failure, when I re-run the 'terraform apply' for the second time it successfully passed. My guess is that, Terarform is trying to destroy the old resource a little too quicker, before the Google's API is gets updated with then new instance template. If the destruction of the 'create_before_destroy' can be delayed by few seconds, we might be able to avoid this.

Error output:

...
google_compute_instance_template.default: Still creating... (10s elapsed)
google_compute_instance_template.default: Creation complete after 12s (ID: example-dev-tcore-test20181211001403447100000001)
google_compute_instance_template.default.deposed: Destroying... (ID: example-dev-core-test20181210235311591900000001)
module.us-east1.google_compute_region_instance_group_manager.default: Modifying... (ID: example-dev-us-east1)
  instance_template: "https://www.googleapis.com/compute/beta/projects/example-dev-udje/global/instanceTemplates/example-dev-core-test201812101" => "https://www.googleapis.com/compute/beta/projects/example-dev-udje/global/instanceTemplates/example-dev-tcore-test201812111"
module.us-east1.google_compute_region_instance_group_manager.default: Still modifying... (ID: example-dev-us-east1, 10s elapsed)
module.us-east1.google_compute_region_instance_group_manager.default: Modifications complete after 12s (ID: example-dev-us-east1)

Error: Error applying plan:

1 error(s) occurred:

* google_compute_instance_template.default (destroy): 1 error(s) occurred:

* google_compute_instance_template.default (deposed #0): 1 error(s) occurred:

* google_compute_instance_template.default (deposed #0): Error deleting instance template: googleapi: Error 400: The instance_template resource 'projects/example-dev-udje/global/instanceTemplates/example-dev-core-test20181210235311591900000001' is already being used by 'projects/example-dev-udje/regions/us-east1/instanceGroupManagers/example-dev-us-east1', resourceInUseByAnotherResource

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

Here is the output when I re-run it. Can clearly see that the new template is refreshed. At the end it the old one gets destroyed.

$ terraform apply
random_id.instance_template: Refreshing state... (ID: ARW97A)
google_compute_instance_template.default: Refreshing state... (ID: example-dev-tcore-test201812111)
google_compute_http_health_check.default: Refreshing state... (ID: example-dev-http-ping)
google_compute_global_address.default: Refreshing state... (ID: example-dev-tcplb-globalip)
google_compute_health_check.default: Refreshing state... (ID: example-dev-dns)
data.google_compute_regions.available: Refreshing state...
data.google_project.project: Refreshing state...
google_compute_backend_service.default: Refreshing state... (ID: example-dev-tcplb-backend)
google_compute_target_pool.default: Refreshing state... (ID: example-dev-us-east1)
google_compute_target_tcp_proxy.default: Refreshing state... (ID: example-dev-tcplb)
google_compute_region_instance_group_manager.default: Refreshing state... (ID: example-dev-us-east1)
google_compute_instance_template.default.deposed: Destroying... (ID: example-dev-core-test201812101)
google_compute_instance_template.default (deposed #0): Still destroying... (ID: example-dev-core-test201812101, 10s elapsed)
google_compute_instance_template.default.deposed: Destruction complete after 11s

Apply complete! Resources: 0 added, 0 changed, 1 destroyed.

[keedon]

I'm also seeing this problem with terraform 0.11.13 and provider.google v2.11.0

[hashibot]

Hello! 🤖

This issue relates to an older version of Terraform that is no longer in active development, and because the area of Terraform it relates to has changed significantly since the issue was opened we suspect that the issue is either fixed or that the circumstances around it have changed enough that we'd need an updated issue report in order to reproduce and address it.

If you're still seeing this or a similar issue in the latest version of Terraform, please do feel free to open a new bug report! Please be sure to include all of the information requested in the template, even if it might seem redundant with the information already shared in this issue, because the internal details relating to this problem are likely to be different in the current version of Terraform.

Thanks!

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.