Wait for ALB Target Group health threshold

[hashibot]

This issue was originally opened by @apparentlymart as hashicorp/terraform#11451. It was migrated here as part of the provider split. The original body of the issue is below.

---

There's a common pattern with autoscaling groups in Terraform of having the aws_autoscaling_group resource wait until a certain number of instances are healthy in an associated ELB before considering the autoscaling group as created.

This pattern doesn't work for the new Application Load Balancer because there's an extra indirection in the form of a target group.

A naive attempt to use the usual pattern with ALB might look like this:

resource "aws_alb" "example" {
  # ...
}

resource "aws_alb_target_group" "example" {
  name = "example-${var.version_id}"
  # ...

  lifecycle {
    create_before_destroy = true
  }
}

resource "aws_autoscaling_group" "example" {
  name = "example-${var.version_id}"
  target_group_arns = ["${aws_alb_target_group.example.arn}"]
  min_elb_capacity = 2
  # ...

  lifecycle {
    create_before_destroy = true
  }
}

resource "aws_alb_listener" "example" {
  load_balancer_arn = "${aws_alb.example.arn}"
  # ...

  default_action {
    target_group_arn = "${aws_alb_target_group.example.arn}"
    type = "forward"
  }

  # Don't update target_group_arn to new target group until
  # the autoscaling group has healthy instances.
  depends_on = ["aws_autoscaling_group.example"]
}

The problem here is the non-obvious dependency cycle that causes a deadlock: the min_elb_capacity attribute on aws_autoscaling_group causes that resource to block completion until it sees instances in the ELB, but the instances can't get into the ELB until the aws_alb_listener resource is updated to include the new target group.

---

The DescribeTargetHealth operation allows us to ask for the healthcheck status of the members of a target group without first attaching the target group to an ALB.

Terraform could detect that target_group_arns is set on the aws_autoscaling_group resource and prefer to use the ALB DescribeTargetHealth operation instead of the ELB status operation, which would then allow the subsequent aws_alb_listener update to complete only once the target group has become healthy enough to satisfy the constraint.

It seems that we could do this without changing the configuration schema at all, because wait_for_elb_capacity can just be treated as the minimum value of (number of healthy classic ELB instances + number of distinct healthy instances across all ALB target groups), without the user needing to think about whether it's a classic ELB or an ALB that's providing the capacity.

[matt-deboer]

Slight issue...the DescribeTargetHealth operation shows a status of unused until the target group is attached to an ALB.

As a potential workaround, the new target group could be attached using a rule for a host or path that is unexpected to be triggered--then, once the minimum expected healthy count is confirmed, it could replace the default rule.

[salvianreynaldi]

does this mean that this fix was not working? hashicorp/terraform#10243

[jcomeaux]

@salvianreynaldi I can say for sure that "wait_for_elb_capacity" doesn't work for me.

[pearcedavis]

@salvianreynaldi @jcomeaux - That fix works fine for me, but only if I use the target_group_arns attribute on the autoscaling_group resource, rather than using an autoscaling_attachment resource, which results in a short period of downtime.

[minac]

@pdedmon You have no idea how long I was fighting these weird 504s between deployments! Removing the autoscaling_attachment and recreating the ASG did the trick for good. 200 OKs only from now on! 💯

[rahulwa]

Thank you very much for providing this solution.

[github-actions]

Marking this issue as stale due to inactivity. This helps our maintainers find and focus on the active issues. If this issue receives no comments in the next 30 days it will automatically be closed. Maintainers can also remove the stale label.

If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.