AWS ALB http/https listener creation/destruction unstable and caused errors for dependencies

[dohoangkhiem]

Hi there,

Recently we've found out the creation/destruction of our ALB http/https (especially https one) listener become very unstable, it's very common (but not reproducible every time) that it failed the first time (with error described below) - the symptom is like the aws_alb_listener resource is created but ARN is not recorded in state - that caused failure for dependent resources like aws_alb_listener_rule, or it's destroyed during terraform destroy but is somehow not completely gone so aws_alb_target_group deletion failed (as target group is in-use by the listener).

We don't get these errors every time but it's increasingly happening recently and a small test with just few resources regarding ALB and running several apply and destroy continuously (like terraform destroy -force && terraform apply && terraform destroy -force && terraform apply) would occasionally produce such errors (with our real production code which is much more complex the errors happened more often):

here is the TF configuration for test

variable "domain_name" {
  default = "int.mytest.com"
}

variable "ssl_policy" {
  default = "ELBSecurityPolicy-2016-08"
}

data "aws_acm_certificate" "mgnl_certificate" {
  domain = "*.${var.domain_name}"
}

resource "aws_alb" "alb" {
  name = "khiem-test-alb"
  internal = false
  security_groups = ["sg-27cfa641"]
  subnets = ["subnet-d0aa1fb7", "subnet-c7c51e8e"]

  lifecycle {
    create_before_destroy = true
  }
}

resource "aws_alb_target_group" "author_target_group" {
  name = "khiem-author-target-group"
  port = 8080
  protocol = "HTTP"
  vpc_id   = "vpc-72d23715"

  health_check = {
    protocol = "HTTP"
    path = "/.healthcheck/"
    port = 8080
    healthy_threshold = 5
    unhealthy_threshold = 2
    timeout = 5
    interval = 30
    matcher = "200"
  }

  lifecycle {
    create_before_destroy = true
  }
}

resource "aws_alb_target_group_attachment" "author_target_group_att" {
  target_group_arn = "${aws_alb_target_group.author_target_group.arn}"
  target_id = "i-0285315cd59a13c17"
  port = 8080

  lifecycle {
    create_before_destroy = true
  }
}

resource "aws_alb_target_group" "public_target_group" {
  name = "khiem-public-target-group"
  port = 8080
  protocol = "HTTP"
  vpc_id   = "vpc-72d23715"

  health_check = {
    protocol = "HTTP"
    path = "/.healthcheck/"
    port = 8080
    healthy_threshold = 5
    unhealthy_threshold = 2
    timeout = 5
    interval = 30
    matcher = "200"
  }

  lifecycle {
    create_before_destroy = true
  }
}

resource "aws_alb_target_group_attachment" "public_target_group_att" {
  target_group_arn = "${aws_alb_target_group.public_target_group.arn}"
  target_id = "i-0285315cd59a13c17"
  port = 8080

  lifecycle {
    create_before_destroy = true
  }
}

# http listener
resource "aws_alb_listener" "alb_http_listener" {
  load_balancer_arn = "${aws_alb.alb.arn}"
  port = "80"
  protocol = "HTTP"

  default_action {
    target_group_arn = "${aws_alb_target_group.public_target_group.arn}"
    type             = "forward"
  }

  lifecycle {
    create_before_destroy = true
  }
}

# http listener rules
resource "aws_alb_listener_rule" "alb_http_public_rule" {
  listener_arn = "${aws_alb_listener.alb_http_listener.arn}"
  priority = 100

  action {
    type = "forward"
    target_group_arn = "${aws_alb_target_group.public_target_group.arn}"
  }

  condition {
    field = "host-header"
    values = ["public-khiem.${var.domain_name}"]
  }

  lifecycle {
    create_before_destroy = true
  }
}

resource "aws_alb_listener_rule" "alb_http_author_rule" {
  listener_arn = "${aws_alb_listener.alb_http_listener.arn}"
  priority = 99

  action {
    type = "forward"
    target_group_arn = "${aws_alb_target_group.author_target_group.arn}"
  }

  condition {
    field = "host-header"
    values = ["author-khiem.${var.domain_name}"]
  }

  lifecycle {
    create_before_destroy = true
  }
}

# https listener
resource "aws_alb_listener" "alb_https_listener" {
  load_balancer_arn = "${aws_alb.alb.arn}"
  port = "443"
  protocol = "HTTPS"

  ssl_policy        = "${var.ssl_policy}"
  certificate_arn   = "${data.aws_acm_certificate.mgnl_certificate.arn}"

  default_action {
    target_group_arn = "${aws_alb_target_group.public_target_group.arn}"
    type             = "forward"
  }

  lifecycle {
    create_before_destroy = true
  }
}

# https listener rules
resource "aws_alb_listener_rule" "alb_https_public_rule" {
  listener_arn = "${aws_alb_listener.alb_https_listener.arn}"
  priority = 100

  action {
    type = "forward"
    target_group_arn = "${aws_alb_target_group.public_target_group.arn}"
  }

  condition {
    field = "host-header"
    values = ["public-khiem.${var.domain_name}"]
  }

  lifecycle {
    create_before_destroy = true
  }
}

resource "aws_alb_listener_rule" "alb_https_author_rule" {
  listener_arn = "${aws_alb_listener.alb_https_listener.arn}"
  priority = 99

  action {
    type = "forward"
    target_group_arn = "${aws_alb_target_group.author_target_group.arn}"
  }

  condition {
    field = "host-header"
    values = ["author-khiem.${var.domain_name}"]
  }

  lifecycle {
    create_before_destroy = true
  }
}

Apply Error

data.aws_acm_certificate.mgnl_certificate: Refreshing state...
aws_alb_target_group.author_target_group: Creating...
  arn:                                "" => "<computed>"
  arn_suffix:                         "" => "<computed>"
  deregistration_delay:               "" => "300"
  health_check.#:                     "" => "1"
  health_check.0.healthy_threshold:   "" => "5"
  health_check.0.interval:            "" => "30"
  health_check.0.matcher:             "" => "200"
  health_check.0.path:                "" => "/.healthcheck/"
  health_check.0.port:                "" => "8080"
  health_check.0.protocol:            "" => "HTTP"
  health_check.0.timeout:             "" => "5"
  health_check.0.unhealthy_threshold: "" => "2"
  name:                               "" => "khiem-author-target-group"
  port:                               "" => "8080"
  protocol:                           "" => "HTTP"
  stickiness.#:                       "" => "<computed>"
  target_type:                        "" => "instance"
  vpc_id:                             "" => "vpc-72d23715"
aws_alb_target_group.public_target_group: Creating...
  arn:                                "" => "<computed>"
  arn_suffix:                         "" => "<computed>"
  deregistration_delay:               "" => "300"
  health_check.#:                     "" => "1"
  health_check.0.healthy_threshold:   "" => "5"
  health_check.0.interval:            "" => "30"
  health_check.0.matcher:             "" => "200"
  health_check.0.path:                "" => "/.healthcheck/"
  health_check.0.port:                "" => "8080"
  health_check.0.protocol:            "" => "HTTP"
  health_check.0.timeout:             "" => "5"
  health_check.0.unhealthy_threshold: "" => "2"
  name:                               "" => "khiem-public-target-group"
  port:                               "" => "8080"
  protocol:                           "" => "HTTP"
  stickiness.#:                       "" => "<computed>"
  target_type:                        "" => "instance"
  vpc_id:                             "" => "vpc-72d23715"
aws_alb.alb: Creating...
  access_logs.#:              "" => "<computed>"
  arn:                        "" => "<computed>"
  arn_suffix:                 "" => "<computed>"
  dns_name:                   "" => "<computed>"
  enable_deletion_protection: "" => "false"
  idle_timeout:               "" => "60"
  internal:                   "" => "false"
  ip_address_type:            "" => "<computed>"
  load_balancer_type:         "" => "application"
  name:                       "" => "khiem-test-alb"
  security_groups.#:          "" => "1"
  security_groups.930362799:  "" => "sg-27cfa641"
  subnets.#:                  "" => "2"
  subnets.1419775440:         "" => "subnet-c7c51e8e"
  subnets.3706636568:         "" => "subnet-d0aa1fb7"
  vpc_id:                     "" => "<computed>"
  zone_id:                    "" => "<computed>"
aws_alb_target_group.public_target_group: Creation complete after 1s (ID: arn:aws:elasticloadbalancing:ap-southea...m-public-target-group/8c83c5482782160c)
aws_alb_target_group_attachment.public_target_group_att: Creating...
  port:             "" => "8080"
  target_group_arn: "" => "arn:aws:elasticloadbalancing:ap-southeast-1:218832052474:targetgroup/khiem-public-target-group/8c83c5482782160c"
  target_id:        "" => "i-0285315cd59a13c17"
aws_alb_target_group.author_target_group: Creation complete after 1s (ID: arn:aws:elasticloadbalancing:ap-southea...m-author-target-group/e600a57f2882299b)
aws_alb_target_group_attachment.author_target_group_att: Creating...
  port:             "" => "8080"
  target_group_arn: "" => "arn:aws:elasticloadbalancing:ap-southeast-1:218832052474:targetgroup/khiem-author-target-group/e600a57f2882299b"
  target_id:        "" => "i-0285315cd59a13c17"
aws_alb_target_group_attachment.public_target_group_att: Creation complete after 0s (ID: arn:aws:elasticloadbalancing:ap-southea...5482782160c-20171128161046303200000001)
aws_alb_target_group_attachment.author_target_group_att: Creation complete after 0s (ID: arn:aws:elasticloadbalancing:ap-southea...57f2882299b-20171128161046332800000002)
aws_alb.alb: Still creating... (10s elapsed)
aws_alb.alb: Still creating... (20s elapsed)
aws_alb.alb: Still creating... (30s elapsed)
aws_alb.alb: Still creating... (40s elapsed)
aws_alb.alb: Still creating... (50s elapsed)
aws_alb.alb: Still creating... (1m0s elapsed)
aws_alb.alb: Still creating... (1m10s elapsed)
aws_alb.alb: Still creating... (1m20s elapsed)
aws_alb.alb: Still creating... (1m30s elapsed)
aws_alb.alb: Still creating... (1m40s elapsed)
aws_alb.alb: Still creating... (1m50s elapsed)
aws_alb.alb: Still creating... (2m0s elapsed)
teraws_alb.alb: Still creating... (2m10s elapsed)
aws_alb.alb: Still creating... (2m20s elapsed)
aws_alb.alb: Creation complete after 2m22s (ID: arn:aws:elasticloadbalancing:ap-southea...er/app/khiem-test-alb/3e1f7eaee507ffea)
aws_alb_listener.alb_http_listener: Creating...
  arn:                               "" => "<computed>"
  default_action.#:                  "" => "1"
  default_action.0.target_group_arn: "" => "arn:aws:elasticloadbalancing:ap-southeast-1:218832052474:targetgroup/khiem-public-target-group/8c83c5482782160c"
  default_action.0.type:             "" => "forward"
  load_balancer_arn:                 "" => "arn:aws:elasticloadbalancing:ap-southeast-1:218832052474:loadbalancer/app/khiem-test-alb/3e1f7eaee507ffea"
  port:                              "" => "80"
  protocol:                          "" => "HTTP"
  ssl_policy:                        "" => "<computed>"
aws_alb_listener.alb_https_listener: Creating...
  arn:                               "" => "<computed>"
  certificate_arn:                   "" => "arn:aws:acm:ap-southeast-1:218832052474:certificate/2819229d-6c29-4849-a476-b123f5b51f56"
  default_action.#:                  "" => "1"
  default_action.0.target_group_arn: "" => "arn:aws:elasticloadbalancing:ap-southeast-1:218832052474:targetgroup/khiem-public-target-group/8c83c5482782160c"
  default_action.0.type:             "" => "forward"
  load_balancer_arn:                 "" => "arn:aws:elasticloadbalancing:ap-southeast-1:218832052474:loadbalancer/app/khiem-test-alb/3e1f7eaee507ffea"
  port:                              "" => "443"
  protocol:                          "" => "HTTPS"
  ssl_policy:                        "" => "ELBSecurityPolicy-2016-08"
aws_alb_listener.alb_http_listener: Creation complete after 0s
aws_alb_listener.alb_https_listener: Creation complete after 0s (ID: arn:aws:elasticloadbalancing:ap-southea...-alb/3e1f7eaee507ffea/0d5df35cf6425343)
aws_alb_listener_rule.alb_https_author_rule: Creating...
  action.#:                      "" => "1"
  action.0.target_group_arn:     "" => "arn:aws:elasticloadbalancing:ap-southeast-1:218832052474:targetgroup/khiem-author-target-group/e600a57f2882299b"
  action.0.type:                 "" => "forward"
  arn:                           "" => "<computed>"
  condition.#:                   "" => "1"
  condition.3686469405.field:    "" => "host-header"
  condition.3686469405.values.#: "" => "1"
  condition.3686469405.values.0: "" => "author-khiem.int.magnolia-now.com"
  listener_arn:                  "" => "arn:aws:elasticloadbalancing:ap-southeast-1:218832052474:listener/app/khiem-test-alb/3e1f7eaee507ffea/0d5df35cf6425343"
  priority:                      "" => "99"
aws_alb_listener_rule.alb_https_public_rule: Creating...
  action.#:                     "" => "1"
  action.0.target_group_arn:    "" => "arn:aws:elasticloadbalancing:ap-southeast-1:218832052474:targetgroup/khiem-public-target-group/8c83c5482782160c"
  action.0.type:                "" => "forward"
  arn:                          "" => "<computed>"
  condition.#:                  "" => "1"
  condition.590182385.field:    "" => "host-header"
  condition.590182385.values.#: "" => "1"
  condition.590182385.values.0: "" => "public-khiem.int.magnolia-now.com"
  listener_arn:                 "" => "arn:aws:elasticloadbalancing:ap-southeast-1:218832052474:listener/app/khiem-test-alb/3e1f7eaee507ffea/0d5df35cf6425343"
  priority:                     "" => "100"
aws_alb_listener_rule.alb_https_author_rule: Creation complete after 0s (ID: arn:aws:elasticloadbalancing:ap-southea...ffea/0d5df35cf6425343/17350e80003a00f9)
aws_alb_listener_rule.alb_https_public_rule: Creation complete after 0s (ID: arn:aws:elasticloadbalancing:ap-southea...ffea/0d5df35cf6425343/0a204f8fe7701e0c)
Error applying plan:

2 error(s) occurred:

* aws_alb_listener_rule.alb_http_public_rule: Resource 'aws_alb_listener.alb_http_listener' does not have attribute 'arn' for variable 'aws_alb_listener.alb_http_listener.arn'
* aws_alb_listener_rule.alb_http_author_rule: Resource 'aws_alb_listener.alb_http_listener' does not have attribute 'arn' for variable 'aws_alb_listener.alb_http_listener.arn'

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

In this case actually the Listener is already created in AWS.

Destroy Error (after a successful apply)

ubuntu@ip-172-31-29-175:/vagrant/provision/terraform-test/alb_listener$ terraform destroy -force && terraform apply && terraform destroy -force && terraform apply
aws_alb_target_group.public_target_group: Refreshing state... (ID: arn:aws:elasticloadbalancing:ap-southea...m-public-target-group/f03543805196b4ee)
aws_alb_target_group.author_target_group: Refreshing state... (ID: arn:aws:elasticloadbalancing:ap-southea...m-author-target-group/ecd5669c14c43e5d)
aws_alb.alb: Refreshing state... (ID: arn:aws:elasticloadbalancing:ap-southea...er/app/khiem-test-alb/4a210b69d6ae0f76)
data.aws_acm_certificate.mgnl_certificate: Refreshing state...
aws_alb_target_group_attachment.author_target_group_att: Refreshing state... (ID: arn:aws:elasticloadbalancing:ap-southea...69c14c43e5d-20171128153710934400000002)
aws_alb_target_group_attachment.public_target_group_att: Refreshing state... (ID: arn:aws:elasticloadbalancing:ap-southea...3805196b4ee-20171128153710873100000001)
aws_alb_listener.alb_https_listener: Refreshing state... (ID: arn:aws:elasticloadbalancing:ap-southea...-alb/4a210b69d6ae0f76/f668a85fcab2d5b7)
aws_alb_listener.alb_http_listener: Refreshing state... (ID: arn:aws:elasticloadbalancing:ap-southea...-alb/4a210b69d6ae0f76/12f64559728eb05f)
aws_alb_listener_rule.alb_https_public_rule: Refreshing state... (ID: arn:aws:elasticloadbalancing:ap-southea...0f76/f668a85fcab2d5b7/ec3e976e8691387b)
aws_alb_listener_rule.alb_https_author_rule: Refreshing state... (ID: arn:aws:elasticloadbalancing:ap-southea...0f76/f668a85fcab2d5b7/a8eb533e8fb47bfb)
aws_alb_listener_rule.alb_http_public_rule: Refreshing state... (ID: arn:aws:elasticloadbalancing:ap-southea...0f76/12f64559728eb05f/14f5d14b2ded5572)
aws_alb_listener_rule.alb_https_author_rule: Destroying... (ID: arn:aws:elasticloadbalancing:ap-southea...0f76/f668a85fcab2d5b7/a8eb533e8fb47bfb)
aws_alb_target_group_attachment.author_target_group_att: Destroying... (ID: arn:aws:elasticloadbalancing:ap-southea...69c14c43e5d-20171128153710934400000002)
aws_alb_listener_rule.alb_http_public_rule: Destroying... (ID: arn:aws:elasticloadbalancing:ap-southea...0f76/12f64559728eb05f/14f5d14b2ded5572)
aws_alb_listener_rule.alb_https_public_rule: Destroying... (ID: arn:aws:elasticloadbalancing:ap-southea...0f76/f668a85fcab2d5b7/ec3e976e8691387b)
aws_alb_target_group_attachment.public_target_group_att: Destroying... (ID: arn:aws:elasticloadbalancing:ap-southea...3805196b4ee-20171128153710873100000001)
aws_alb_listener_rule.alb_https_author_rule: Destruction complete after 0s
aws_alb_listener_rule.alb_https_public_rule: Destruction complete after 0s
aws_alb_listener.alb_https_listener: Destroying... (ID: arn:aws:elasticloadbalancing:ap-southea...-alb/4a210b69d6ae0f76/f668a85fcab2d5b7)
aws_alb_target_group_attachment.author_target_group_att: Destruction complete after 0s
aws_alb_target_group.author_target_group: Destroying... (ID: arn:aws:elasticloadbalancing:ap-southea...m-author-target-group/ecd5669c14c43e5d)
aws_alb_target_group_attachment.public_target_group_att: Destruction complete after 0s
aws_alb_listener_rule.alb_http_public_rule: Destruction complete after 0s
aws_alb_listener.alb_http_listener: Destroying... (ID: arn:aws:elasticloadbalancing:ap-southea...-alb/4a210b69d6ae0f76/12f64559728eb05f)
aws_alb_listener.alb_http_listener: Destruction complete after 0s
aws_alb_listener.alb_https_listener: Destruction complete after 0s
aws_alb_target_group.public_target_group: Destroying... (ID: arn:aws:elasticloadbalancing:ap-southea...m-public-target-group/f03543805196b4ee)
aws_alb.alb: Destroying... (ID: arn:aws:elasticloadbalancing:ap-southea...er/app/khiem-test-alb/4a210b69d6ae0f76)
aws_alb_target_group.public_target_group: Destruction complete after 0s
aws_alb.alb: Still destroying... (ID: arn:aws:elasticloadbalancing:ap-southea...er/app/khiem-test-alb/4a210b69d6ae0f76, 10s elapsed)
aws_alb.alb: Still destroying... (ID: arn:aws:elasticloadbalancing:ap-southea...er/app/khiem-test-alb/4a210b69d6ae0f76, 20s elapsed)
aws_alb.alb: Still destroying... (ID: arn:aws:elasticloadbalancing:ap-southea...er/app/khiem-test-alb/4a210b69d6ae0f76, 30s elapsed)
aws_alb.alb: Destruction complete after 35s
Error applying plan:

1 error(s) occurred:

* aws_alb_target_group.author_target_group (destroy): 1 error(s) occurred:

* aws_alb_target_group.author_target_group: Error deleting Target Group: ResourceInUse: Target group 'arn:aws:elasticloadbalancing:ap-southeast-1:218832052474:targetgroup/khiem-author-target-group/ecd5669c14c43e5d' is currently in use by a listener or a rule
	status code: 400, request id: 022c5dfb-d455-11e7-b38c-557a182c4eef

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

And in this case when I try delete the target group from EC2 Console there's no issue (as the listener is actually deleted).

Terraform version is 0.10.7

[hashibot]

This issue has been automatically migrated to hashicorp/terraform-provider-aws#2456 because it looks like an issue with that provider. If you believe this is not an issue with the provider, please reply to this issue and let us know.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.