aws_lb_listener read taking longer than 1 minute timeout

[b-dean]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

I regularly have terraform plan fail because it can't refresh an aws_lb_listener in a minute. I don't know if this matters but we have a lot of listeners on our account, 16 ALBs, 21 NLBs, and 90 ELB classic (not sure if that affects these sort of describes or not).

When I run

time aws elbv2 describe-listeners --listener-arn arn:aws:elasticloadbalancing:us-east-1:......

it almost always completes in less than a second. And most of our terraform plan runs work too, but sometimes it doesn't and gets this error.

It'd be nice to have some way to make this longer than a minute

Terraform Version

terraform 0.11.11
aws provider 1.60.0

Affected Resource(s)

• aws_lb_listener

Terraform Configuration Files

Nothing special, this is from the example from the aws_lb_listener docs. I'm not sure that anything in the configuration is causing the problems.

resource "aws_lb" "front_end" {
  # ...
}

resource "aws_lb_target_group" "front_end" {
  # ...
}

resource "aws_lb_listener" "front_end" {
  load_balancer_arn = "${aws_lb.front_end.arn}"
  port              = "443"
  protocol          = "HTTPS"
  ssl_policy        = "ELBSecurityPolicy-2016-08"
  certificate_arn   = "arn:aws:iam::187416307283:server-certificate/test_cert_rab3wuqwgja25ct3n4jdj2tzu4"

  default_action {
    type             = "forward"
    target_group_arn = "${aws_lb_target_group.front_end.arn}"
  }
}

Expected Behavior

terraform plan or terraform refresh can run without timing out

Actual Behavior

Error: Error refreshing state: 1 error(s) occurred:

* aws_lb_listener.foo: 1 error(s) occurred:

* aws_lb_listener.foo: aws_lb_listener.foo: Error retrieving Listener: timeout while waiting for state to become 'success' (timeout: 1m0s)

Steps to Reproduce

1. have the intermittent aws weirdness that makes this slow, but only sometimes
2. terraform plan

Important Factoids

When we are running our terraform plan we are often running multiple terraform plans, for different workspaces (from different working directories), in parallel. I don't know if the combination of so many describe-listeners requests happening at once is the problem, but it's very difficult to reproduce.

References

The line of code in question in the aws_lb_listener resource:
https://github.com/terraform-providers/terraform-provider-aws/blob/v1.60.0/aws/resource_aws_lb_listener.go#L517

• AWS ALB http/https listener creation/destruction unstable and caused errors for dependencies #2456
• aws_lb_listener: wait for listener creation #5167 - PR to fix AWS ALB http/https listener creation/destruction unstable and caused errors for dependencies #2456 where @bflad said in the review that the timeout for read should be hardcoded to 1 minute.

[nywilken]

Hi @b-dean thanks for opening up this issue, and I'm sorry to hear that you are running into trouble. I appreciate you highlighting the references here. If I understand your description correctly you can only, sometimes, reproduce when trying to run plan multiple times in parallel, correct? If so, do you exhibit this issue with different Terraform configurations that you may have? Possibly indicating an external request issue.

If you enable debug logging at the time of running plan and/or apply you should be able to get some information on what error is being thrown before hitting the 1 minute timeout.

TF_LOG=debug TF_LOG_PATH=log.txt terraform plan

[bflad]

This potentially relates to #7873 if it is being caused by rate limiting/throttling. The resource.Retry() code in that resource does not perform a final isResourceTimeoutError(err) retry of the operation.

[b-dean]

@nywilken, it's the same configuration files but different terraform workspaces, in parallel. So the difference in the aws_lb_listener resources would be they are for different ALB with some "${terraform.workspace}-name}" but otherwise identical. I also can't say for sure that it only happens when we plan workspaces in parallel, I think we might have other AWS accounts with only one workspace where it might have happened there too.

I can get some debug logs although since it doesn't always happen, it might be a few days before I find anything.

[nywilken]

This potentially relates to #7873 if it is being caused by rate limiting/throttling. The resource.Retry() code in that resource does not perform a final isResourceTimeoutError(err) retry of the operation.

@bflad thanks for calling this out. This is the first thing that came to mind for me as well. But want to see if there was anything in the logs that would suggest otherwise.

[Zephirin]

This is happening to me as well, in only one alb, with multiple listeners and many listener rules.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!