Not possible to upgrade to terraform 0.12 if using resource aws_lb_listener_certificate

[Frogvall]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

0.12.0

Affected Resource(s)

• aws_lb_listener_certificate

Terraform Configuration Files

resource "aws_acm_certificate" "example" {
  domain_name       = "${var.uri_prefix}.${var.domain}"
  validation_method = "DNS"

  lifecycle {
    create_before_destroy = true
  }
}

resource "aws_route53_record" "example_dns_validation" {
  name    = aws_acm_certificate.example.domain_validation_options.0.resource_record_name
  type    = aws_acm_certificate.example.domain_validation_options.0.resource_record_type
  zone_id = data.aws_route53_zone.example
  records = [aws_acm_certificate.example.domain_validation_options.0.resource_record_value]
  ttl     = 300
}

resource "aws_acm_certificate_validation" "example" {
  certificate_arn         = aws_acm_certificate.example.arn
  validation_record_fqdns = [aws_route53_record.example_dns_validation.fqdn]
}

resource "aws_lb_listener_certificate" "example" {
  listener_arn    = data.aws_lb_listener.example.arn
  certificate_arn = aws_acm_certificate.example.arn
  depends_on      = aws_acm_certificate_validation.example
}

Debug Output

Panic Output

Expected Behavior

aws_lb_listener_certificate should have been included in the state file.

Actual Behavior

aws_lb_listener_certificate is not included in the state file, and terraform apply fails with the current error message:

Error: Provider produced inconsistent result after apply

When applying changes to aws_lb_listener_certificate.example, provider
"aws" produced an unexpected new value for was present, but now absent.

This is a bug in the provider, which should be reported in the provider's own
issue tracker.

Steps to Reproduce

1. Set up the terraform script listed above together with an alb (using terraform >0.12.0)
2. run terraform apply

Important Factoids

References

• aws_lb_listener_certificate not destroyed upon "force new resource" #7761

[pieterjanpintens]

We have the same problem with aws_iam_role_policy_attachment.
It does attach the policy to the role (verified in aws console) but tf fails.

When applying changes to
module.ssm_patch_management.aws_iam_role_policy_attachment.attach_ssm_policy_to_role3,
provider "aws" produced an unexpected new value for was present, but now
absent.

Note that we actually attach 5 policies. It only complains on one.

[neelam-007]

We have the same problem with aws_iam_role_policy_attachment.
It does attach the policy to the role (verified in aws console) but tf fails.

When applying changes to
module.ssm_patch_management.aws_iam_role_policy_attachment.attach_ssm_policy_to_role3,
provider "aws" produced an unexpected new value for was present, but now
absent.

Note that we actually attach 5 policies. It only complains on one.

I am currently having the same problem - is there any fix or workaround for this?

[pieterjanpintens]

@neelam-007 In our case it was because we were referencing a deprecated AWS provided policy. Check if the policyies your are attaching are AWS provided and deprecated. If so attach policy is doing very weird.

[OscarBarrett]

A temporary workaround is to comment out the aws_lb_listener_certificate resource. As it's not in the state file, terraform wont destroy it.

[Frogvall]

That would work only if the resource has already been created, and adds manual labour to your deploy. As for creating a new resource, after converting to 0.12, you would get this exception from the start (at which point, you could comment it out and commit again).
Also, in our case we deploy the infrastructure to our dev environment and when our automatic tests pass we go on and deploy it to the next environment, and so on.
Since all deploys use the same terraform specification (switching between terraform workspaces), commenting it out would make it not being created at all in the second environment.
To counter this I would have to use a count 1:0 trick, adding one more environment to the 0 side for each environment it creates, making it very tedious to deploy a new service.

At this point it would make more sense to stop using the resource and make a cli call to create it instead.

[vivanov-dp]

I had the same error with aws_iam_role_policy_attachment when attaching an Amazon managed policy to a role. The problem was that the ARN was wrong (developed in Ireland, applying in a China region - aws_partition there is aws-cn and the ARN becomes invalid)

I believe this problem is mostly that the error message is obscure and unrelated. I tried applying the erroneous policy via the aws-cli to see the error it would return and apparently it returns nothing:

velko.ivanov@SOFMBL001 MINGW64 /d/prg/terraform-global/env_alpha_cn/access_points (env_alpha_cn|MERGING)
$ aws iam attach-role-policy --role-name b12tf-alpha-cn-ssh-gate-role --policy-arn arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore --region cn-northwest-1 --profile china

velko.ivanov@SOFMBL001 MINGW64 /d/prg/terraform-global/env_alpha_cn/access_points (env_alpha_cn|MERGING)
$ aws iam attach-role-policy --role-name b12tf-alpha-cn-ssh-gate-role --policy-arn arn:aws-cn:iam::aws:policy/AmazonSSMManagedInstanceCore --region cn-northwest-1 --profile china

velko.ivanov@SOFMBL001 MINGW64 /d/prg/terraform-global/env_alpha_cn/access_points (env_alpha_cn|MERGING)

It behaves the same when the ARN is the right one too and I believe the aws provider's checks after applying get thrown off somehow.

[bondsbw]

Same experience as @vivanov-dp, except applying an arn:aws:... managed AWS policy in GovCloud which should have been arn:aws-us-gov:....

Using the correct partition fixed the issue and did not display this error.

AWS provider v2.58.0

[Frogvall]

Finally had time to look into this. Turns out, the error is not per se with persisting to state, but the read function deletes it from state if the profile running terraform doesn't have the elasticloadbalancing:DescribeListenerCertificates permission. There is no error message exept in debug mode, and it just silently removes the resource.

[YakDriver]

I think the proper "fix" here is to convert Read to return Diagnostics and a true warning that let's you know the resource has been removed from state.

[dossy]

Finally had time to look into this. Turns out, the error is not per se with persisting to state, but the read function deletes it from state if the profile running terraform doesn't have the elasticloadbalancing:DescribeListenerCertificates permission. There is no error message exept in debug mode, and it just silently removes the resource.

Just want to thank you, @Frogvall, that two years later, this is still an issue in Terraform v1.3.0 with hashicorp/aws v4.32.0, and I was absolutely stuck trying to get past this error:

aws_lb_listener_certificate.https_listener[0]: Creating...
╷
│ Error: Provider produced inconsistent result after apply
│ 
│ When applying changes to aws_lb_listener_certificate.https_listener[0],
│ provider "provider[\"registry.terraform.io/hashicorp/aws\"]" produced an
│ unexpected new value: Root resource was present, but now absent.
│ 
│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker.
╵
Releasing state lock. This may take a few moments...

And granting the Terraform role elasticloadbalancing:DescribeListenerCertificates fixed it!

Generally when an API call fails due to insufficient permission, Terraform is good about reporting it - if it did that in this case, it would certainly have saved me a bunch of time.

[Frogvall]

You're welcome! :)

[YakDriver]

Thanks, @Frogvall! Indeed, read was previously checking to see if the certificate was nil before erroring. It will always be nil when there is an error so you would never see an error in that section of code. This should be fixed soon.

[YakDriver]

@pieterjanpintens @neelam-007 @vivanov-dp I looked through the code for aws_iam_role_policy_attachment and do not see the same type of problem that was causing the problem with aws_lb_listener_certificate. It is possible that this problem has already been fixed but because it was lost on this semi-unrelated issue, there's nothing tying it back to the fix.

TL;DR: If you are still seeing issues with aws_iam_role_policy_attachment`, please 🙏 open a new issue so that we can prioritize and look at that!

[vivanov-dp]

@YakDriver Hey, thanks for taking care of this and sorry for sticking an unrelated problem to this issue back then, but in my defense - the error message was the same.

I tried aws_iam_role_policy_attachment with the wrong partition in the ARN:

  # aws_iam_role_policy_attachment.log_server_test will be created
  + resource "aws_iam_role_policy_attachment" "log_server_test" {
      + id         = (known after apply)
      + policy_arn = "arn:aws-cn:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role"
      + role       = "b12tf-dev-log-server"
    }

And the error is:
Error: error reading IAM Role Managed Policy Attachment (b12tf-dev-log-server:arn:aws-cn:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role): couldn't find resource

To me this means that, again, since AWS returns no indication of success or failure, Terraform expects the resource to have been created, can't read it and outputs a generic error.
This message is more helpful in identifying the problem though, because it points in the right direction and I consider it good enough, given the circumstances.

[YakDriver]

This message is more helpful in identifying the problem though, because it points in the right direction and I consider it good enough, given the circumstances.

Yeah, it's definitely better than what was happening with aws_lb_listener_certificate (basically nothing except removing from state) but it would still be better to have a create error. I'll poke around a little to see what it does this.

The normal retry after create actually is in the read. Normally it's in create but here it makes sense because the eventual consistency can happen in more places. It probably wouldn't hurt to also have a wait in create. But, either way it doesn't help much since there's no actual error from AWS. We're just making this one up based on the failure to create. It's a little wonky but hard to do much different based on the API reponses (lack of create/attach error).

[github-actions]

This functionality has been released in v4.51.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.