[Bug]: aws_redshift_parameter_group fails with ClusterParameterGroupNotFound if resource disappears

[petur]

Terraform Core Version

1.3.7

AWS Provider Version

4.52.0

Affected Resource(s)

• aws_redshift_parameter_group

Expected Behavior

If a parameter group created by terraform is deleted from the AWS console, the next terraform plan should show the resource being recreated.

Actual Behavior

The terraform plan fails with the error ClusterParameterGroupNotFound: ClusterParameterGroup test not found.

Relevant Error/Panic Output Snippet

$ terraform plan
aws_redshift_parameter_group.test: Refreshing state... [id=test]
╷
│ Error: reading Redshift Parameter Group (test): ClusterParameterGroupNotFound: ClusterParameterGroup test not found.
│       status code: 404, request id: f7191830-c92e-4bca-b2cc-bafa390c6493
│ 
│   with aws_redshift_parameter_group.test,
│   on main.tf line 1, in resource "aws_redshift_parameter_group" "test":
│    1: resource "aws_redshift_parameter_group" "test" {
│ 
╵

Terraform Configuration Files

resource "aws_redshift_parameter_group" "test" {
  family = "redshift-1.0"
  name   = "test"
}

Steps to Reproduce

1. Run terraform init and terraform apply to create a parameter group.
2. In the AWS console, find the parameter group and delete it.
3. Run terraform plan again.

Debug Output

2023-02-03T11:18:09.931Z [DEBUG] provider.terraform-provider-aws_v4.53.0_x5: Called provider defined Provider Configure: tf_req_id=5354e173-ae78-338f-ce0a-2bf188ec0710 @caller=github.com/hashicorp/terraform-plugin-framework@v1.1.1/internal/fwserver/server_configureprovider.go:20 @module=sdk.framework tf_mux_provider=*proto5server.Server tf_provider_addr=registry.terraform.io/hashicorp/aws tf_rpc=ConfigureProvider timestamp=2023-02-03T11:18:09.931Z
2023-02-03T11:18:09.932Z [DEBUG] ReferenceTransformer: "aws_redshift_parameter_group.test" references: []
aws_redshift_parameter_group.test: Refreshing state... [id=test]
2023-02-03T11:18:09.933Z [DEBUG] provider.terraform-provider-aws_v4.53.0_x5: HTTP Request Sent: http.request.header.x_amz_date=20230203T111809Z http.request.header.x_amz_security_token=***** http.url=https://redshift.us-east-1.amazonaws.com/ tf_rpc=ReadResource http.request.header.content_type="application/x-www-form-urlencoded; charset=utf-8" aws.operation=DescribeClusterParameterGroups aws.region=us-east-1 http.method=POST http.request.header.authorization="AWS4-HMAC-SHA256 Credential=********************/20230203/us-east-1/redshift/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-date;x-amz-security-token, Signature=*****" http.request_content_length=80 tf_mux_provider=*schema.GRPCProviderServer @module=aws tf_resource_type=aws_redshift_parameter_group http.flavor=1.1 tf_req_id=f4385e12-6557-c797-6c4b-198ac5f4a946 aws.service=Redshift http.request.body=Action=DescribeClusterParameterGroups&ParameterGroupName=test&Version=2012-12-01 http.user_agent="APN/1.0 HashiCorp/1.0 Terraform/1.3.7 (+https://www.terraform.io) terraform-provider-aws/4.53.0 (+https://registry.terraform.io/providers/hashicorp/aws) aws-sdk-go/1.44.191 (go1.19.3; linux; amd64)" net.peer.name=redshift.us-east-1.amazonaws.com tf_provider_addr=registry.terraform.io/hashicorp/aws @caller=github.com/hashicorp/aws-sdk-go-base/v2/awsv1shim/v2@v2.0.0-beta.23/logger.go:90 aws.sdk=aws-sdk-go timestamp=2023-02-03T11:18:09.933Z
2023-02-03T11:18:10.585Z [DEBUG] provider.terraform-provider-aws_v4.53.0_x5: HTTP Response Received: http.response.body="<ErrorResponse xmlns="http://redshift.amazonaws.com/doc/2012-12-01/">
  <Error>
    <Type>Sender</Type>
    <Code>ClusterParameterGroupNotFound</Code>
    <Message>ClusterParameterGroup test not found.</Message>
  </Error>
  <RequestId>8a0085d5-2d07-44d9-8962-227d4d971657</RequestId>
</ErrorResponse>
" http.response.header.content_type=text/xml http.response.header.date="Fri, 03 Feb 2023 11:18:10 GMT" http.response.header.x_amzn_requestid=8a0085d5-2d07-44d9-8962-227d4d971657 http.response_content_length=302 tf_mux_provider=*schema.GRPCProviderServer @caller=github.com/hashicorp/aws-sdk-go-base/v2/awsv1shim/v2@v2.0.0-beta.23/logger.go:138 aws.sdk=aws-sdk-go tf_provider_addr=registry.terraform.io/hashicorp/aws tf_req_id=f4385e12-6557-c797-6c4b-198ac5f4a946 tf_resource_type=aws_redshift_parameter_group http.duration=651 @module=aws aws.operation=DescribeClusterParameterGroups aws.region=us-east-1 aws.service=Redshift http.status_code=404 tf_rpc=ReadResource timestamp=2023-02-03T11:18:10.585Z
2023-02-03T11:18:10.586Z [ERROR] provider.terraform-provider-aws_v4.53.0_x5: Response contains error diagnostic: @module=sdk.proto tf_provider_addr=registry.terraform.io/hashicorp/aws tf_resource_type=aws_redshift_parameter_group tf_rpc=ReadResource @caller=github.com/hashicorp/terraform-plugin-go@v0.14.3/tfprotov5/internal/diag/diagnostics.go:55 diagnostic_detail= diagnostic_severity=ERROR diagnostic_summary="reading Redshift Parameter Group (test): ClusterParameterGroupNotFound: ClusterParameterGroup test not found.
        status code: 404, request id: 8a0085d5-2d07-44d9-8962-227d4d971657" tf_proto_version=5.3 tf_req_id=f4385e12-6557-c797-6c4b-198ac5f4a946 timestamp=2023-02-03T11:18:10.585Z
2023-02-03T11:18:10.586Z [ERROR] vertex "aws_redshift_parameter_group.test" error: reading Redshift Parameter Group (test): ClusterParameterGroupNotFound: ClusterParameterGroup test not found.
        status code: 404, request id: 8a0085d5-2d07-44d9-8962-227d4d971657
2023-02-03T11:18:10.587Z [ERROR] vertex "aws_redshift_parameter_group.test (expand)" error: reading Redshift Parameter Group (test): ClusterParameterGroupNotFound: ClusterParameterGroup test not found.
        status code: 404, request id: 8a0085d5-2d07-44d9-8962-227d4d971657
2023-02-03T11:18:10.587Z [INFO]  backend/local: plan operation completed
╷
│ Error: reading Redshift Parameter Group (test): ClusterParameterGroupNotFound: ClusterParameterGroup test not found.
│       status code: 404, request id: 8a0085d5-2d07-44d9-8962-227d4d971657
│ 
│   with aws_redshift_parameter_group.test,
│   on main.tf line 1, in resource "aws_redshift_parameter_group" "test":
│    1: resource "aws_redshift_parameter_group" "test" {
│ 
╵

Panic Output

No response

Important Factoids

No response

References

No response

Would you like to implement a fix?

No

[github-actions]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[petur]

Destroying via terraform also returns an error:

$ terraform destroy
aws_redshift_parameter_group.test: Refreshing state... [id=test]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  - destroy

Terraform will perform the following actions:

  # aws_redshift_parameter_group.test will be destroyed
  - resource "aws_redshift_parameter_group" "test" {
      - arn         = "arn:aws:redshift:us-east-1:****************:parametergroup:test" -> null
      - description = "Managed by Terraform" -> null
      - family      = "redshift-1.0" -> null
      - id          = "test" -> null
      - name        = "test" -> null
      - tags        = {} -> null
      - tags_all    = {} -> null
    }

Plan: 0 to add, 0 to change, 1 to destroy.

Do you really want to destroy all resources?
  Terraform will destroy all your managed infrastructure, as shown above.
  There is no undo. Only 'yes' will be accepted to confirm.

  Enter a value: yes

aws_redshift_parameter_group.test: Destroying... [id=test]
╷
│ Error: deleting Redshift Parameter Group (test): %!s(<nil>)
│ 
│ 
╵

[petur]

This error in terraform destroy actually causes the acceptance tests to fail:

$ make testacc TESTS=TestAccRedshiftParameterGroup PKG=redshift
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./internal/service/redshift/... -v -count 1 -parallel 20 -run='TestAccRedshiftParameterGroup'  -timeout 180m
=== RUN   TestAccRedshiftParameterGroup_basic
=== PAUSE TestAccRedshiftParameterGroup_basic
=== RUN   TestAccRedshiftParameterGroup_withParameters
=== PAUSE TestAccRedshiftParameterGroup_withParameters
=== RUN   TestAccRedshiftParameterGroup_withoutParameters
=== PAUSE TestAccRedshiftParameterGroup_withoutParameters
=== RUN   TestAccRedshiftParameterGroup_withTags
=== PAUSE TestAccRedshiftParameterGroup_withTags
=== CONT  TestAccRedshiftParameterGroup_basic
=== CONT  TestAccRedshiftParameterGroup_withoutParameters
=== CONT  TestAccRedshiftParameterGroup_withParameters
=== CONT  TestAccRedshiftParameterGroup_withTags
=== CONT  TestAccRedshiftParameterGroup_withoutParameters
    testing_new.go:84: Error running post-test destroy, there may be dangling resources: exit status 1
        
        Error: deleting Redshift Parameter Group (test-terraform-3337553649151077186): %!s(<nil>)
        
--- FAIL: TestAccRedshiftParameterGroup_withoutParameters (26.08s)
=== CONT  TestAccRedshiftParameterGroup_withParameters
    testing_new.go:84: Error running post-test destroy, there may be dangling resources: exit status 1
        
        Error: deleting Redshift Parameter Group (test-terraform-1547568073148137787): %!s(<nil>)
        
--- FAIL: TestAccRedshiftParameterGroup_withParameters (26.28s)
=== CONT  TestAccRedshiftParameterGroup_basic
    testing_new.go:84: Error running post-test destroy, there may be dangling resources: exit status 1
        
        Error: deleting Redshift Parameter Group (test-terraform-2780350361488450119): %!s(<nil>)
        
--- FAIL: TestAccRedshiftParameterGroup_basic (26.28s)
=== CONT  TestAccRedshiftParameterGroup_withTags
    testing_new.go:84: Error running post-test destroy, there may be dangling resources: exit status 1
        
        Error: deleting Redshift Parameter Group (test-terraform-4244436034503235798): %!s(<nil>)
        
--- FAIL: TestAccRedshiftParameterGroup_withTags (58.28s)
FAIL
FAIL    github.com/hashicorp/terraform-provider-aws/internal/service/redshift   58.347s
FAIL
make: *** [GNUmakefile:93: testacc] Error 1

[petur]

Bisected using make testacc TESTS=TestAccRedshiftParameterGroup PKG=redshift:

$ git bisect log
git bisect start
# good: [ef7b6ad8a06dac041f601a3df21515299b1239e5] Update CHANGELOG.md for #28472
git bisect good ef7b6ad8a06dac041f601a3df21515299b1239e5
# bad: [757460886c8f9019454f4ffe792f208fbb24779f] Update CHANGELOG.md after v4.53.0
git bisect bad 757460886c8f9019454f4ffe792f208fbb24779f
# good: [73bc5649ccc107a4eb12b8fc585ab6f895774d37] Add CHANGELOG entry.
git bisect good 73bc5649ccc107a4eb12b8fc585ab6f895774d37
# bad: [69f67793e7b5de6c01aa9f3f462536a130bbe1e2] d/aws_auditmanager_control: docs
git bisect bad 69f67793e7b5de6c01aa9f3f462536a130bbe1e2
# good: [a4fde29a11742a165388a4e7a526c89c6b639743] TestSubdirectoryFromLocationURI: Support object storage locations.
git bisect good a4fde29a11742a165388a4e7a526c89c6b639743
# bad: [cd0112262c4dd18beb5ee6f1fe1c5a14d9ccc163] Test fixes
git bisect bad cd0112262c4dd18beb5ee6f1fe1c5a14d9ccc163
# skip: [56821d6f81fd30c74ddf1b03456b212ce42128fb] Fixes bare errors in `emr`
git bisect skip 56821d6f81fd30c74ddf1b03456b212ce42128fb
# bad: [a19251459c6358cb7fa7cc3d54c0cea2aab10e2f] Adds semgrep rule to default file
git bisect bad a19251459c6358cb7fa7cc3d54c0cea2aab10e2f
# skip: [fb3a12e9fcc748949a8a0faee8cea1e073dd9732] Fixes bare errors in `events`
git bisect skip fb3a12e9fcc748949a8a0faee8cea1e073dd9732
# skip: [0fe4aae170b37e30de881860afce718cfa74ce82] Fixes bare errors in `ecr`
git bisect skip 0fe4aae170b37e30de881860afce718cfa74ce82
# good: [c676bb691ef063a81c6f84df0b3685bde790a98c] Fixes bare errors in `waf`
git bisect good c676bb691ef063a81c6f84df0b3685bde790a98c
# skip: [81981170a3569876c746bed797889cbceefe4f91] Fixes bare errors in `elastictranscoder`
git bisect skip 81981170a3569876c746bed797889cbceefe4f91
# bad: [afe3427f2af1a905166afb72399d5a59ebbf5e30] Fixes bare errors in `rds`
git bisect bad afe3427f2af1a905166afb72399d5a59ebbf5e30
# good: [0f163db12903fd917b8e5ab12172d0cdab12eea0] Fixes bare errors in `serverlessrepo`
git bisect good 0f163db12903fd917b8e5ab12172d0cdab12eea0
# good: [592d3f785afcf8a37bec83122e5ebffe6ab0d4ea] Fixes bare errors in `s3`
git bisect good 592d3f785afcf8a37bec83122e5ebffe6ab0d4ea
# good: [4bf93e0b70fd73d09b27e469c28d8097920522a3] Fixes bare errors in `redshiftserverless`
git bisect good 4bf93e0b70fd73d09b27e469c28d8097920522a3
# bad: [6ff207a108317e980bc3e245bcb62d2e28e67662] Fixes bare errors in `redshift`
git bisect bad 6ff207a108317e980bc3e245bcb62d2e28e67662
# first bad commit: [6ff207a108317e980bc3e245bcb62d2e28e67662] Fixes bare errors in `redshift`

Seems to be introduced by 6ff207a

[ewbankkit]

@petur Thanks for raising this issue 👏.
I noticed this while doing some tech debt work in #27991 and have fixed it in that PR.
I aim to get it merged for next week's release.

[sebastianvillarroel]

@ewbankkit hi! Is there any workaround possible? I destroyed a Redshift deployment for my verification and wanted to re-apply; however, I face the same as @petur

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.