Error describing Service Catalog Provisioned Product (pp-*******): unexpected state 'TAINTED', wanted target 'AVAILABLE'. last error: %!s(<nil>) & Not able to run update or destroy after this state

[kush-sumit]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

Terraform v1.1.9
on darwin_arm64

• provider registry.terraform.io/hashicorp/aws v4.12.1
• provider registry.terraform.io/hashicorp/random v3.1.3

Affected Resource(s)

• aws_servicecatalog_provisioned_product

Service Catalog Product Yaml File

AWSTemplateFormatVersion: 2010-09-09
Description: Sumit Initial CloudFormation script

Parameters:
  AwsLogicalAccountNameLowerCase:
    Description: Logical AWS account name e.g. kush
    Type: String

Resources:
################S3 Buckets################
  KushResourceBucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: !Join ['', [!Ref AwsLogicalAccountNameLowerCase, '-learning-1000-resouces']]
      BucketEncryption:
        ServerSideEncryptionConfiguration:
        - ServerSideEncryptionByDefault:
            SSEAlgorithm: AES256
      PublicAccessBlockConfiguration:
          BlockPublicAcls: True
          BlockPublicPolicy: True
          IgnorePublicAcls: True
          RestrictPublicBuckets: True

################S3 Bucket Policies################
  KushResourceBucketPolicy:
    Type: AWS::S3::BucketPolicy
    Properties:
      Bucket: !Ref KushResourceBucket
      PolicyDocument:
        Statement:
          - Sid: DenyInsecureConnections
            Effect: Deny
            Principal: '*'
            Action: '*'
            Resource: !Join ['', ['arn:aws:s3:::', !Ref AwsLogicalAccountNameLowerCase, '-learning-1000-resouces/*']]
            Condition:
              Bool:
                aws:SecureTransport: false
          - Sid: DenyIncorrectEncryptionHeader
            Effect: Deny
            Principal: '*'
            Action:
              - s3:PutObject
            Resource: !Join ['', ['arn:aws:s3:::', !Ref AwsLogicalAccountNameLowerCase, '-learning-1000-resouces/*']]
            Condition:
              StringNotEquals:
                s3:x-amz-server-side-encryption: AES256
          - Sid: DenyUnEncryptedObjectUploads
            Effect: Deny
            Principal: '*'
            Action:
              - s3:PutObject
            Resource: !Join ['', ['arn:aws:s3:::', !Ref AwsLogicalAccountNameLowerCase, '-learning-1000-resouces/*']]
            Condition:
              'Null':
                s3:x-amz-server-side-encryption: true
          - Sid: DenyUnwantedCannedACLs
            Effect: Deny
            Principal: '*'
            Action:
              - s3:PutObject
            Resource: !Join ['', ['arn:aws:s3:::', !Ref AwsLogicalAccountNameLowerCase, '-learning-1000-resouces/*']]
            Condition:
              StringEquals:
                s3:x-amz-acl:
                  - public-read
                  - public-read-write
                  - aws-exec-read
                  - authenticated-read
                  - bucket-owner-read
                  - log-delivery-write

Terraform Configuration Files

Please include all Terraform configurations required to reproduce the bug. Bug reports without a functional reproduction may be closed without investigation.

# version.tf

terraform {
  // Fix version to ensure future compatibility if an updates is deployed
  required_version = ">= 1.1.9"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = ">= 3.74"
    }
  }
  }

# provider.tf

provider "aws" {
  region = "eu-central-1"
}

# variables.tf

variable "product_id" {
  description = "Enter Product_Id"
  type        = string
  default     = "prod-f***********"
}

variable "provisioning_artifact_id" {
  description = "Enter provisioning_artifact_id"
  type        = string
  default     = "pa-k**************"
}

variable "aws_logical_account_name_lower_case" {
  description = "Enter aws_logical_account_name_lower_case"
  type        = string
  default     = "kush3"
}

variable "provisioned_product_name" {
  description = "Enter provisioned_product_name"
  type        = string
  default     = "Test_Bucket_Creation"
}

# main.tf


resource "aws_servicecatalog_provisioned_product" "s3_bucket" {
  // Product Name
  name         = var.provisioned_product_name

  # Service catalog parameters
  product_id                = var.product_id
  provisioning_artifact_id  = var.provisioning_artifact_id
  // Service catalog provisioning parameters

  provisioning_parameters {
    key   = "AwsLogicalAccountNameLowerCase"
    value = var.aws_logical_account_name_lower_case
  }

}

# Give all the outputs exported from service catalog
output "Provisioned_Product_Outputs" {
  value = aws_servicecatalog_provisioned_product.s3_bucket.outputs
}

Expected Behaviour

Terraform should provide the actual error message which is encountered during provisioning/deployment of product.

Actual Behaviour

Error: error describing Service Catalog Provisioned Product (pp-*********): unexpected state 'TAINTED', wanted target 'AVAILABLE'. last error: %!s()
│

Steps to Reproduce

1. terraform init
2. terraform plan
3. terraform apply

Once the bucket is created, Create an aws bucket in the account by name “kush-learning-1000-resouces”.
now update your variable "aws_logical_account_name_lower_case" value to kush and then hit apply.

4. terraform apply

you will see that deployment fails with error as a bucket is already present by this name and put the product in tainted state and all you will get is the error message

Error: error describing Service Catalog Provisioned Product (pp-*********): unexpected state 'TAINTED', wanted target 'AVAILABLE'. last error: %!s()

now if you try to revert back the value to "kush3" and hit terraform apply you will only get Error and same is case with below terraform destory

6. terraform destroy

Important Factoids

There are two bugs as below.

1. Once while provisioning the product, if during provisioning of product an error is encountered terraform throws the error shown in Actual behaviour, which is highly confusion. The expectation is to display actual error in the field of **last error: ** instead of %!s(),

I tried simply running the aws cli command AWS CLI Documentation
aws servicecatalog describe-provisioned-product --id pp-***** and as per the output the value from "StatusMessage" should be displayed.

2. Major Bug is once the product goes into the tainted or Error state and then when we try to run an update/destroy of product by applying terraform apply, post making changes in input which caused the provisioning to fail. Terraform fails to apply update to the product and simply throws same error Error: error describing Service Catalog Provisioned Product (pp-*****): unexpected state 'TAINTED', wanted target 'AVAILABLE'. last error: %!s()

References

• #0000

[kush-sumit]

I am still getting the same below 2 issue. Kindly re-open the issue @anGie44

1. Not able to run any updates using terraform apply or terraform destroy in case of provisioned products goes into tainted and error state.
2. Error Message is not displayed properly its just says last error: %!s(), it should rather display value obtained in status message field of describe provisioned product api.

terraform -version

Please find below screenshot for reference

[kush-sumit]

@anGie44 @justinretzolk :

I tried to test scenarios for the bugs raises with latest release version 4.14

We are still getting same issue, where in we are not able to run update or destroy on the provisioned product whose state has become either tainted/error, Could you please open the issue again.

The other issue about populating the error message is solved, we could now see the error message.
please find below screenshot for reference:

[anGie44]

Hi @kush-sumit , thanks for your reply here. Can you confirm which errors you're seeing now with v4.14.0 of the aws provider. Does it include one or both of the following:

• error describing Service Catalog Provisioned Product (<product-name>): unexpected state...
• error describing Service Catalog Provisioned Product (<product-name>) Record (<record-id>): unexpected state...

Thanks in advance!

[kush-sumit]

Hi @kush-sumit , thanks for your reply here. Can you confirm which errors you're seeing now with v4.14.0 of the aws provider. Does it include one or both of the following:

• error describing Service Catalog Provisioned Product (<product-name>): unexpected state...
• error describing Service Catalog Provisioned Product (<product-name>) Record (<record-id>): unexpected state...

Thanks in advance!

@anGie44 : Thanks for quick acknowledgement, when i hit terraform apply i could see only this one which is second one in your list

aws_servicecatalog_provisioned_product.s3_bucket: Refreshing state... [id=pp-ixoexjt2mopy2]
╷
│ Error: error describing Service Catalog Provisioned Product (pp-ixoexjt2mopy2) Record (rec-7ye5hezyooqu4): unexpected state 'FAILED', wanted target 'SUCCEEDED, AVAILABLE'. last error: 1 error occurred:
│ * : Errors from CloudFormation: [{LogicalResourceId : SC-514039224447-pp-ixoexjt2mopy2, ResourceType : AWS::CloudFormation::Stack, StatusReason : The following resource(s) failed to update: [KushResourceBucket]. }, {LogicalResourceId : KushResourceBucket, ResourceType : AWS::S3::Bucket, StatusReason : kush2-learning-1000-resouces already exists}, {LogicalResourceId : KushResourceBucket, ResourceType : AWS::S3::Bucket, StatusReason : Requested update requires the creation of a new physical resource; hence creating one.}, {LogicalResourceId : SC-514039224447-pp-ixoexjt2mopy2, ResourceType : AWS::CloudFormation::Stack, StatusReason : User Initiated}, {LogicalResourceId : KushResourceBucketPolicy, ResourceType : AWS::S3::BucketPolicy, StatusReason : Resource creation Initiated}, {LogicalResourceId : KushResourceBucket, ResourceType : AWS::S3::Bucket, StatusReason : Resource creation Initiated}, {LogicalResourceId : SC-514039224447-pp-ixoexjt2mopy2, ResourceType : AWS::CloudFormation::Stack, StatusReason : User Initiated}]

I belive we should take the error message from

• error describing Service Catalog Provisioned Product (<product-name>) Record (<record-id>): unexpected state... and display if during deployment a failure occurs and product goes into tainted or error state like one which we are getting now with error message much clearer

But when a subsequent update or destroy is triggered then the product should either be triggered for update or terminate in case of destroy command and not display same error and exit which was original bug

[anGie44]

I see, thanks for confirming which error message you're most recently running in to @kush-sumit ! I'll re-open this on account that it's related to the original issue, just occurring a bit further down in the resource code:

terraform-provider-aws/internal/service/servicecatalog/provisioned_product.go

Lines 420 to 429 in 6e8fe78

	recordOutput, err := WaitRecordReady(conn, acceptLanguage, aws.StringValue(detail.LastProvisioningRecordId), RecordReadyTimeout)
	if !d.IsNewResource() && tfawserr.ErrCodeEquals(err, servicecatalog.ErrCodeResourceNotFoundException) {
	log.Printf("[WARN] Service Catalog Provisioned Product (%s) Record (%s) not found, unable to set tags", d.Id(), aws.StringValue(detail.LastProvisioningRecordId))
	return nil
	}
	if err != nil {
	return fmt.Errorf("error describing Service Catalog Provisioned Product (%s) Record (%s): %w", d.Id(), aws.StringValue(detail.LastProvisioningRecordId), err)
	}

[kush-sumit]

Hi @anGie44: Thanks for working on the issue and releasing the fixes.
I Just now tried to test the fix and below are my findings.

1. Error Message display issue --> Fixed
2. Terraform update or destroy run on provisioned product state (tainted or error) --> Fixed

Issue Introduced or Still Exists
Now when the product goes into tainted states instead of throwing that as error terraform is now displaying the product to be in available state, where as on aws console we could see the error, this was working fine after first fix for error message display was release.

I assume this is happening as you have added tainted also into wanted target instead of unexpected state.

Could you please re-open this ticket and fix the issue.

please find below snippet for your reference.

Wrong Status from Terraform Console

Actual Status in AWS Console

Error Message in AWS Console

please let me know any inputs further required from my end.

Regards
Sumit

[kush-sumit]

Hi @anGie44 @justinretzolk : Could you share any leads on above feedback regarding issue of tainted state?

[anGie44]

Hi @kush-sumit , the errors you are seeing in the AWS Console (in the provider, they are similarly fetched when calling the API method DescribeRecord not DescribeProvisionedProduct) are visible if you set your env var TF_LOG=warn when using terraform as we do not want to exit during the read phase, else we'll enter that irreversible state again where you can't read nor delete the resource without manual intervention. Nevertheless, I would still think that the status parameter within the resource would return TAINTED not AVAILABLE..unless behind the scenes, that occurs after a failed update and a subsequent number of reads. Do you mind confirming?

I do see how the tainted state should warrant a more immediate error to the user so I'll update that portion to return with the detailed status message.

I'll also make a note in the documentation about debugging if it's not already present.

[kush-sumit]

Hi @anGie44: While doing the update of a provisioned product, if the there is a failure and product becomes tainted with the recent changes as part of fix terraform thinks that the deployment was successfull and as you could see in first screenshot it simply outputs modification got completed and 1 changed successfully.

The request would be to just move out tainted stated from list of wanted state and we could just keep state "available" only there like before. This would then automatically throw the error record like it was doing after first fix where as the product went into tainted state terraform threw the output stating product is in tainted state and displayed the error record.

[kush-sumit]

Hi @anGie44: Thanks for releasing the fix. I did a round of testing and all seems to be working fine now. Thanks for your help on this issue

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

[YakDriver]

I apologize for the back and forth on this and that this was closed when the error is still happening. We have a new fix in the works. However, please note that now this resource will continue to function when a stack has a TAINTED status. This is because the stack is in a

Stable state, ready to perform any operation. The stack has completed the requested operation but is not exactly what was requested. For example, a request to update to a new version failed and the stack rolled back to the current version.

You can use status and status_message or logging to find out more about if this happens to your stack.

This seems like the right and only way forward but please let us know if you disagree.

[github-actions]

This functionality has been released in v4.63.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.