aws_s3_bucket_lifecycle_configuration and two rules with same prefix

[Tenzer]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

Terraform v1.1.5
on darwin_arm64
+ provider registry.terraform.io/hashicorp/archive v2.2.0
+ provider registry.terraform.io/hashicorp/aws v4.0.0

Affected Resource(s)

• aws_s3_bucket_lifecycle_configuration

Terraform Configuration Files

Please include all Terraform configurations required to reproduce the bug. Bug reports without a functional reproduction may be closed without investigation.

resource "aws_s3_bucket" "static" {
  bucket = "static-assets"
}

resource "aws_s3_bucket_lifecycle_configuration" "static" {
  bucket = aws_s3_bucket.static.id

  rule {
    id     = "Keep previous version 30 days, then in Glacier another 60"
    status = "Enabled"

    noncurrent_version_transition {
      noncurrent_days = 30
      storage_class   = "GLACIER"
    }

    noncurrent_version_expiration {
      noncurrent_days = 90
    }
  }

  rule {
    id     = "Delete old incomplete multi-part uploads"
    status = "Enabled"

    abort_incomplete_multipart_upload {
      days_after_initiation = 7
    }
  }
}

Expected Behavior

The two lifecycle rules should continue working as they did in the pre-4.0.0 provider versions.

Actual Behavior

I can import the resource fine, but when I try to apply again, the following error comes up:

╷
│ Error: error updating S3 Bucket Lifecycle Configuration (static-assets): InvalidRequest: Found two rules with same prefix ''
│ 	status code: 400, request id: YX4MWJPJS7REVCCB, host id: bXl1MJ3xNbTq/moOawG69ulbUL4qnkYYI7Bdc1MbdmRQI+WJ6XG9+y4PO3Mt84X2hy46B3DHaqU=
│
│   with aws_s3_bucket_lifecycle_configuration.static,
│   on aws-s3-static.tf line 57, in resource "aws_s3_bucket_lifecycle_configuration" "static":
│   57: resource "aws_s3_bucket_lifecycle_configuration" "static" {

Steps to Reproduce

1. terraform apply

Important Factoids

Putting all the rules inside the same rule {} block seems to work, but it makes it more difficult to organise rules that aren't directly related.

Looking in the debug log output, I can see the payload being sent to AWS is:

<LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
    <Rule>
        <NoncurrentVersionTransition>
            <NoncurrentDays>30</NoncurrentDays>
            <StorageClass>GLACIER</StorageClass>
        </NoncurrentVersionTransition>
        <Prefix></Prefix>
        <Status>Enabled</Status>
        <ID>Keep previous version 30 days, then in Glacier another 60</ID>
        <NoncurrentVersionExpiration>
            <NoncurrentDays>90</NoncurrentDays>
        </NoncurrentVersionExpiration>
    </Rule>
    <Rule>
        <AbortIncompleteMultipartUpload>
            <DaysAfterInitiation>7</DaysAfterInitiation>
        </AbortIncompleteMultipartUpload>
        <ID>Delete old incomplete multi-part uploads</ID>
        <Prefix></Prefix>
        <Status>Enabled</Status>
    </Rule>
</LifecycleConfiguration>

I can see Prefix is deprecated according to the documentation, could that be related? https://docs.aws.amazon.com/AmazonS3/latest/API/API_LifecycleRule.html#AmazonS3-Type-LifecycleRule-Prefix

References

• Discussion around changes made to the aws_s3_bucket resource in AWS Provider version 4.0 #23106

[YakDriver]

@Tenzer Thank you for raising this! I believe this is related to #23129 but I will test to make sure the same fix works for both.

[github-actions]

This functionality has been released in v4.1.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[Tenzer]

Could this be reopened again, please? It's now a problem again with version 4.2.0 of the provider, probably due to the changes in #23252.

[anGie44]

Hi @Tenser, thank you for reaching out. Happy to re-open this if the following suggestion does not prove to be effective. Going by the description provided, it sounds like you are first importing these new resources into terraform state and then performing an update via terraform apply? I believe in general the issue stems from how filter is represented in terraform which was not exactly configurable in the original aws_s3_bucket resource but was configured behind the scenes via the AWS Go SDK.

** If you have the aws CLI handy, could you call aws s3api get-bucket-lifecycle-configuration <bucket-name> to help inform how filter should be specified in terraform.

Based on the aws CLI result (or other information available), map each rule's Filter value in Terraform e.g. if Filter gets returned as "Filter": {} then add the filter block as follows:

resource "aws_s3_bucket_lifecycle_configuration" "static" {
  bucket = aws_s3_bucket.static.id

  rule {
    id     = "Keep previous version 30 days, then in Glacier another 60"
    status = "Enabled"

    filter {}

    noncurrent_version_transition {
      noncurrent_days = 30
      storage_class   = "GLACIER"
    }

    noncurrent_version_expiration {
      noncurrent_days = 90
    }
  }

  rule {
    id     = "Delete old incomplete multi-part uploads"
    status = "Enabled"

    filter {}

    abort_incomplete_multipart_upload {
      days_after_initiation = 7
    }
  }
}

If instead the rule's Filter gets returned as "Filter": {"Prefix": "" }, then add filter { prefix = "" } as follows:

resource "aws_s3_bucket_lifecycle_configuration" "static" {
  bucket = aws_s3_bucket.static.id

  rule {
    id     = "Keep previous version 30 days, then in Glacier another 60"
    status = "Enabled"

    filter {
      prefix = ""
    }

    noncurrent_version_transition {
      noncurrent_days = 30
      storage_class   = "GLACIER"
    }

    noncurrent_version_expiration {
      noncurrent_days = 90
    }
  }

  rule {
    id     = "Delete old incomplete multi-part uploads"
    status = "Enabled"

    filter {
      prefix = ""
    }

    abort_incomplete_multipart_upload {
      days_after_initiation = 7
    }
  }
}

[Tenzer]

The response in the aws command is:

{
    "Rules": [
        {
            "ID": "Keep previous version 30 days, then in Glacier another 60",
            "Filter": {
                "Prefix": ""
            },
            "Status": "Enabled",
            "NoncurrentVersionTransitions": [
                {
                    "NoncurrentDays": 30,
                    "StorageClass": "GLACIER"
                }
            ],
            "NoncurrentVersionExpiration": {
                "NoncurrentDays": 90
            }
        },
        {
            "ID": "Delete old incomplete multi-part uploads",
            "Filter": {
                "Prefix": ""
            },
            "Status": "Enabled",
            "AbortIncompleteMultipartUpload": {
                "DaysAfterInitiation": 7
            }
        }
    ]
}

Adding empty filter {} blocks to both of the rule blocks means Terraform no longer sees a diff when I apply changes.

It would seem like the empty filter {} block only is required when you have two or more rules which operate on the entire bucket. Is this something that could be fixed or should it be documented anywhere?

[anGie44]

Hi @Tenzer my understanding from the AWS API is that, in general, the filter block is required when prefix is not specified and not contingent on the number of rules present as seen here and

The Filter is used to identify objects that a Lifecycle Rule applies to. A Filter must have exactly one of Prefix, Tag, or And specified. Filter is required if the LifecycleRule does not containt a Prefix element.

Since prefix alone is deprecated by S3,

Prefix identifying one or more objects to which the rule applies. This is no longer used; use Filter instead.

filter should be the one in use. The documentation was recently updated to reflect this, but if it's. still not clear, we can certainly add more clarifiying details.

[anGie44]

Adding empty filter {} blocks to both of the rule blocks means Terraform no longer sees a diff when I apply changes.

After import, the aws_s3_bucket_lifecycle_configuration resource technically shouldn't need an update or subsequent terraform apply unless there is an intended configuration change e.g. changing the prefix value in the filter block to a non-empty string.

[gtmtech]

I added another ticket which is kinda a duplicate, #23730

But this one is still Closed and this very much needs fixing, as it's not currently possible in v4 to create both an expiry days and an expiry expired_object_delete_marker without it - which is allowed by the api (and needed to save storage costs)

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.