Using certain features when creating s3 bucket lifecycle rules on all files throws errors

[reelacmnaes]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

When I try to create my S3 bucket lifecycle rule with noncurrent expiration and transition rules on all objects in a bucket, I get the following error

Error: error creating S3 Lifecycle Configuration for bucket (omitted): InvalidRequest: NewerNoncurrentVersions element can only be used in Lifecycle V2.
status code: 400, request id: omitted, host id: omitted

Failing code

This is the code that produces the above error (some of it is omitted for brevity)

resource "aws_s3_bucket" "this" {
  bucket = "${local.prefix}-bucket"

  lifecycle {
    ignore_changes = [
      lifecycle_rule
    ]
  }
}

resource "aws_s3_bucket_public_access_block" "this" {
  bucket = aws_s3_bucket.this.bucket

  block_public_acls   = true
  block_public_policy = true
}

resource "aws_s3_bucket_versioning" "this" {
  bucket = aws_s3_bucket.this.bucket

  versioning_configuration {
    status = "Enabled"
  }
}

resource "aws_s3_bucket_lifecycle_configuration" "this" {
  bucket = aws_s3_bucket.this.bucket 
  rule {
    status = "Enabled"
    id     = "${local.prefix}-rotation"
    # delete if objects are older than noncurrent_days and older than n-newer_noncurrent_versions
    noncurrent_version_expiration {
      noncurrent_days           = 30
      newer_noncurrent_versions = 2
    }
    # move to standard IA if older than 30 days for all noncurrent versions
    noncurrent_version_transition {
      storage_class             = "STANDARD_IA"
      noncurrent_days           = 30
    }
  }
}

Working code

By adding an empty filter {} block to the aws_s3_bucket_lifecycle_configuration resource, the error stops appearing, and the configuration applies successfully (again, some code missing for brevity):

resource "aws_s3_bucket" "this" {
  bucket = "${local.prefix}-bucket"

  lifecycle {
    ignore_changes = [
      lifecycle_rule
    ]
  }
}

resource "aws_s3_bucket_public_access_block" "this" {
  bucket = aws_s3_bucket.this.bucket

  block_public_acls   = true
  block_public_policy = true
}

resource "aws_s3_bucket_versioning" "this" {
  bucket = aws_s3_bucket.this.bucket

  versioning_configuration {
    status = "Enabled"
  }
}

resource "aws_s3_bucket_lifecycle_configuration" "this" {
  bucket = aws_s3_bucket.this.bucket
  rule {
    filter {} # workaround error
    status = "Enabled"
    id     = "${local.prefix}-rotation"
    # delete if objects are older than noncurrent_days and older than n-newer_noncurrent_versions
    noncurrent_version_expiration {
      noncurrent_days           = 30
      newer_noncurrent_versions = 2
    }
    # move to standard IA if older than 30 days for all noncurrent versions
    noncurrent_version_transition {
      storage_class             = "STANDARD_IA"
      noncurrent_days           = 30
    }
  }
}

Here is my AWS provider config:

terraform {
  required_version = ">=0.14.4"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 4.1" 
    }
}
provider "aws" {
  region = "eu-west-1"
}

I think the provider just needs to be updated to default to Lifecycle V2 rules? Though I appreciate that it might not be as straightforward as that.

Affected Resources

aws_s3_bucket_lifecycle_configuration

[kjschiroo]

We're having a very similar issue to this and found the same work around. The nuance we're running into now is that the workaround (filter {}) causes resource churn. If you apply and then apply again terraform will still think it needs to add the filter

          + filter {
            }

So it gets us around the error, but it isn't a great long term solution.

[YakDriver]

The new challenge here is getting the resource to work well for apply and import. This is likely related to fixes for importing in #23144.

[reelacmnaes]

@kjschiroo I stop getting resource churn by adding an ignore_changes on the aws_s3_bucket_likecycle_configuration resource. HTH

resource "aws_s3_bucket_lifecycle_configuration" "this" {
  bucket = aws_s3_bucket.this.bucket
  
  rule {

    filter {} # Workaround for https://github.com/hashicorp/terraform-provider-aws/issues/23228

    status = "Enabled"
    id     = "${local.prefix}-rotation"
    # delete if objects are older than noncurrent_days and older than n-newer_noncurrent_versions
    noncurrent_version_expiration {
      noncurrent_days           = 30
      newer_noncurrent_versions = 2
    }
    # move to standard IA if older than 30 days for all noncurrent versions
    noncurrent_version_transition {
      storage_class   = "STANDARD_IA"
      noncurrent_days = 30
    }
  }

  # Stops resource churn from above workaround - https://github.com/hashicorp/terraform-provider-aws/issues/23228
  lifecycle {
    ignore_changes = [rule[0].filter]
  }
}

[anGie44]

Hi @kjschiroo @kjschiroo , a couple more questions for verification.

(1) is the source S3 bucket relatively new or has it existed for a while before moving to v4.x of the provider?

(2) do you mind providing a snippet of what the AWS CLI returns with aws s3api get-bucket-lifecycle-configuration --bucket <your-bucket-name> ? I'm essentially curious how Filter and Prefix are getting returned and if the provider resource is prepared to read that value back into terraform state. I've come to notice that bucket lifecycle configurations created with either no prefix or a prefix of "" get returned as the following with the AWS CLI:

 "Filter": {
                "Prefix": ""
            },

thanks in advance!

[kjschiroo]

Hi @anGie44

(1) Both actually. We initially discovered it with buckets that had existed for a while but recreated a minimal example that had a brand new bucket. Here is that minimal example for reference:

terraform {
  required_version = ">= 1.1"
  # plus the backend s3 stuff
}

provider "aws" {
  region = "us-west-2"
}

resource "aws_s3_bucket" "b" {
  bucket = "a-rogue-bucket-appears"
}

resource "aws_s3_bucket_lifecycle_configuration" "bucket_config" {
  bucket = aws_s3_bucket.b.bucket

  rule {
    id     = "expiring"
    status = "Enabled"

    expiration {
      days = 60
    }

    filter {
      prefix = "stuff_that_ages/"
    }
  }

  rule {
    id     = "transitioning"
    status = "Enabled"

    noncurrent_version_transition {
      noncurrent_days = 30
      storage_class   = "GLACIER"
    }

    filter {
      # Including or excluding prefix doesn't appear to make a difference
      prefix = ""
    }
  }
}

(2) Here is what I get back:

> aws s3api get-bucket-lifecycle-configuration --bucket a-rogue-bucket-appears
{
    "Rules": [
        {
            "Expiration": {
                "Days": 60
            },
            "ID": "expiring",
            "Filter": {
                "Prefix": "stuff_that_ages/"
            },
            "Status": "Enabled"
        },
        {
            "ID": "transitioning",
            "Filter": {},
            "Status": "Enabled",
            "NoncurrentVersionTransitions": [
                {
                    "NoncurrentDays": 30,
                    "StorageClass": "GLACIER"
                }
            ]
        }
    ]
}

Does that help?

[anGie44]

Fantastic, thank you @kjschiroo exactly what i needed!

So from what I'm seeing, I think we need to make sure that terraform configs map to those JSON values as close as possible so that terraform plan doesn't suggest more changes.

with what you've provided in (2), the second rule filter should be filter {} in your terraform config as you experimented with earlier (#23228 (comment)).

While v4.1 shows the non-empty plan, I believe v4.0 will work (again using filter {} where needed in your aws_s3_bucket_lifecycle_configuration) unless that option has already been exhausted @kjschiroo ?

[kjschiroo]

@anGie44 I can confirm you are correct. Pinning down to v4.0 makes it show an empty plan.

[reelacmnaes]

I think this needs reopened. I can see the docs were updated to say that if no filter is specified it'll default to prefix = "", but should we be defaulting to a deprecated parameter? It feels a bit clunky to have to add an empty filter {} block to provision a rule for all files that leverages newer features.

resource "aws_s3_bucket_lifecycle_configuration" "this" {
  bucket = aws_s3_bucket.this.bucket 
  rule {
    filter {} # Workaround for https://github.com/hashicorp/terraform-provider-aws/issues/23228
    status = "Enabled"
    id     = "${local.prefix}-rotation"
    # delete if objects are older than noncurrent_days and older than n-newer_noncurrent_versions
    noncurrent_version_expiration {
      noncurrent_days           = 30
      newer_noncurrent_versions = 2
    }
    # move to standard IA if older than 30 days for all noncurrent versions
    noncurrent_version_transition {
      storage_class   = "STANDARD_IA"
      noncurrent_days = 30
    }
  }
}

If I take the empty filter {} block away with 4.2 provider, I still get the same error at the top of the issue.

[anGie44]

Hi @reelacmnaes, thank you for providing feedback. So per the AWS S3 API documentation available here and as seen in the snippet that follows

The Filter is used to identify objects that a Lifecycle Rule applies to. A Filter must have exactly one of Prefix, Tag, or And specified. Filter is required if the LifecycleRule does not contain a Prefix element.

the filter block is indeed required to leverage new features when prefix is not specified. Since the upstream API (and AWS Go SDK) also defaults to the deprecated parameter, the provider resource follows in suit to ensure a proper translation layer between the API and terraform.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.