Error creating S3 bucket with tags - error getting S3 bucket tags: NoSuchBucket: The specified bucket does not exist

[kxavier-ims]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

Terraform v0.11.7

Affected Resource(s)

• aws_s3_bucket

Terraform Configuration Files

Here is a simplified version of the configuration file:

terraform = {
  required_version = ">=0.11.7"
}

provider "aws" {
  version = "2.27.0"
}

resource "random_uuid" "random" {}

resource "aws_s3_bucket" "kxavier_poc_bucket" {
  bucket = "kxavier-poc-${random_uuid.random.result}"
  acl    = "private"

  tags = {
    Name = "My Name"
    TagA = "My Tag A"
    TagB = "My Tag B"
    TagC = "My Tag C"
    TagD = "My Tag D"
  }

  lifecycle_rule {
    id      = "kxavier_poc_bucket_retention"
    enabled = true

    expiration {
      days = 2556
    }
  }

  server_side_encryption_configuration {
    rule {
      apply_server_side_encryption_by_default {
        kms_master_key_id = "some valid kms arn"
        sse_algorithm     = "aws:kms"
      }
    }
  }
}

Expected Behavior

You have an S3 bucket after terraform apply

Actual Behavior

In our builds we have randomly the error message "error getting S3 bucket tags: NoSuchBucket: The specified bucket does not exist", and the build breaks. It seems to be some race condition.

If we look at CloudTrail we can see that we have the CreateBucket event, but not the PutBucketTagging event when the build fails.

Steps to Reproduce

Here is a script to try to reproduce it.

#!/bin/bash

for run in {1..10} ; do
  terraform init
  terraform apply -auto-approve
  terraform destroy -auto-approve
  rm -rf .terraform/
done

[mscansian]

I'm also having this issue in 0.12.10.

Workaround

A little annoying but easy to fix

• Remove the tags and encryption config from the bucket resource
• terraform apply (creates the bucket)
• Restore tags and encryption in the template
• terraform apply (update config)

[GFlores94]

Running into this issue with 0.12.2.
Happens at random, difficult to reproduce.

@mscansian would work however we can't do this on Terraform Enterprise

[bpiper]

Not sure the Terraform version here is relevant (0.10.8 for me), but we spin up a test stack several times daily, and according to our logs (which go back years), the first occurrence of this error (which by the way has no correspondence with any changes to our resource definitions) was on September 23rd, on version 2.29.0 of the provider. We've also noticed it happening much more frequently just in the past week.

That's not to say that changes in the provider precipitated this issue though, as it could easily be due to a change on Amazon's end (i.e. eventual consistency becoming rather more 'eventual'). Nevertheless, I do wonder if there should be some retry logic around getting bucket tags.

[jurajseffer]

We have started seeing the same thing roughly about the same time using boto3 call such as this:

s3_client.create_bucket(
  Bucket=bucket_name,
  ACL="private",
)
waiter = s3_client.get_waiter('bucket_exists')
waiter.wait(Bucket=bucket_name)

s3_client.put_bucket_encryption(
    Bucket=bucket_name,
    ServerSideEncryptionConfiguration={
        "Rules": [
            {
                "ApplyServerSideEncryptionByDefault": {
                    "SSEAlgorithm": "aws:kms"
                }
            }
        ]
    },
)
s3_client.put_bucket_tagging(
    Bucket=bucket_name,
    Tagging={
        "TagSet": [
            {"Key": "Name", "Value": bucket_name},
        ]
    },
)

It would occasionally fail to put the tagging on because encryption operation makes the bucket "not exist". Perhaps waiters are needed around every single operation on the bucket now.

[wking]

We hit this too. CloudTrail logs of one such case here, which show:

• 2019-10-18T04:49:45Z, CreateBucket success
• 2019-10-18T04:58:24Z, DeleteBucket success

our failed s3/GetObjectTagging came back with Date: Fri, 18 Oct 2019 04:49:46 GMT, but its request ID does not show up in our CloudTrail logs. I suspect AWS has some low-probability internal flake, and have filed a ticket to follow up with them. Will report back here if I get anything interesting back. A waiter to retry these in case they are eventual-consistency issues would probably help too, even if it's just protecting us from occasional AWS-internal hiccups.

[tiny-dancer]

@wking any luck with the AWS ticket?

[wking]

Oops, yeah. Bruce B on Fri Oct 18 2019 07:06:57 GMT-0700:

Thank you for the detailed information provided, from this I was able to see the initial Create Bucket operation occurred two seconds before the failed request (REST.GET.OBJECT_TAGGING). The reason is related to S3's eventual consistency as you thought.

The Bucket operations are atomic in nature and if a request is accepted via an S3 service host that has not received the Bucket Metadata, the 404 - NoSuchBucket will be returned [1].
This specific type of request was not logged in CloudTrail as the resource is at the object level and would only be captured as a data event [2].

It is possible to enable these data events at the account level allowing them to be captured across resources before they are created, however they will generate a significant amounts of logging as every API action would be recorded.
...
Reference:
[1] https://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#ConsistencyModel
[2] https://docs.aws.amazon.com/awscloudtrail/latest/userguide/logging-management-and-data-events-with-cloudtrail.html#logging-data-events

So seems like there's no workaround besides queueing a retry, and dying if that fails for longer than we're willing to wait for AWS to resolve its eventual consistency.

[bflad]

We have also seen this pop up randomly in our daily acceptance testing for the project, e.g.

--- FAIL: TestAccAWSS3Bucket_tagsWithSystemTags (11.28s)
    testing.go:615: Step 0 error: errors during apply:
        
        Error: error getting S3 bucket tags: NoSuchBucket: The specified bucket does not exist

--- FAIL: TestAccAWSCodeBuildProject_Artifacts_ArtifactIdentifier (7.39s)
    testing.go:615: Step 0 error: errors during apply:
        
        Error: error getting S3 bucket tags: NoSuchBucket: The specified bucket does not exist

Submitted a fix here: #10863

[ghost]

This has been released in version 2.36.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template for triage. Thanks!

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!