Error putting S3 ACL: NoSuchBucket: The specified bucket does not exist

[hashibot]

This issue was originally opened by @abguy as hashicorp/terraform#10121. It was migrated here as part of the provider split. The original body of the issue is below.

---

My test deployment periodically failed with Error putting S3 ACL: NoSuchBucket: The specified bucket does not exist message. It looks like there is an issue with dependencies.
I actually had to add an additional attempt to my deployment script, but it fails sometimes anyway.

Some parts of my deployment script with workaround:

PLAN="/tmp/deployment.tfplan"
ARGS="-var build_number=237 -var branch=dc2ff0f3e435da28181c79d6fb556701f47f2e7c -var-file=mode/migration-demo.tfvars -out=$PLAN"
#...
    echo "Recreate S3 buckets..."
    if /opt/terraform/terraform show | grep -q "aws_s3_bucket.private"; then
        /opt/terraform/terraform taint aws_s3_bucket.private
    fi
    if /opt/terraform/terraform show | grep -q "aws_s3_bucket.public"; then
        /opt/terraform/terraform taint aws_s3_bucket.public
    fi
    /opt/terraform/terraform plan $ARGS
    /opt/terraform/terraform apply $PLAN || (sleep 5 && /opt/terraform/terraform apply $PLAN)

Despite the workaround it fails sometimes. I see this failure several times per a week. Let me know if you need additional details and I will help to reproduce this issue.

The part of my configuration is main.tf
My Jenkins build console output with debug enabled console.log

ubuntu@bastion:~$ terraform --version
Terraform v0.7.10

[Stephan1984]

I see this issue when deleting an bucket, then creating one again with the same name:
NoSuchBucket: The specified bucket does not exist

When I ran apply again it will modify the bucket:
versioning.0.enabled: "false" => "true"

Terraform v0.9.11

[easyawslearn]

facing the same issue in latest version Terraform v0.11.3

Error inspecting states in the "s3" backend:
NoSuchBucket: The specified bucket does not exist
status code: 404, request id: 93FB5B7D694D41A2, host id: s8XW36imcXCkCLGlr9iu7tMcM0zq9AB0ZCrgQSJLwEiCyYFUjr0Uc/m1m8f8nOFcqcOwpojVGHQ=

[ibayer]

I'm having the same issue (I can see the bucket on S3).

terraform -v
Terraform v0.11.3

terraform {
 backend "s3" {
    encrypt = true
    bucket = "agile-remote-state"
    dynamodb_table = "terraform-state-lock-dynamo"
    region = "eu-central-1"
    key = "prod/terrafrom.tfstate"
 }
}

Error inspecting states in the "s3" backend:
    NoSuchBucket: The specified bucket does not exist
	status code: 404, request id: 93012D18C781BCB9, host id: HVst07vK8CGvO+uphsfyJE6QiYc/8UjweWjGptWxTG7tdRUBksY7xZRhfTuSE5tYww8/qtnNAIs=

[ibayer]

I does work when I use the backend definition in the same folder where I create the bucket.
Could it be that the bucket name alone is not enough and the state file from the bucket creation is needed too to find the bucket?

[SanCoder-Q]

I was facing the same problem when trying to deploy an S3 bucket with its bucket policy together. And the bucket name of the bucket policy is not a reference from the bucket resource but a variable shared by those two resources. Then terraform cannot figure out the dependency between those two resources.

After explicit the dependency by using depends_on, everything works well.

An alternative is to use the resource reference to get the bucket name so that terraform can have a clue to figure out the dependencies.

[ctippur]

Having same issue with 0.11.4.
Any updates on this please. I am able to pull the s3 bucket attributes using aws cli.

[ctippur]

The error I see is

2018/03/20 11:33:14 [DEBUG] [aws-sdk-go] DEBUG: Validate Response s3/ListObjects failed, not retrying, error NoSuchBucket: The specified bucket does not exist

[yamaszone]

I am seeing the same error with 0.11.7. Any update on this? Workaround mentioned here worked for me but ideally it should be tolerant/transparent.

[AJVA2580]

Hello,

I have the same problem and I using Terraform v0.11.8.

[vcardenas]

Same here
Error putting S3 lifecycle: NoSuchBucket: The specified bucket does not exist

Actually, it is quite easy to reproduce. My guess is that it has to do with AWS eventual consistency when you delete and create a bucket with the same name as before, even if it was not deleted recently.

Consider this terraform

terraform = {
  required_version = ">=0.11.2" //tried with up to 0.11.8
}

provider "aws" {
  version = "~> 2.0"    // previous versions have been tried with same results
  max_retries = 10      // naive attempt to make it work
  /*** complete config with your env ***/
}

resource "aws_s3_bucket" "mybucket" {
    bucket = "vcardenas-bucket"
    force_destroy = true
    versioning {
        enabled = true
    }
    lifecycle_rule {
        id = "tests-expire"
        enabled = true
        prefix = "myprefix/"
        expiration {
            days = 3
        }
        noncurrent_version_expiration {
            days = 3
        }
    }
}

Now init, apply, destroy, clean? and repeat... I experience the issue with a 40% of occurrence.
For convenience I perform the test with this bash script:

#!/bin/bash

for run in {1..10} ; do
  terraform init
  terraform apply -auto-approve
  terraform destroy -auto-approve

  # Workaround mentioned in other comments. Does not work.
  rm -rf .terraform/
done

Even inserting periods of sleep with seconds up to a minute between iterations, results are the same.
I have not tried more than 1 minute, I don't think the solution should be "wait X before creating your bucket"

I am going to try to look at the source code to see if I can figure out something about this issue.

UPDATE: Follow up in #7803

[simonclausen]

I'm seeing something akin to this: Error putting S3 notification configuration: NoSuchBucket: The specified bucket does not exist

This when creating a bucket with aws_s3_bucket and then creating notification settings with aws_s3_bucket_notification.

Simplified config:

resource "aws_s3_bucket" "mybucket" {
  bucket = "mybucket"
  acl = "private"
  tags = {
    terraform   = true
    environment = var.environment
  }
}

resource "aws_s3_bucket_notification" "mybucket_notification" {
  bucket = aws_s3_bucket.mybucket.id
  queue {
    queue_arn     = aws_sqs_queue.myqueue.arn
    events        = ["s3:ObjectCreated:*"]
    filter_suffix = ".log.gz"
  }
}

The reference from aws_s3_bucket_notification to aws_s3_bucket should enable terraform to recognize the dependency and create things in proper order, which it also does most of the time.

The bucket name has not been used before; this is running in a CI pipeline with unique IDs in the bucket name. But I do think it's a matter of eventual consistency in S3 config, as mentioned in #7803 and #10068.

It seems plausible that the fix for the OP's issue with ACLs and these notification settings be the same as the two linked issues, adding retry.

Should I create a new issue for handling of notification related consistency issues? Or can both be handled in this issue?

[simonclausen]

I'm seeing something akin to this again, but with a different error message this time: Error: Error putting S3 notification configuration: OperationAborted: A conflicting conditional operation is currently in progress against this resource. Please try again.

Same setup as previous example, so attempting to create notification config immediately after bucket creation.

[github-actions]

Marking this issue as stale due to inactivity. This helps our maintainers find and focus on the active issues. If this issue receives no comments in the next 30 days it will automatically be closed. Maintainers can also remove the stale label.

If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.