Initialization s3 Backend no working

[ClaireBellivier]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

Terraform version: v0.11.10

• provider.aws v1.46.0

Affected Resource(s)

Initialization s3 backend.

Terraform Configuration Files

variables.tf

######## AWS provider vars ########
# AWS Access Key 
variable "aws_access_key" {
  description = "AWS API Access Key"
}

# AWS Secret Key 
variable "aws_secret_key" {
  description = "AWS API Secret Key"
}

# AWS Region
variable "aws_region" {
  description = "AWS Region to deploy to"
}

######## Project vars ########
# Project name
variable "project_name" {
  description = "Name of the project"
}

# Project Owner
variable "project_owner" {
  description = "Owner of the project"
}

provider.tf

# Configure the AWS Provider
provider "aws" {
  access_key = "${var.aws_access_key}"
  secret_key = "${var.aws_secret_key}"
  region     = "${var.aws_region}"
}

backend.tf

# Decalaration of the terraform backend
terraform {
  backend "s3" {
    bucket         = "remote-state-test01-me-us-east-1"
    encrypt        = true
    key            = "remote-state-test01-me-us-east-1/backend/terraform.tfstate"
    dynamodb_table = "remote-state-test01-me-us-east-1"
    region         = "us-east-1"
  }
}

# Get current user canonical ID
data "aws_canonical_user_id" "current" {}

# S3 bucket policy template file creation
data "template_file" "tf_state_bucket_policy_template" {
  template = "${file("${path.module}/policy/s3.tpl")}"

  vars {
    s3_bucket_name = "${aws_s3_bucket.tf_state_bucket.id}"
    canonical_user_id = "${data.aws_canonical_user_id.current.id}"
  }
}

# S3 buccket policy creation
resource "aws_s3_bucket_policy" "tf_state_bucket_policy" {
  bucket = "${aws_s3_bucket.tf_state_bucket.id}"

  policy = "${data.template_file.tf_state_bucket_policy_template.rendered}"
}

# AWS S3 bucket creation
resource "aws_s3_bucket" "tf_state_bucket" {
  bucket  = "${var.project_name}-${var.project_owner}-${var.aws_region}"
  acl     = "private"

  tags {
    Name  = "${var.project_name}"
    Owner = "${var.project_owner}"
  }

  server_side_encryption_configuration {
    rule {
      apply_server_side_encryption_by_default {
        sse_algorithm = "AES256"
      }
    }
  }
  
  versioning {
    enabled = true
  }

  lifecycle {
    prevent_destroy = true
  }
}

# DynamoDB table creation
resource "aws_dynamodb_table" "tf_state_dynamodb" {
  name           = "${var.project_name}-${var.project_owner}-${var.aws_region}"
  hash_key       = "LockID"
  read_capacity  = 20
  write_capacity = 20
 
  attribute {
    name         = "LockID"
    type         = "S"
  }
 
  tags {
    Name         = "${var.project_name}"
    Owner        = "${var.project_owner}"
  }

  lifecycle {
    prevent_destroy = true
  }
}

S3 policy s3.tpl

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {"CanonicalUser":"${canonical_user_id}"},
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::${s3_bucket_name}"
    },
    {
      "Effect": "Allow",
      "Principal": {"CanonicalUser":"${canonical_user_id}"},
      "Action": ["s3:GetObject", "s3:PutObject"],
      "Resource": "arn:aws:s3:::${s3_bucket_name}/*"
    }
  ]
}

Debug Output

Initializing the backend...

Error configuring the backend "s3": No valid credential sources found for AWS Provider.
        Please see https://terraform.io/docs/providers/aws/index.html for more information on
        providing credentials for the AWS Provider

Please update the configuration in your Terraform files to fix this error
then run this command again.

Expected Behavior

Use of the bucket s3 as a backend with the terraform.tfstate stored in the s3 bucket.

Actual Behavior

Backend cannot be initialized.

Steps to Reproduce

1. Comment backend info in the backend.tf
2. terraform init
3. terraform apply
4. After successful apply uncomment backend info in the backend.tf
5. terraform init

Important Factoids

Run on a new account.

References

• Intructions followed to create the backend

[nathanielks]

I've experienced the same. I found if I remove the terraform configuration block, it can provision the providers. I tested this in versions 0.11.10, 0.11.7, and 0.11.3. None of those versions can init when a terraform configuration block with a S3 backend is specified. Using a local backend works fine.

[nathanielks]

Removing the additional encrypt and dynamodb_table settings have no effect, either.

[ClaireBellivier]

Thanks @nathanielks well spotted it's true
By the way, I used the "profile" variable so after export AWS_PROFILE that fixed it, not sure it's that but I guess, seems quite logical.

[nathanielks]

@ClaireBellivier it occurred to me yesterday that this is expected behavior! The backend authenticates separately from the provider, so if we don't have environment variables such as AWS_PROFILE or AWS_ACCESS_KEY_ID, then the backend can't authenticate! Ruh roh!

[nathanielks]

To add to this, backends cannot contain interpolation, so either the environment variables are present or we have to hardcode the parameters into the backend definition.

[ClaireBellivier]

Actually, that's what I discovered :) and yes of course I did.

[nathanielks]

If we're good here, it'd be good to close the issue. Their backlog is already quite large 😃

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!