Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Terraform 1.6.x - SignatureDoesNotMatch Error - S3 Backend (GetObject) #34136

Open
lambbuster opened this issue Oct 25, 2023 · 10 comments
Open

Terraform 1.6.x - SignatureDoesNotMatch Error - S3 Backend (GetObject) #34136

lambbuster opened this issue Oct 25, 2023 · 10 comments
Assignees
@gdavison
Labels
backend/s3 bug new new issue not yet triaged

Comments

@lambbuster
Copy link

lambbuster commented Oct 25, 2023 •
edited
Loading

Terraform Version

Terraform v1.6.0

on darwin_arm64

+ provider registry.terraform.io/hashicorp/aws v5.21.0

+ provider registry.terraform.io/hashicorp/external v2.3.1

+ provider registry.terraform.io/hashicorp/time v0.9.1

+ provider registry.terraform.io/venafi/venafi v0.16.0

Terraform Configuration Files

terraform {
  backend "s3" {
    skip_region_validation = true
    encrypt                = true
    dynamodb_table         = "tfstate"
    bucket                 = "xxx-eu-west-1-terraform"
    region                 = "eu-west-1"
    key                    = "terraform.tfstate"
    profile                = "saml"
    #    access_key = "xxx"
    #    secret_key = "xxx"
    #    token = "xxx"

    #    assume_role = {
    #      role_arn = "arn:aws:iam::[REDACTED]:role/jenkins"
    #      external_id = "opbk"
    #    }
  }
}

terraform {
  required_version = "~> 1.6.0"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
    venafi = {
      source  = "venafi/venafi"
      version = "0.16.0"
    }
    external = {
      source  = "hashicorp/external"
      version = "~> 2.2"
    }
  }
}

Debug Output

Initializing the backend...
2023-10-19T13:56:11.140+0100 [DEBUG] backend-s3.aws-base: Resolving credentials provider: tf_backend.operation=Configure tf_backend.req_id=e36c1bc5-a9e8-a929-6236-ec2628af7de8 tf_backend.s3.bucket=ob-[REDACTED]-eu-west-1-terraform tf_backend.s3.path=terraform.tfstate
2023-10-19T13:56:11.140+0100 [DEBUG] backend-s3.aws-base: Using profile: tf_backend.operation=Configure tf_backend.req_id=e36c1bc5-a9e8-a929-6236-ec2628af7de8 tf_backend.s3.bucket=ob-[REDACTED]-eu-west-1-terraform tf_backend.s3.path=terraform.tfstate tf_aws.profile=saml tf_aws.profile.source=provider
2023-10-19T13:56:11.140+0100 [DEBUG] backend-s3.aws-base: Loading profile: tf_backend.operation=Configure tf_backend.req_id=e36c1bc5-a9e8-a929-6236-ec2628af7de8 tf_backend.s3.bucket=ob-[REDACTED]-eu-west-1-terraform tf_backend.s3.path=terraform.tfstate tf_aws.profile=saml
2023-10-19T13:56:11.140+0100 [DEBUG] backend-s3.aws-base: Setting profile: tf_backend.operation=Configure tf_backend.req_id=e36c1bc5-a9e8-a929-6236-ec2628af7de8 tf_backend.s3.bucket=ob-[REDACTED]-eu-west-1-terraform tf_backend.s3.path=terraform.tfstate tf_aws.profile=saml tf_aws.profile.source=provider
2023-10-19T13:56:11.141+0100 [DEBUG] backend-s3.aws-base: Loading configuration: tf_backend.operation=Configure tf_backend.req_id=e36c1bc5-a9e8-a929-6236-ec2628af7de8 tf_backend.s3.bucket=ob-[REDACTED]-eu-west-1-terraform tf_backend.s3.path=terraform.tfstate
2023-10-19T13:56:11.143+0100 [DEBUG] backend-s3.aws-base: Retrieving credentials: tf_backend.operation=Configure tf_backend.req_id=e36c1bc5-a9e8-a929-6236-ec2628af7de8 tf_backend.s3.bucket=ob-[REDACTED]-eu-west-1-terraform tf_backend.s3.path=terraform.tfstate
2023-10-19T13:56:11.143+0100 [INFO]  backend-s3.aws-base: Retrieved credentials: tf_backend.operation=Configure tf_backend.req_id=e36c1bc5-a9e8-a929-6236-ec2628af7de8 tf_backend.s3.bucket=ob-[REDACTED]-eu-west-1-terraform tf_backend.s3.path=terraform.tfstate tf_aws.credentials_source="SharedConfigCredentials: /Users/[REDACTED]/.aws/credentials"
2023-10-19T13:56:11.143+0100 [DEBUG] backend-s3.aws-base: Loading configuration: tf_backend.operation=Configure tf_backend.req_id=e36c1bc5-a9e8-a929-6236-ec2628af7de8 tf_backend.s3.bucket=ob-[REDACTED]-eu-west-1-terraform tf_backend.s3.path=terraform.tfstate
2023-10-19T13:56:11.146+0100 [DEBUG] backend-s3.aws-base: Retrieving caller identity from STS: tf_backend.operation=Configure tf_backend.req_id=e36c1bc5-a9e8-a929-6236-ec2628af7de8 tf_backend.s3.bucket=ob-[REDACTED]-eu-west-1-terraform tf_backend.s3.path=terraform.tfstate
2023-10-19T13:56:11.147+0100 [DEBUG] backend-s3.aws-base: HTTP Request Sent: aws.operation=GetCallerIdentity aws.region=eu-west-1 aws.sdk=aws-sdk-go-v2 aws.service=STS tf_backend.operation=Configure tf_backend.req_id=e36c1bc5-a9e8-a929-6236-ec2628af7de8 tf_backend.s3.bucket=ob-[REDACTED]-eu-west-1-terraform tf_backend.s3.path=terraform.tfstate http.method=POST http.url=https://sts.eu-west-1.amazonaws.com/ http.request_content_length=43 http.request.header.x_amz_security_token="*****" http.request.header.amz_sdk_request="attempt=1; max=5" http.request.header.x_amz_date=20231019T125611Z http.request.header.content_type=application/x-www-form-urlencoded http.request.header.amz_sdk_invocation_id=4028fcfd-14f0-4f09-8055-331c1785a2b2 net.peer.name=sts.eu-west-1.amazonaws.com http.user_agent="APN/1.0 HashiCorp/1.0 Terraform/1.6.0 (+https://www.terraform.io) aws-sdk-go-v2/1.21.0 os/macos lang/go#1.21.1 md/GOOS#darwin md/GOARCH#arm64 api/sts#1.21.5" http.request.header.authorization="AWS4-HMAC-SHA256 Credential=ASIA************VBYM/20231019/eu-west-1/sts/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-date;x-amz-security-token, Signature=*****"
  http.request.body=
  | Action=GetCallerIdentity&Version=2011-06-15
  
2023-10-19T13:56:11.607+0100 [DEBUG] backend-s3.aws-base: HTTP Response Received: aws.operation=GetCallerIdentity aws.region=eu-west-1 aws.sdk=aws-sdk-go-v2 aws.service=STS tf_backend.operation=Configure tf_backend.req_id=e36c1bc5-a9e8-a929-6236-ec2628af7de8 tf_backend.s3.bucket=ob-[REDACTED]-eu-west-1-terraform tf_backend.s3.path=terraform.tfstate http.response.header.content_type=text/xml http.response.header.date="Thu, 19 Oct 2023 12:56:11 GMT" http.response.header.proxy_connection=Keep-Alive http.response.header.connection=Keep-Alive
  http.response.body=
  | <GetCallerIdentityResponse xmlns="https://sts.amazonaws.com/doc/2011-06-15/">
  |   <GetCallerIdentityResult>
  |     <Arn>arn:aws:sts::[REDACTED]:assumed-role/[REDACTED]/[REDACTED]</Arn>
  |     <UserId>AROA*************PSOA:[REDACTED]</UserId>
  |     <Account>[REDACTED]</Account>
  |   </GetCallerIdentityResult>
  |   <ResponseMetadata>
  |     <RequestId>266e2059-c950-4e0c-b46b-b55ade6bef6f</RequestId>
  |   </ResponseMetadata>
  | </GetCallerIdentityResponse>
   http.duration=460 http.response.header.x_amzn_requestid=266e2059-c950-4e0c-b46b-b55ade6bef6f http.response.header.cache_control=proxy-revalidate http.status_code=200 http.response_content_length=439
2023-10-19T13:56:11.608+0100 [INFO]  backend-s3.aws-base: Retrieved caller identity from STS: tf_backend.operation=Configure tf_backend.req_id=e36c1bc5-a9e8-a929-6236-ec2628af7de8 tf_backend.s3.bucket=ob-[REDACTED]-eu-west-1-terraform tf_backend.s3.path=terraform.tfstate
2023-10-19T13:56:11.608+0100 [DEBUG] backend-s3.aws-base: Retrieving caller identity from STS: tf_backend.operation=Configure tf_backend.req_id=e36c1bc5-a9e8-a929-6236-ec2628af7de8 tf_backend.s3.bucket=ob-[REDACTED]-eu-west-1-terraform tf_backend.s3.path=terraform.tfstate
2023-10-19T13:56:11.608+0100 [DEBUG] backend-s3.aws-base: HTTP Request Sent: aws.operation=GetCallerIdentity aws.region=eu-west-1 aws.sdk=aws-sdk-go-v2 aws.service=STS tf_backend.operation=Configure tf_backend.req_id=e36c1bc5-a9e8-a929-6236-ec2628af7de8 tf_backend.s3.bucket=ob-[REDACTED]-eu-west-1-terraform tf_backend.s3.path=terraform.tfstate http.method=POST net.peer.name=sts.eu-west-1.amazonaws.com http.request.header.content_type=application/x-www-form-urlencoded http.request.header.amz_sdk_request="attempt=1; max=5" http.request.header.x_amz_date=20231019T125611Z
  http.request.body=
  | Action=GetCallerIdentity&Version=2011-06-15
   http.url=https://sts.eu-west-1.amazonaws.com/ http.user_agent="APN/1.0 HashiCorp/1.0 Terraform/1.6.0 (+https://www.terraform.io) aws-sdk-go-v2/1.21.0 os/macos lang/go#1.21.1 md/GOOS#darwin md/GOARCH#arm64 api/sts#1.21.5" http.request_content_length=43 http.request.header.authorization="AWS4-HMAC-SHA256 Credential=ASIA************VBYM/20231019/eu-west-1/sts/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-date;x-amz-security-token, Signature=*****" http.request.header.x_amz_security_token="*****" http.request.header.amz_sdk_invocation_id=97a6a188-0dec-4d50-9d10-319b6127d938
2023-10-19T13:56:11.671+0100 [DEBUG] backend-s3.aws-base: HTTP Response Received: aws.operation=GetCallerIdentity aws.region=eu-west-1 aws.sdk=aws-sdk-go-v2 aws.service=STS tf_backend.operation=Configure tf_backend.req_id=e36c1bc5-a9e8-a929-6236-ec2628af7de8 tf_backend.s3.bucket=ob-[REDACTED]-eu-west-1-terraform tf_backend.s3.path=terraform.tfstate http.response.header.proxy_connection=Keep-Alive http.response.header.connection=Keep-Alive http.response.header.x_amzn_requestid=5095e69f-2def-4a86-b0aa-622643fecbe1 http.response.header.date="Thu, 19 Oct 2023 12:56:11 GMT"
  http.response.body=
  | <GetCallerIdentityResponse xmlns="https://sts.amazonaws.com/doc/2011-06-15/">
  |   <GetCallerIdentityResult>
  |     <Arn>arn:aws:sts::[REDACTED]:assumed-role/[REDACTED]/[REDACTED]</Arn>
  |     <UserId>AROA*************PSOA:[REDACTED]</UserId>
  |     <Account>[REDACTED]</Account>
  |   </GetCallerIdentityResult>
  |   <ResponseMetadata>
  |     <RequestId>5095e69f-2def-4a86-b0aa-622643fecbe1</RequestId>
  |   </ResponseMetadata>
  | </GetCallerIdentityResponse>
   http.duration=52 http.status_code=200 http.response_content_length=439 http.response.header.cache_control=proxy-revalidate http.response.header.content_type=text/xml
...
2023-10-19T13:58:04.762+0100 [INFO]  backend/local: starting Apply operation
2023-10-19T13:58:04.767+0100 [DEBUG] backend-s3: HTTP Request Sent: aws.operation=ListObjectsV2 aws.region=eu-west-1 aws.sdk=aws-sdk-go-v2 aws.service=S3 tf_backend.operation=Workspaces tf_backend.req_id=efd44fde-c0a5-a6f7-8d86-fac47232aacc tf_backend.s3.bucket=ob-[REDACTED]-eu-west-1-terraform tf_backend.workspace-prefix=env:/ http.request.header.x_amz_content_sha256=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 http.method=GET http.url="https://ob-[REDACTED]-eu-west-1-terraform.s3.eu-west-1.amazonaws.com/?list-type=2&max-keys=1000&prefix=env%3A%2F" net.peer.name=ob-[REDACTED]-eu-west-1-terraform.s3.eu-west-1.amazonaws.com http.user_agent="APN/1.0 HashiCorp/1.0 Terraform/1.6.0 (+https://www.terraform.io) aws-sdk-go-v2/1.21.0 os/macos lang/go#1.21.1 md/GOOS#darwin md/GOARCH#arm64 api/s3#1.38.5" http.request.header.authorization="AWS4-HMAC-SHA256 Credential=ASIA************VBYM/20231019/eu-west-1/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date;x-amz-security-token, Signature=*****" http.request.header.accept_encoding=identity http.request.header.x_amz_security_token="*****" http.request.header.amz_sdk_request="attempt=1; max=5" http.request.header.x_amz_date=20231019T125804Z http.request.header.amz_sdk_invocation_id=cf1a5ee3-e8ea-4faa-8eaf-ba9547215401 http.request.body=""
2023-10-19T13:58:05.132+0100 [DEBUG] backend-s3: HTTP Response Received: aws.operation=ListObjectsV2 aws.region=eu-west-1 aws.sdk=aws-sdk-go-v2 aws.service=S3 tf_backend.operation=Workspaces tf_backend.req_id=efd44fde-c0a5-a6f7-8d86-fac47232aacc tf_backend.s3.bucket=ob-[REDACTED]-eu-west-1-terraform tf_backend.workspace-prefix=env:/ http.duration=364 http.response.header.x_amz_bucket_region=eu-west-1 http.status_code=200 http.response.header.x_amz_id_2=sZjjoTx6Z2iAMbvhQyIkFxjA9NQkCatocObVZ4ZNi5Cuv/OCC25paf5QA4QVz3Qu6lnTs1jY9fo= http.response.header.age=0 http.response.header.x_amz_request_id=XKPW9DHXZDS3MSFZ http.response.header.cache_control=proxy-revalidate http.response.header.server=AmazonS3 http.response.header.proxy_connection=Keep-Alive http.response.header.content_type=application/xml http.response.header.date="Thu, 19 Oct 2023 12:58:06 GMT" http.response.header.connection=Keep-Alive
  http.response.body=
  | <?xml version="1.0" encoding="UTF-8"?>
  | <ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>ob-[REDACTED]-eu-west-1-terraform</Name><Prefix>env:/</Prefix><KeyCount>0</KeyCount><MaxKeys>1000</MaxKeys><IsTruncated>false</IsTruncated></ListBucketResult>
  
2023-10-19T13:58:05.134+0100 [INFO]  backend-s3: Locking remote state: tf_backend.lock.id=55746556-14c1-c611-0b56-12e210a1d7b3 tf_backend.lock.info="" tf_backend.lock.operation=OperationTypeApply tf_backend.lock.path=ob-[REDACTED]-eu-west-1-terraform/terraform.tfstate tf_backend.lock.version=1.6.0 tf_backend.lock.who=[REDACTED] tf_backend.operation=Lock tf_backend.req_id=da082e9c-d5a6-aa83-6101-0cb213335053 tf_backend.s3.bucket=ob-[REDACTED]-eu-west-1-terraform tf_backend.s3.path=terraform.tfstate
2023-10-19T13:58:05.137+0100 [DEBUG] backend-s3: HTTP Request Sent: aws.operation=PutItem aws.region=eu-west-1 aws.sdk=aws-sdk-go-v2 aws.service=DynamoDB tf_backend.lock.id=55746556-14c1-c611-0b56-12e210a1d7b3 tf_backend.lock.info="" tf_backend.lock.operation=OperationTypeApply tf_backend.lock.path=ob-[REDACTED]-eu-west-1-terraform/terraform.tfstate tf_backend.lock.version=1.6.0 tf_backend.lock.who=[REDACTED] tf_backend.operation=Lock tf_backend.req_id=da082e9c-d5a6-aa83-6101-0cb213335053 tf_backend.s3.bucket=ob-[REDACTED]-eu-west-1-terraform tf_backend.s3.path=terraform.tfstate http.request.header.x_amz_date=20231019T125805Z http.request_content_length=473 http.request.header.accept_encoding=identity http.user_agent="APN/1.0 HashiCorp/1.0 Terraform/1.6.0 (+https://www.terraform.io) aws-sdk-go-v2/1.21.0 os/macos lang/go#1.21.1 md/GOOS#darwin md/GOARCH#arm64 api/dynamodb#1.21.1" http.request.header.x_amz_security_token="*****" http.request.header.amz_sdk_invocation_id=b6443168-801f-4e75-8ec4-b7ad8524fef2 http.method=POST http.url=https://dynamodb.eu-west-1.amazonaws.com/
  http.request.body=
  | {"ConditionExpression":"attribute_not_exists(LockID)","Item":{"LockID":{"S":"ob-[REDACTED]-eu-west-1-terraform/terraform.tfstate"},"Info":{"S":"{\"ID\":\"55746556-14c1-c611-0b56-12e210a1d7b3\",\"Operation\":\"OperationTypeApply\",\"Info\":\"\",\"Who\":\"[REDACTED]\",\"Version\":\"1.6.0\",\"Created\":\"2023-10-19T12:58:05.133785Z\",\"Path\":\"ob-[REDACTED]-eu-west-1-terraform/terraform.tfstate\"}"}},"TableName":"tfstate"}
   net.peer.name=dynamodb.eu-west-1.amazonaws.com http.request.header.authorization="AWS4-HMAC-SHA256 Credential=ASIA************VBYM/20231019/eu-west-1/dynamodb/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-date;x-amz-security-token;x-amz-target, Signature=*****" http.request.header.content_type=application/x-amz-json-1.0 http.request.header.x_amz_target=DynamoDB_20120810.PutItem http.request.header.amz_sdk_request="attempt=1; max=5"
Acquiring state lock. This may take a few moments...
2023-10-19T13:58:05.608+0100 [DEBUG] backend-s3: HTTP Response Received: aws.operation=PutItem aws.region=eu-west-1 aws.sdk=aws-sdk-go-v2 aws.service=DynamoDB tf_backend.lock.id=55746556-14c1-c611-0b56-12e210a1d7b3 tf_backend.lock.info="" tf_backend.lock.operation=OperationTypeApply tf_backend.lock.path=ob-[REDACTED]-eu-west-1-terraform/terraform.tfstate tf_backend.lock.version=1.6.0 tf_backend.lock.who=[REDACTED] tf_backend.operation=Lock tf_backend.req_id=da082e9c-d5a6-aa83-6101-0cb213335053 tf_backend.s3.bucket=ob-[REDACTED]-eu-west-1-terraform tf_backend.s3.path=terraform.tfstate http.duration=470 http.status_code=200 http.response.header.connection=Keep-Alive http.response.header.server=Server http.response.header.content_type=application/x-amz-json-1.0 http.response.header.x_amzn_requestid=AKE7BV97AC409M5LCIRHUPS0I3VV4KQNSO5AEMVJF66Q9ASUAAJG
  http.response.body=
  | {}
   http.response_content_length=2 http.response.header.proxy_connection=Keep-Alive http.response.header.x_amz_crc32=2745614147 http.response.header.date="Thu, 19 Oct 2023 12:58:05 GMT" http.response.header.cache_control=proxy-revalidate
2023-10-19T13:58:05.608+0100 [INFO]  backend-s3: Downloading remote state: tf_backend.operation=Get tf_backend.req_id=63c46ab0-c213-4f8d-1c9b-8b5a0ac6520b tf_backend.s3.bucket=ob-[REDACTED]-eu-west-1-terraform tf_backend.s3.path=terraform.tfstate
2023-10-19T13:58:05.608+0100 [DEBUG] backend-s3: HTTP Request Sent: aws.operation=HeadObject aws.region=eu-west-1 aws.sdk=aws-sdk-go-v2 aws.service=S3 tf_backend.operation=Get tf_backend.req_id=63c46ab0-c213-4f8d-1c9b-8b5a0ac6520b tf_backend.s3.bucket=ob-[REDACTED]-eu-west-1-terraform tf_backend.s3.path=terraform.tfstate http.request.header.amz_sdk_invocation_id=d2ab369f-c7d0-4820-b839-723030056884 http.request.body="" net.peer.name=ob-[REDACTED]-eu-west-1-terraform.s3.eu-west-1.amazonaws.com http.request.header.x_amz_security_token="*****" http.request.header.x_amz_content_sha256=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 http.request.header.accept_encoding=identity http.request.header.amz_sdk_request="attempt=1; max=5" http.request.header.x_amz_date=20231019T125805Z http.method=HEAD http.url=https://ob-[REDACTED]-eu-west-1-terraform.s3.eu-west-1.amazonaws.com/terraform.tfstate http.user_agent="APN/1.0 HashiCorp/1.0 Terraform/1.6.0 (+https://www.terraform.io) aws-sdk-go-v2/1.21.0 os/macos lang/go#1.21.1 md/GOOS#darwin md/GOARCH#arm64 api/s3#1.38.5" http.request.header.authorization="AWS4-HMAC-SHA256 Credential=ASIA************VBYM/20231019/eu-west-1/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date;x-amz-security-token, Signature=*****"
2023-10-19T13:58:05.675+0100 [DEBUG] backend-s3: HTTP Response Received: aws.operation=HeadObject aws.region=eu-west-1 aws.sdk=aws-sdk-go-v2 aws.service=S3 tf_backend.operation=Get tf_backend.req_id=63c46ab0-c213-4f8d-1c9b-8b5a0ac6520b tf_backend.s3.bucket=ob-[REDACTED]-eu-west-1-terraform tf_backend.s3.path=terraform.tfstate http.response.header.x_amz_request_id=XKPGJW5QCDDJA0TG http.response.header.content_type=application/json http.status_code=200 http.response.header.date="Thu, 19 Oct 2023 12:58:06 GMT" http.response.header.accept_ranges=bytes http.response.header.last_modified="Wed, 18 Oct 2023 13:43:56 GMT" http.response.header.x_amz_server_side_encryption=AES256 http.response.header.proxy_connection=Keep-Alive http.response.header.server=AmazonS3 http.response.body="" http.duration=66 http.response_content_length=78153 http.response.header.connection=Keep-Alive http.response.header.x_amz_id_2="v2SyhwwkLc8bqlz4U1dctQcY0qVKUL4VCeNfhKOuNw4+oVaBgoJoSQfYHk05qUYFxAxTtDq3IJI=" http.response.header.x_amz_version_id=WUYrxTVdg0eQFhSYVpZxp3BNVlhTxlAQ http.response.header.etag="\"e005e0206157a4ee4466d20f821bc318\"" http.response.header.cache_control=proxy-revalidate
2023-10-19T13:58:05.675+0100 [DEBUG] backend-s3: HTTP Request Sent: aws.operation=GetObject aws.region=eu-west-1 aws.sdk=aws-sdk-go-v2 aws.service=S3 tf_backend.operation=Get tf_backend.req_id=63c46ab0-c213-4f8d-1c9b-8b5a0ac6520b tf_backend.s3.bucket=ob-[REDACTED]-eu-west-1-terraform tf_backend.s3.path=terraform.tfstate http.request.body="" net.peer.name=ob-[REDACTED]-eu-west-1-terraform.s3.eu-west-1.amazonaws.com http.request.header.authorization="AWS4-HMAC-SHA256 Credential=ASIA************VBYM/20231019/eu-west-1/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;range;x-amz-content-sha256;x-amz-date;x-amz-security-token, Signature=*****" http.request.header.x_amz_security_token="*****" http.request.header.accept_encoding=identity http.url=https://ob-[REDACTED]-eu-west-1-terraform.s3.eu-west-1.amazonaws.com/terraform.tfstate?x-id=GetObject http.user_agent="APN/1.0 HashiCorp/1.0 Terraform/1.6.0 (+https://www.terraform.io) aws-sdk-go-v2/1.21.0 os/macos lang/go#1.21.1 md/GOOS#darwin md/GOARCH#arm64 api/s3#1.38.5 ft/s3-transfer" http.request.header.amz_sdk_request="attempt=1; max=5" http.request.header.range=bytes=0-5242879 http.request.header.x_amz_content_sha256=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 http.method=GET http.request.header.amz_sdk_invocation_id=b109984a-408a-44b9-a9e8-b49c46aa3385 http.request.header.x_amz_date=20231019T125805Z
2023-10-19T13:58:05.766+0100 [DEBUG] backend-s3: HTTP Response Received: aws.operation=GetObject aws.region=eu-west-1 aws.sdk=aws-sdk-go-v2 aws.service=S3 tf_backend.operation=Get tf_backend.req_id=63c46ab0-c213-4f8d-1c9b-8b5a0ac6520b tf_backend.s3.bucket=ob-[REDACTED]-eu-west-1-terraform tf_backend.s3.path=terraform.tfstate http.response.header.proxy_connection=Keep-Alive http.response.header.connection=Keep-Alive http.duration=60 http.response.header.date="Thu, 19 Oct 2023 12:58:05 GMT" http.response.header.server=AmazonS3 http.response.header.cache_control=proxy-revalidate http.response.header.x_amz_request_id=XKPYFG87BPS57CX7 http.status_code=403 http.response.header.x_amz_id_2="U7iKvrG1tQoSygW7mehKxqLHT3yCsZgYuuaoAVmNMpSAz8b7WS41++NWCrdVYzwigSZYAio7ruY=" http.response.header.age=0 http.response.header.content_type=application/xml
  http.response.body=
  | <?xml version="1.0" encoding="UTF-8"?>
  | <Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><AWSAccessKeyId>ASIA************VBYM</AWSAccessKeyId><StringToSign>AWS4-HMAC-SHA256
  | 20231019T125805Z
  | 20231019/eu-west-1/s3/aws4_request
  | e712eb376d5c8966e49476bdf5ed859ca5410507254bf9c107b1e63a6c25cc67</StringToSign><SignatureProvided>89073fd6d1cb5d4add1720b954f1c7bc1d87fcfa829dfedebc2a7a2317269c16</SignatureProvided><StringToSignBytes>41 57 53 34 2d 48 4d 41 43 2d 53 48 41 32 35 36 0a 32 30 32 33 31 30 31 39 54 31 32 35 38 30 35 5a 0a 32 30 32 33 31 30 31 39 2f 65 75 2d 77 65 73 74 2d 31 2f 73 33 2f 61 77 73 34 5f 72 65 71 75 65 73 74 0a 65 37 31 32 65 62 33 37 36 64 35 63 38 39 36 36 65 34 39 34 37 36 62 64 66 35 65 64 38 35 39 63 61 35 34 31 30 35 30 37 32 35 34 62 66 39 63 31 30 37 62 31 65 36 33 61 36 63 32 35 63 63 36 37</StringToSignBytes><CanonicalRequest>GET
  | /terraform.tfstate
  | x-id=GetObject
  | accept-encoding:identity
  | amz-sdk-invocation-id:b109984a-408a-44b9-a9e8-b49c46aa3385
  | amz-sdk-request:attempt=1; max=5
  | host:ob-[REDACTED]-eu-west-1-terraform.s3.eu-west-1.amazonaws.com
  | range:
  | x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
  | x-amz-date:20231019T125805Z
  | x-amz-security-token:IQoJb3JpZ2luX2VjEKX//////////wEaCWV1LXdlc3QtMSJIMEYCIQDVAJjWItEqp1aJipucGZkqndGkHvMKEF6z6ngbe6ddoQIhAK9F7LrbAc09NppzxLvYhc736xkghnBXE9ul+i4JUabKKvACCL7//////////wEQAxoMNjU4MDg0ODg5NjExIgxOEurbb7lKF2jiYXEqxAKoDC/tApUtseKWZymLr8PHDKmpcJoyZMqYp0mGhRoNBC7rs2iLgMKGcBOMEnFNrLTFvulE/KtTSkPUle7JoIaqBZ155UJlB0IQS6EoKucwte0Tnft6uQXRmK2X7seQC/oya7fNJgB/PLfxqRIgnVqomOmDrX3JQOL/l0aiCr+doCafMh73JZqyVeEakMqEIZq1naugVnFD8iIZh4p519XJrU7DIr50C4fq8VpPsjBkDE9FVmg4XIvGt3/jw/CI5MocT9I+i55GQSIAO2QbI3mHgNcDY2NCuj6j0qPngw2Cl9LZGr8VszH25g9Lf+bujiKFIWsT+Lcdd6Qy2KVaN4HJV1kHnWhGC/nMrGvSXpBg3AJFmojiuFi+PXOnt5wTXxOpY1hgj1uvuq+5uIszY676dGAbSNIiOxOTMIy4Tqt4tVTD4BYwv8fEqQY6mgF5eLs7tNbWs++1LMqex7wzCrq1NSk8sMvogGk1cZdlV7mCzV2eRGLHfNYVY32xTJopr3bHFFh6f3w9k8kANDwW29y0qXsPAMiqoAJIK7FIumIzOBOUPECeRtwaOv4ILIj6zoA8wEvhSOmryOxkBZb9eH9mkR6DkOzBDg1xdGS05FyH/mTixrsdmvcBNxDSlHCVpxYScjv5I+hB
  | 
  | accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;range;x-amz-content-sha256;x-amz-date;x-amz-security-token
  | e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855</CanonicalRequest><CanonicalRequestBytes>47 45 54 0a 2f 70 68 69 6c 2d 6e 67 69 6e 78 2d 61 63 63 6f 75 6e 74 2e 74 66 73 74 61 74 65 0a 78 2d 69 64 3d 47 65 74 4f 62 6a 65 63 74 0a 61 63 63 65 70 74 2d 65 6e 63 6f 64 69 6e 67 3a 69 64 65 6e 74 69 74 79 0a 61 6d 7a 2d 73 64 6b 2d 69 6e 76 6f 63 61 74 69 6f 6e 2d 69 64 3a 62 31 30 39 39 38 34 61 2d 34 30 38 61 2d 34 34 62 39 2d 61 39 65 38 2d 62 34 39 63 34 36 61 61 33 33 38 35 0a 61 6d 7a 2d 73 64 6b 2d 72 65 71 75 65 73 74 3a 61 74 74 65 6d 70 74 3d 31 3b 20 6d 61 78 3d 35 0a 68 6f 73 74 3a 6f 62 2d 36 35 38 30 38 34 38 38 39 36 31 31 2d 65 75 2d 77 65 73 74 2d 31 2d 74 65 72 72 61 66 6f 72 6d 2e 73 33 2e 65 75 2d 77 65 73 74 2d 31 2e 61 6d 61 7a 6f 6e 61 77 73 2e 63 6f 6d 0a 72 61 6e 67 65 3a 0a 78 2d 61 6d 7a 2d 63 6f 6e 74 65 6e 74 2d 73 68 61 32 35 36 3a 65 33 62 30 63 34 34 32 39 38 66 63 31 63 31 34 39 61 66 62 66 34 63 38 39 39 36 66 62 39 32 34 32 37 61 65 34 31 65 34 36 34 39 62 39 33 34 63 61 34 39 35 39 39 31 62 37 38 35 32 62 38 35 35 0a 78 2d 61 6d 7a 2d 64 61 74 65 3a 32 30 32 33 31 30 31 39 54 31 32 35 38 30 35 5a 0a 78 2d 61 6d 7a 2d 73 65 63 75 72 69 74 79 2d 74 6f 6b 65 6e 3a 49 51 6f 4a 62 33 4a 70 5a 32 6c 75 58 32 56 6a 45 4b 58 2f 2f 2f 2f 2f 2f 2f 2f 2f 2f 77 45 61 43 57 56 31 4c 58 64 6c 63 33 51 74 4d 53 4a 49 4d 45 59 43 49 51 44 56 41 4a 6a 57 49 74 45 71 70 31 61 4a 69 70 75 63 47 5a 6b 71 6e 64 47 6b 48 76 4d 4b 45 46 36 7a 36 6e 67 62 65 36 64 64 6f 51 49 68 41 4b 39 46 37 4c 72 62 41 63 30 39 4e 70 70 7a 78 4c 76 59 68 63 37 33 36 78 6b 67 68 6e 42 58 45 39 75 6c 2b 69 34 4a 55 61 62 4b 4b 76 41 43 43 4c 37 2f 2f 2f 2f 2f 2f 2f 2f 2f 2f 77 45 51 41 78 6f 4d 4e 6a 55 34 4d 44 67 30 4f 44 67 35 4e 6a 45 78 49 67 78 4f 45 75 72 62 62 37 6c 4b 46 32 6a 69 59 58 45 71 78 41 4b 6f 44 43 2f 74 41 70 55 74 73 65 4b 57 5a 79 6d 4c 72 38 50 48 44 4b 6d 70 63 4a 6f 79 5a 4d 71 59 70 30 6d 47 68 52 6f 4e 42 43 37 72 73 32 69 4c 67 4d 4b 47 63 42 4f 4d 45 6e 46 4e 72 4c 54 46 76 75 6c 45 2f 4b 74 54 53 6b 50 55 6c 65 37 4a 6f 49 61 71 42 5a 31 35 35 55 4a 6c 42 30 49 51 53 36 45 6f 4b 75 63 77 74 65 30 54 6e 66 74 36 75 51 58 52 6d 4b 32 58 37 73 65 51 43 2f 6f 79 61 37 66 4e 4a 67 42 2f 50 4c 66 78 71 52 49 67 6e 56 71 6f 6d 4f 6d 44 72 58 33 4a 51 4f 4c 2f 6c 30 61 69 43 72 2b 64 6f 43 61 66 4d 68 37 33 4a 5a 71 79 56 65 45 61 6b 4d 71 45 49 5a 71 31 6e 61 75 67 56 6e 46 44 38 69 49 5a 68 34 70 35 31 39 58 4a 72 55 37 44 49 72 35 30 43 34 66 71 38 56 70 50 73 6a 42 6b 44 45 39 46 56 6d 67 34 58 49 76 47 74 33 2f 6a 77 2f 43 49 35 4d 6f 63 54 39 49 2b 69 35 35 47 51 53 49 41 4f 32 51 62 49 33 6d 48 67 4e 63 44 59 32 4e 43 75 6a 36 6a 30 71 50 6e 67 77 32 43 6c 39 4c 5a 47 72 38 56 73 7a 48 32 35 67 39 4c 66 2b 62 75 6a 69 4b 46 49 57 73 54 2b 4c 63 64 64 36 51 79 32 4b 56 61 4e 34 48 4a 56 31 6b 48 6e 57 68 47 43 2f 6e 4d 72 47 76 53 58 70 42 67 33 41 4a 46 6d 6f 6a 69 75 46 69 2b 50 58 4f 6e 74 35 77 54 58 78 4f 70 59 31 68 67 6a 31 75 76 75 71 2b 35 75 49 73 7a 59 36 37 36 64 47 41 62 53 4e 49 69 4f 78 4f 54 4d 49 79 34 54 71 74 34 74 56 54 44 34 42 59 77 76 38 66 45 71 51 59 36 6d 67 46 35 65 4c 73 37 74 4e 62 57 73 2b 2b 31 4c 4d 71 65 78 37 77 7a 43 72 71 31 4e 53 6b 38 73 4d 76 6f 67 47 6b 31 63 5a 64 6c 56 37 6d 43 7a 56 32 65 52 47 4c 48 66 4e 59 56 59 33 32 78 54 4a 6f 70 72 33 62 48 46 46 68 36 66 33 77 39 6b 38 6b 41 4e 44 77 57 32 39 79 30 71 58 73 50 41 4d 69 71 6f 41 4a 49 4b 37 46 49 75 6d 49 7a 4f 42 4f 55 50 45 43 65 52 74 77 61 4f 76 34 49 4c 49 6a 36 7a 6f 41 38 77 45 76 68 53 4f 6d 72 79 4f 78 6b 42 5a 62 39 65 48 39 6d 6b 52 36 44 6b 4f 7a 42 44 67 31 78 64 47 53 30 35 46 79 48 2f 6d 54 69 78 72 73 64 6d 76 63 42 4e 78 44 53 6c 48 43 56 70 78 59 53 63 6a 76 35 49 2b 68 42 0a 0a 61 63 63 65 70 74 2d 65 6e 63 6f 64 69 6e 67 3b 61 6d 7a 2d 73 64 6b 2d 69 6e 76 6f 63 61 74 69 6f 6e 2d 69 64 3b 61 6d 7a 2d 73 64 6b 2d 72 65 71 75 65 73 74 3b 68 6f 73 74 3b 72 61 6e 67 65 3b 78 2d 61 6d 7a 2d 63 6f 6e 74 65 6e 74 2d 73 68 61 32 35 36 3b 78 2d 61 6d 7a 2d 64 61 74 65 3b 78 2d 61 6d 7a 2d 73 65 63 75 72 69 74 79 2d 74 6f 6b 65 6e 0a 65 33 62 30 63 34 34 32 39 38 66 63 31 63 31 34 39 61 66 62 66 34 63 38 39 39 36 66 62 39 32 34 32 37 61 65 34 31 65 34 36 34 39 62 39 33 34 63 61 34 39 35 39 39 31 62 37 38 35 32 62 38 35 35</CanonicalRequestBytes><RequestId>XKPYFG87BPS57CX7</RequestId><HostId>U7iKvrG1tQoSygW7mehKxqLHT3yCsZgYuuaoAVmNMpSAz8b7WS41++NWCrdVYzwigSZYAio7ruY=</HostId></Error>
  | [truncated...]
  
2023-10-19T13:58:05.772+0100 [DEBUG] backend-s3: request failed with unretryable error https response error StatusCode: 403, RequestID: XKPYFG87BPS57CX7, HostID: U7iKvrG1tQoSygW7mehKxqLHT3yCsZgYuuaoAVmNMpSAz8b7WS41++NWCrdVYzwigSZYAio7ruY=, api error SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.: tf_backend.operation=Get tf_backend.req_id=63c46ab0-c213-4f8d-1c9b-8b5a0ac6520b tf_backend.s3.bucket=ob-[REDACTED]-eu-west-1-terraform tf_backend.s3.path=terraform.tfstate

Expected Behavior

Terraform should be able to refresh the state and initialise properly.

Actual Behavior

Terraform attempts to refresh the state (GetObject) and a SignatureDoesNotMatch error is thrown.

Steps to Reproduce

  1. terraform init (after first apply)

Additional Context

Terraform 1.6.0 is not able to refresh an existing state file after first run for a new state held in an s3 backend.

The first run is able to perform the ListBucketsV2, HeadObject and PutObject operations to init and apply. On subsequent runs, the ListBucketsV2 and HeadObject operations are successful, but the GetObject operation fails (HeadObject returns that an object with the key name is present) with a SignatureDoesNotMatch error.

This behaviour is only observed when running Terraform locally. When the same execution is made via a jenkins pipeline, no issue occurs. The only difference between the two is that our jenkins pipeline gets session credentials via an assume-role CLI call and exports them to environment variables, whereas locally we are using a federated AD role to gain session credentials.

I have tried numerous configurations for the s3 backend:

  • assume role config
  • hardcoding the keys/token on the backend config block (see above)
  • setting/unsetting the related environment variables (AWS_PROFILE, AWS_SESSION_TOKEN etc.) to no avail.
  • changing profile config in ~/.aws/credentials
  • have also tried with 1.6.1 and 1.6.2, behaviour is the same

My only summary is that the signature for the GetObject request is not being created properly. From debug logs i can see the session credentials are received correctly whichever way I configure the backend, so I don't believe the issue is in this area. It's just that GetObject call that Terraform doesn't like!

NOTE: The exact same configuration works perfectly if I switch back to either version 1.3.1 or 1.5.3 (these are the only other pre-1.6.x versions available to us).

Any help would be greatly appreciated.

References

No response
@lambbuster lambbuster added bug new new issue not yet triaged labels Oct 25, 2023
@crw crw added the backend/s3 label Oct 25, 2023
@josh-keller
Copy link

I believe I may be encountering a similar issue

Terraform v1.5.7
on darwin_arm64
+ provider registry.terraform.io/hashicorp/aws v5.24.0
+ provider registry.terraform.io/hashicorp/external v2.3.1
+ provider registry.terraform.io/hashicorp/null v3.2.1

I am listing objects in a bucket and then getting details about each:

data "aws_s3_objects" "ova_objects" {
  bucket = var.src_bucket
  prefix = var.prefix
}

data "aws_s3_object" "object_details" {
  for_each = toset(data.aws_s3_objects.ova_objects.keys)

  bucket = var.src_bucket
  key = each.value
}

The aws_s3_objects works fine and all but one of the aws_s3_object requests works. But one errors with this:

 Error: downloading S3 Bucket ([redacted]-us-west-2-[redacted]) Object (<object-prefix>/encrypt_password.txt): operation error S3: GetObject, https response error StatusCode: 403, RequestID: ***************, HostID: *************, api error SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.

I've researched other causes of this problem like clock sync and credentials issues. I've ruled those out because all my other requests succeed. I have other prefixes that have similar files in them. The requests also fail on the encrypt_password.txt file in those key prefixes as well.

Is it possible there is a bug in the signature generation?

@lambbuster
Copy link
Author

lambbuster commented Nov 10, 2023 •
edited
Loading

Thanks for replying @josh-keller - I’ve still not found a solution after countless hours troubleshooting.

I just updated my initial comment to mention that the exact same configuration works perfectly with previous versions 1.3.1 and 1.5.3, so it seems this could either be a bug or some behavioural change based off the same configuration with 1.6.x.

@josh-keller
Copy link

josh-keller commented Nov 10, 2023 •
edited
Loading

@lambbuster I have tried on 1.3.1 and 1.5.3 and it doesn't seem to work. Wondering if this could be in the AWS provider. What version of that are you using with those earlier versions of Terraform?

@josh-keller
Copy link

Actually, I have narrowed this down. My request works with v5.16.2 of the AWS provider but not v5.17.0. So there appears to be a breaking change in v5.17.0. @lambbuster can you confirm?

@josh-keller josh-keller mentioned this issue Nov 10, 2023
Open
@josh-keller
Copy link

I see in the v5.17.0 CHANGELOG that the AWS SDK for Go was upgraded to v2. So I'm guessing it's likely that there is a bug upstream with v2 of the SDK.
I've opened an issue on the provider: hashicorp/terraform-provider-aws#34351

@lambbuster
Copy link
Author

@josh-keller - for the older TF versions (1.3.1/1.5.3) we had the constraint set to use the latest version 4 release (~> 4.0), so 4.67.0.

I’ve just tried using 1.6.0 (and 1.6.3!) with 5.16.2 of the AWS provider and still get the same error :(

Thanks for helping with the troubleshooting!

@josh-keller
Copy link

I’ve just tried using 1.6.0 (and 1.6.3!) with 5.16.2 of the AWS provider and still get the same error :(

Interesting, maybe it's not the Go SDK v2. Or maybe there are two different issues going on. Anyway, I have a version that works for me at the moment and need to move forward with that for now.

@gdavison gdavison mentioned this issue Nov 14, 2023
Merged
@Echeoss
Copy link

Echeoss commented Nov 17, 2023 •
edited
Loading

We have performed verification with #34243 and fix mentioning this Issue does not resolve it.

This issue is caused by PRs #33669 #33843 - which is switch from s3 client to s3 transfer manager. As s3 transfer manager is using HTTP range header and also includes it in AWS SigV4 generation so when proxy drop this header, AWS cannot correctly verify signature

In above samples we can see
SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;range;x-amz-content-sha256;x-amz-date;x-amz-security-token, Outbound: http.request.header.range=bytes=0-5242879 AWS Reponse to SigV4 verification failure : range (empty)

Issue will occur for everyone that uses proxy that does not support HTTP Header "Range" Reference: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Range

Proxies can have 3 behaviors:
a) does not support Range : when they see such header they drop it before outbound traffic request
b) support Range thru caching : from client perspective range is supported but proxy is dropping it for outbound traffic taking full response and on its own provide range capability thru caching of outbound response
c) support fully Range - passing it outbound as sent by client

This change was not part of 1.6.0-beta1 which still worked correctly with proxies not supporting range.

There are couple of options how we can approach this problem:
a) there will be a change to exclude or opt-out of using range header in SigV4 by s3 transfer manager
b) revert back to s3 client from s3 transfer manager
b) there will be no plan to change and fix this means breaking change and information in documentation that remote s3 backend will not work with proxies not supporting Range Headers

@rahul6941
Copy link

@Echeoss
Did you find solution for this ?

@Echeoss
Copy link

Echeoss commented Feb 13, 2024

@Echeoss Did you find solution for this ?

Hi. We switched for testing to local tfstate file. On remote execution we do not have proxy so issue does not exists.

@gdavison gdavison self-assigned this Apr 26, 2024
@gdavison gdavison mentioned this issue Jul 22, 2024
Closed
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@gdavison gdavison

Labels
backend/s3 bug new new issue not yet triaged
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@crw @gdavison @rahul6941 @josh-keller @lambbuster @Echeoss