[Bug]: Wrong url for SSO refresh in Govcloud with fips

[tmccombs]

Terraform Core Version

1.3.6

AWS Provider Version

4.54.0

Affected Resource(s)

All. This error is with configuring the provider itself.

Expected Behavior

Terraform should be able to refresh the token when using a profile that uses AWS SSO.

Actual Behavior

Error when refreshing the token which results in not being able to

Relevant Error/Panic Output Snippet

╷
│ Error: configuring Terraform AWS Provider: no valid credential sources for Terraform AWS Provider found.
│ 
│ Please see https://registry.terraform.io/providers/hashicorp/aws
│ for more information about providing credentials.
│ 
│ Error: failed to refresh cached credentials, operation error STS: AssumeRole, failed to sign request: failed to retrieve credentials: operation error STS: AssumeRole, failed to sign request: failed to retrieve credentials: operation error SSO: GetRoleCredentials, https response error StatusCode: 0, RequestID: , request send failed, Get "https://portal.sso-fips.us-gov-west-1.amazonaws.com/federation/credentials?account_id=<REDACTED>9&role_name=<REDACTED>": dial tcp: lookup portal.sso-fips.us-gov-west-1.amazonaws.com on 10.12.65.158:53: no such host
│ 
│ 
│   with provider["registry.terraform.io/hashicorp/aws"],
│   on config.tf line 26, in provider "aws":
│   26: provider "aws" {
│ 
╵

Terraform Configuration Files

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "4.53"
    }
}

provider "aws" {
  profile = var.terraform_profile
  region  = "us-gov-west-1"

  use_fips_endpoint = true

  endpoints {
    # terraform uses the wrong endpoint for this. See https://github.com/hashicorp/terraform-provider-aws/issues/23619
    ram             = "https://ram.us-gov-west-1.amazonaws.com"
    route53resolver = "https://route53resolver.us-gov-west-1.amazonaws.com"
    kinesis         = "https://kinesis.us-gov-west-1.amazonaws.com"
    appautoscaling  = "https://application-autoscaling.us-gov-west-1.amazonaws.com"
    cloudfront      = "https://cloudfront-fips.amazonaws.com"
    sns             = "https://sns.us-gov-west-1.amazonaws.com"
    cloudformation  = "https://cloudformation.us-gov-west-1.amazonaws.com"
  }
}

Steps to Reproduce

Run a plan, apply, refresh etc. with config like above, using an AWS profile that uses AWS IAM Identity Center (previously SSO)

Debug Output

No response

Panic Output

No response

Important Factoids

This appears to be a regression in version 4.54.0. I don't get this error in version 4.53.0.

References

Possibly related to #23619

Would you like to implement a fix?

None

[github-actions]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[tmccombs]

Is there any workaround for this?

[tmccombs]

I think this was mostly likely caused by the upgrade from aws-sdk-go-base from v2.0.0-beta-22 to v2.0.0-beta-23

[tmccombs]

It looks like this is a bug in upstream. But it isn't possible to work around, because any endpoint override for sso isn't applied to the credentials provider. It does look like the endpoints for iam and sts are though. So at least maybe pass the sso endpoint override through to the awsbase package for the credentials provider.

[YakDriver]

This is an upstream AWS issue. Looking through the changes in hashicorp/aws-sdk-go-base (where credentials are handled) from v2.0.0-beta.22 (AWS provider v4.53.0) to v2.0.0-beta.23 (AWS provider v4.54.0), our changes were rather minor and I don't see anything that would cause or contribute to this FIPS problem except the changes in AWS package dependencies. In order of likelihood of being the problem, below are the changes in AWS packages we were using from beta.22 to beta.23.

github.com/aws/aws-sdk-go-v2/credentials v1.12.0 --> v1.13.12
github.com/aws/aws-sdk-go-v2/service/sts v1.16.4 --> v1.18.3
github.com/aws/aws-sdk-go-v2/config v1.15.4 --> v1.18.12
github.com/aws/aws-sdk-go-v2/service/iam v1.18.4 --> v1.19.2
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.12.4 --> v1.12.22
github.com/aws/aws-sdk-go-v2 v1.16.3 --> v1.17.4

[tmccombs]

github.com/aws/aws-sdk-go-v2/service/sts v1.16.4 --> v1.18.3

From my memory when I investigated before, I think this was likely the source.

[YakDriver]

There are a couple workarounds to try:

• In your AWS config file, you can specify the correct SSO URL.
• If the issue is coming up because of STS, you can set a custom STS endpoint in the provider.

[tmccombs]

In your AWS config file, you can specify the correct SSO URL.

It is correct.

If the issue is coming up because of STS, you can set a custom STS endpoint in the provider.

The problem is with the SSO endpoint. And setting that in the provider doesn't impact the credentials provider.

[tmccombs]

It sounds like the upstream issue has been fixed: aws/aws-sdk-go-v2#2166 (comment)

[jar-b]

v5.21.0 (tentative release date this Thursday, 10/12) will include an updated version of the AWS Go SDK in which this issue should be resolved. Please let us know if this issue persists after upgrading!

[github-actions]

This functionality has been released in v5.21.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.