Cannot create many resources with use_fips_endpoint true due to incorrect endpoints

[tmccombs]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

Terraform v1.1.7
on linux_amd64
+ provider registry.terraform.io/hashicorp/aws v4.4.0

Affected Resource(s)

• aws_ram_resource_share

Terraform Configuration Files

Please include all Terraform configurations required to reproduce the bug. Bug reports without a functional reproduction may be closed without investigation.

provider "aws" {
  profile = "my-profile"
  region  = "us-gov-west-1"

  use_fips_endpoint = true
}

resource "aws_ram_resource_share" "gateway_share_west" {
  name                      = "gateway_share"
  allow_external_principals = true
}

Debug Output

Relevant portion:

�[0m�[1maws_ram_resource_share.gateway_share_west: Creating...�[0m�[0m
2022-03-10T12:58:59.605-0700 [INFO]  Starting apply for aws_ram_resource_share.gateway_share_west
2022-03-10T12:58:59.605-0700 [DEBUG] aws_ram_resource_share.gateway_share_west: applying the planned Create change
2022-03-10T12:58:59.606-0700 [DEBUG] provider.terraform-provider-aws_v4.4.0_x5: setting computed for "tags_all" from ComputedKeys: timestamp=2022-03-10T12:58:59.606-0700
2022-03-10T12:58:59.606-0700 [DEBUG] provider.terraform-provider-aws_v4.4.0_x5: Create RAM resource share request: {
  AllowExternalPrincipals: true,
  Name: "gateway_share"
}: timestamp=2022-03-10T12:58:59.606-0700
2022-03-10T12:58:59.606-0700 [DEBUG] provider.terraform-provider-aws_v4.4.0_x5: [aws-sdk-go] DEBUG: Request RAM/CreateResourceShare Details:
---[ REQUEST POST-SIGN ]-----------------------------
POST /createresourceshare HTTP/1.1
Host: ram-fips.us-gov-west-1.amazonaws.com
User-Agent: APN/1.0 HashiCorp/1.0 Terraform/1.1.7 (+https://www.terraform.io) terraform-provider-aws/4.4.0 (+https://registry.terraform.io/providers/hashicorp/aws) aws-sdk-go/1.43.9 (go1.17.6; linux; amd64)
Content-Length: 55
Authorization: REDACTED
Content-Type: application/json
X-Amz-Date: 20220310T195859Z
X-Amz-Security-Token: REDACTED
Accept-Encoding: gzip

{"allowExternalPrincipals":true,"name":"gateway_share"}
-----------------------------------------------------: timestamp=2022-03-10T12:58:59.606-0700
2022-03-10T12:59:00.377-0700 [DEBUG] provider.terraform-provider-aws_v4.4.0_x5: [aws-sdk-go] DEBUG: Send Request RAM/CreateResourceShare failed, attempt 0/25, error RequestError: send request failed
caused by: Post "https://ram-fips.us-gov-west-1.amazonaws.com/createresourceshare": dial tcp: lookup ram-fips.us-gov-west-1.amazonaws.com on 10.12.1.190:53: no such host: timestamp=2022-03-10T12:59:00.377-0700
2022-03-10T12:59:00.436-0700 [DEBUG] provider.terraform-provider-aws_v4.4.0_x5: [aws-sdk-go] DEBUG: Retrying Request RAM/CreateResourceShare, attempt 1: timestamp=2022-03-10T12:59:00.436-0700
2022-03-10T12:59:00.437-0700 [DEBUG] provider.terraform-provider-aws_v4.4.0_x5: [aws-sdk-go] DEBUG: Request RAM/CreateResourceShare Details:
---[ REQUEST POST-SIGN ]-----------------------------
POST /createresourceshare HTTP/1.1
Host: ram-fips.us-gov-west-1.amazonaws.com
User-Agent: APN/1.0 HashiCorp/1.0 Terraform/1.1.7 (+https://www.terraform.io) terraform-provider-aws/4.4.0 (+https://registry.terraform.io/providers/hashicorp/aws) aws-sdk-go/1.43.9 (go1.17.6; linux; amd64)
Content-Length: 55
Authorization: REDACTED
Content-Type: application/json
X-Amz-Date: 20220310T195900Z
X-Amz-Security-Token: REDACTED
Accept-Encoding: gzip

{"allowExternalPrincipals":true,"name":"gateway_share"}
-----------------------------------------------------: timestamp=2022-03-10T12:59:00.436-0700
2022-03-10T12:59:01.143-0700 [DEBUG] provider.terraform-provider-aws_v4.4.0_x5: [aws-sdk-go] DEBUG: Send Request RAM/CreateResourceShare failed, attempt 1/25, error RequestError: send request failed

Expected Behavior

The resource share is created

Actual Behavior

The following error:

╷
│ Error: Error creating RAM resource share: RequestError: send request failed
│ caused by: Post "https://ram-fips.us-gov-west-1.amazonaws.com/createresourceshare": dial tcp: lookup ram-fips.us-gov-west-1.amazonaws.com on 10.12.1.190:53: no such host
│ 
│   with aws_ram_resource_share.gateway_share_west,
│   on gateways.tf line 12, in resource "aws_ram_resource_share" "gateway_share_west":
│   12: resource "aws_ram_resource_share" "gateway_share_west" {
│ 
╵

Notice that the url used is ram-fips.us-gov-west-1.amazonaws.com, but it should be ram.us-gov-west-1.amazonaws.com. See https://aws.amazon.com/compliance/fips/.

Steps to Reproduce

1. terraform apply

[tmccombs]

If I add

  endpoints {
    ram = "https://ram.us-gov-west-1.amazonaws.com"
  }

to the provider definition, then it works.

[tmccombs]

the route53resolver endpoint is also incorrect. terraform is trying to use "https://route53resolver-fips.us-gov-west-1.amazonaws.com/" instead of https://route53resolver.us-gov-west-1.amazonaws.com. (that endpoint may not actually support fips though...)

[tmccombs]

Also, kinesis should be "https://kinesis.us-gov-west-1.amazonaws.com" instead of kinesis-fips.us-gov-west-1.amazonaws.com

[andyshinn]

Same with cloudfront:

│ caused by: Get "https://cloudfront-fips.us-east-1.amazonaws.com/2020-05-31/response-headers-policy/fb6bbcb5-d38d-4a16-a859-62644c6d0839": dial tcp: lookup cloudfront-fips.us-east-1.amazonaws.com on 10.124.124.1:53: no such host

[tmccombs]

Also for the following services, with what ti should be in us-gov-west-1 in parentheses

• ram (https://ram.us-gov-west-1.amazonaws.com)
• appautoscaling (https://application-autoscaling.us-gov-west-1.amazonaws.com)
• sns (https://sns.us-gov-west-1.amazonaws.com)
• cloudformation (https://cloudformation.us-gov-west-1.amazonaws.com)

[chadgeary]

the tagging endpoint is also broken when use_fips_endpoint = true, additionally the tagging endpoint is not customizable in provider "aws" { endpoints {} }

used by aws_resourcegroupstaggingapi_resources, e.g.:

data "aws_resourcegroupstaggingapi_resources" "this" {
  resource_type_filters = ["route53:hostedzone"]
}

[jameshochadel]

WAF endpoint is also broken in us-gov-west-1; should be https://waf-regional-fips.us-gov-west-1.amazonaws.com.

[andyshinn]

Add servicequotas to the list. The only correct ones are servicequotas.us-gov-east-1.amazonaws.com and servicequotas.us-gov-west-1.amazonaws.com but it tries to connect to servicequotas-fips.us-east-1.amazonaws.com.

[mebays]

Is there an update on when this may get worked on. I have a scenario where there are multiple environments some need fips and some don't so trying to set the AWS_USE_FIPS_ENDPOINT=true for the environments that do require fips to be used and the plans blow up with the environment and not without.

It would be nice to possibly make a list of all the services that are messed up, so that maybe it could be turned on for some of those terraform environment runs.

[lorengordon]

I'm curious if this is an issue with terraform, or actually an underlying issue with the AWS SDK. The option use_fips_endpoint is really a feature of the AWS SDK... See: https://docs.aws.amazon.com/sdkref/latest/guide/feature-endpoints.html. Anyone test the problem outside of terraform, maybe just using the aws-cli?

[tmccombs]

The cli uses the python sdk, but it is not an issue there.

It's possible that upgrading to v2 of the go aws sdk would fix this problem.

[RanVaknin]

Hello all, Go SDK maintainer here.

We are still waiting on the SSO and Redshift service teams to fix their FIPS endpoint configurations.
AFAIK the rest of the services are all fixed. If you are seeing this still is being an issue, you need to update your SDK version.

If anyone here is blocked from onboarding to FIPS, please consider using the AWS console to file an additional support ticket and reference the existing internal tickets (SSO: P80341048, Redshift P80336641) the more people escalate this, the faster it would get resolved.

Apologies for the inconvenience.
P.S. I don't monitor the correspondence not on AWS repos so will likely not see future comments on this thread.

All the best,
Ran~

[tmccombs]

On a related note, it would be helpful if you could specify use_fips_endpoint per service, instead of having it only at the global level.

[andyshinn]

Is this now broken in 5.x with custom endpoints? After an upgrade I now get an error Invalid Configuration: FIPS and custom endpoint are not supported. I see PR #34233 but not sure when this broke. Does anyone know the last version to work?

[tmccombs]

Indeed it is broken

[gdavison]

On a related note, it would be helpful if you could specify use_fips_endpoint per service, instead of having it only at the global level.

With the behaviour of AWS SDK for Go v2, it does seem that this would be a better design, for both the provider and the AWS SDKs

[gdavison]

Related: aws/aws-sdk-go-v2#2686

[gdavison]

@tmccombs, @andyshinn custom endpoints can be used along with use_fips_endpoint as of v5.53.0 thanks to #34233.

It doesn't resolve the similar issue with use_dualstack_endpoint and several deeper issues around FIPS endpoints, such as #33952

[github-actions]

Warning

This issue has been closed, meaning that any additional comments are hard for our team to see. Please assume that the maintainers will not see them.

Ongoing conversations amongst community members are welcome, however, the issue will be locked after 30 days. Moving conversations to another venue, such as the AWS Provider forum, is recommended. If you have additional concerns, please open a new issue, referencing this one where needed.

[github-actions]

This functionality has been released in v5.55.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.