Allow custom endpoints to override the use_fips_endpoint flag for individual services.

[danielgmyers-figma]

Description

The use_fips_endpoint flag is handled as a global setting by terraform-provider-aws; this is generally a good thing, since if you're using FIPS endpoints you probably always want to use FIPS endpoints.

Unfortunately the AWS SDK does not always handle this correctly. For some services, such as AppConfig or CloudFront, the SDK generates a nonexistent or incorrect FIPS endpoint, so those resources can't be managed with the flag enabled.

It seems this has come up before in various issues (e.g. #23619, #25732) and the recommended workaround in those cases seems to have been to override the endpoint like this:

provider "aws" {
  # use fips endpoints by default...
  use_fips_endpoint = true
  endpoints = {
    # ... except where the service doesn't have one.
    appconfig = "https://appconfig.us-west-2.amazonaws.com"
  }
}

However, that workaround stopped working at some point; the AWS SDK now emits this error:

│ Error: reading AppConfig Environment (9lnorq1) for Application (gg5bczc)
│
│   with module.appconfig.aws_appconfig_environment.my_environment,
│   on ../modules/appconfig/main.tf line 45, in resource "aws_appconfig_environment" "my_environment":
│   45: resource "aws_appconfig_environment" "my_environment" {
│
│ operation error AppConfig: GetEnvironment, failed to resolve service endpoint, endpoint rule error, Invalid Configuration: FIPS and custom endpoint are not supported

In principle, the correct solution is to report the issue to AWS so that they can update the SDK with the correct endpoint configuration in these cases, then wait for the SDK to release an update, then wait for terraform-provider-aws to pick it up. However, users still need a workaround in the meantime, and the previously-used workaround no longer works.

This change addresses that problem; now, the provider's AWS client initialization code checks for the case where both use_fips_endpoint is true and a custom endpoint is provided and, if so, explicitly disables the client's UseFIPSEndpoint configuration prior to initialization. The custom endpoint is now honored and a warning is logged.

I updated the documentation for the provider's use_fips_endpoint and endpoints fields to reflect the new behavior.

I added two new acceptance tests (TestAccProvider_useFipsEndpointFlag and TestAccProvider_overrideUseFipsEndpointFlagForOneService) and renamed one of the existing tests (TestAccProvider_fipsEndpoint -> TestAccProvider_customEndpoint).

Relations

This is related to several open issues (many of the ones in this search) but I would hesitate to say it fixes any of them; instead this change simply allows a previously-recommended workaround to continue functioning.

References

• Per-service FIPS endpoints for AWS

Output from Acceptance Testing

% make testacc TESTS=TestAccProvider PKG=../provider                 
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./internal/service/../provider/... -v -count 1 -parallel 20 -run='TestAccProvider'  -timeout 360m
?       github.com/hashicorp/terraform-provider-aws/internal/provider/fwprovider        [no test files]
=== RUN   TestAccProvider_DefaultTags_emptyBlock
=== PAUSE TestAccProvider_DefaultTags_emptyBlock
=== RUN   TestAccProvider_DefaultTagsTags_none
=== PAUSE TestAccProvider_DefaultTagsTags_none
=== RUN   TestAccProvider_DefaultTagsTags_one
=== PAUSE TestAccProvider_DefaultTagsTags_one
=== RUN   TestAccProvider_DefaultTagsTags_multiple
=== PAUSE TestAccProvider_DefaultTagsTags_multiple
=== RUN   TestAccProvider_DefaultAndIgnoreTags_emptyBlocks
=== PAUSE TestAccProvider_DefaultAndIgnoreTags_emptyBlocks
=== RUN   TestAccProvider_endpoints
=== PAUSE TestAccProvider_endpoints
=== RUN   TestAccProvider_customEndpoint
=== PAUSE TestAccProvider_customEndpoint
=== RUN   TestAccProvider_unusualEndpoints
=== PAUSE TestAccProvider_unusualEndpoints
=== RUN   TestAccProvider_useFipsEndpointFlag
=== PAUSE TestAccProvider_useFipsEndpointFlag
=== RUN   TestAccProvider_overrideUseFipsEndpointFlagForOneService
=== PAUSE TestAccProvider_overrideUseFipsEndpointFlagForOneService
=== RUN   TestAccProvider_IgnoreTags_emptyBlock
=== PAUSE TestAccProvider_IgnoreTags_emptyBlock
=== RUN   TestAccProvider_IgnoreTagsKeyPrefixes_none
=== PAUSE TestAccProvider_IgnoreTagsKeyPrefixes_none
=== RUN   TestAccProvider_IgnoreTagsKeyPrefixes_one
=== PAUSE TestAccProvider_IgnoreTagsKeyPrefixes_one
=== RUN   TestAccProvider_IgnoreTagsKeyPrefixes_multiple
=== PAUSE TestAccProvider_IgnoreTagsKeyPrefixes_multiple
=== RUN   TestAccProvider_IgnoreTagsKeys_none
=== PAUSE TestAccProvider_IgnoreTagsKeys_none
=== RUN   TestAccProvider_IgnoreTagsKeys_one
=== PAUSE TestAccProvider_IgnoreTagsKeys_one
=== RUN   TestAccProvider_IgnoreTagsKeys_multiple
=== PAUSE TestAccProvider_IgnoreTagsKeys_multiple
=== RUN   TestAccProvider_Region_c2s
=== PAUSE TestAccProvider_Region_c2s
=== RUN   TestAccProvider_Region_china
=== PAUSE TestAccProvider_Region_china
=== RUN   TestAccProvider_Region_commercial
=== PAUSE TestAccProvider_Region_commercial
=== RUN   TestAccProvider_Region_govCloud
=== PAUSE TestAccProvider_Region_govCloud
=== RUN   TestAccProvider_Region_sc2s
=== PAUSE TestAccProvider_Region_sc2s
=== RUN   TestAccProvider_Region_stsRegion
=== PAUSE TestAccProvider_Region_stsRegion
=== RUN   TestAccProvider_AssumeRole_empty
=== PAUSE TestAccProvider_AssumeRole_empty
=== CONT  TestAccProvider_DefaultTags_emptyBlock
=== CONT  TestAccProvider_IgnoreTagsKeyPrefixes_one
=== CONT  TestAccProvider_Region_china
=== CONT  TestAccProvider_useFipsEndpointFlag
=== CONT  TestAccProvider_IgnoreTagsKeys_none
=== CONT  TestAccProvider_customEndpoint
=== CONT  TestAccProvider_IgnoreTagsKeyPrefixes_none
=== CONT  TestAccProvider_IgnoreTags_emptyBlock
=== CONT  TestAccProvider_overrideUseFipsEndpointFlagForOneService
=== CONT  TestAccProvider_unusualEndpoints
=== CONT  TestAccProvider_DefaultTagsTags_multiple
=== CONT  TestAccProvider_endpoints
=== CONT  TestAccProvider_DefaultAndIgnoreTags_emptyBlocks
=== CONT  TestAccProvider_Region_govCloud
=== CONT  TestAccProvider_AssumeRole_empty
=== CONT  TestAccProvider_Region_c2s
=== CONT  TestAccProvider_Region_commercial
=== CONT  TestAccProvider_IgnoreTagsKeyPrefixes_multiple
=== CONT  TestAccProvider_Region_sc2s
=== CONT  TestAccProvider_IgnoreTagsKeys_one
--- PASS: TestAccProvider_Region_sc2s (82.89s)
=== CONT  TestAccProvider_IgnoreTagsKeys_multiple
--- PASS: TestAccProvider_Region_c2s (83.91s)
=== CONT  TestAccProvider_DefaultTagsTags_one
--- PASS: TestAccProvider_Region_govCloud (84.02s)
=== CONT  TestAccProvider_Region_stsRegion
--- PASS: TestAccProvider_Region_commercial (84.25s)
=== CONT  TestAccProvider_DefaultTagsTags_none
--- PASS: TestAccProvider_Region_china (85.60s)
--- PASS: TestAccProvider_IgnoreTagsKeyPrefixes_none (98.09s)
--- PASS: TestAccProvider_IgnoreTagsKeys_one (98.80s)
--- PASS: TestAccProvider_IgnoreTagsKeys_none (99.80s)
--- PASS: TestAccProvider_DefaultTags_emptyBlock (100.75s)
--- PASS: TestAccProvider_DefaultTagsTags_multiple (101.49s)
--- PASS: TestAccProvider_IgnoreTags_emptyBlock (101.62s)
--- PASS: TestAccProvider_DefaultAndIgnoreTags_emptyBlocks (103.74s)
--- PASS: TestAccProvider_IgnoreTagsKeyPrefixes_multiple (104.55s)
--- PASS: TestAccProvider_IgnoreTagsKeyPrefixes_one (105.90s)
--- PASS: TestAccProvider_AssumeRole_empty (108.70s)
--- PASS: TestAccProvider_endpoints (120.80s)
--- PASS: TestAccProvider_unusualEndpoints (120.87s)
--- PASS: TestAccProvider_useFipsEndpointFlag (121.56s)
--- PASS: TestAccProvider_Region_stsRegion (45.35s)
--- PASS: TestAccProvider_overrideUseFipsEndpointFlagForOneService (135.61s)
--- PASS: TestAccProvider_DefaultTagsTags_one (51.76s)
--- PASS: TestAccProvider_IgnoreTagsKeys_multiple (52.95s)
--- PASS: TestAccProvider_DefaultTagsTags_none (52.18s)
--- PASS: TestAccProvider_customEndpoint (136.68s)
PASS
ok      github.com/hashicorp/terraform-provider-aws/internal/provider   140.856s
...

[github-actions]

Community Note

Voting for Prioritization

• Please vote on this pull request by adding a 👍 reaction to the original post to help the community and maintainers prioritize this pull request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

For Submitters

• Review the contribution guide relating to the type of change you are making to ensure all of the necessary steps have been taken.
• For new resources and data sources, use skaff to generate scaffolding with comments detailing common expectations.
• Whether or not the branch has been rebased will not impact prioritization, but doing so is always a welcome surprise.

[github-actions]

Welcome @danielgmyers-figma 👋

It looks like this is your first Pull Request submission to the Terraform AWS Provider! If you haven’t already done so please make sure you have checked out our CONTRIBUTOR guide and FAQ to make sure your contribution is adhering to best practice and has all the necessary elements in place for a successful approval.

Also take a look at our FAQ which details how we prioritize Pull Requests for inclusion.

Thanks again, and welcome to the community! 😃

[danielgmyers-figma]

I could use some help understanding something. My new acceptance test passed, but when I did terraform apply on my real stack it failed with Invalid Configuration: FIPS and custom endpoint are not supported as if my change hadn't been made. That led me to realize that I had to run make gen manually. Once I did that, terraform apply succeeded.

However, now I don't understand why my new acceptance test was passing. If I remove the endpoint override from the test config, it fails with a "no such host" error trying to resolve the nonexistent FIPS endpoint for AppConfig, as I expect; however, with the endpoint override in place, it successfully creates the appconfig resource with or without my code change.

One possibility that comes to mind is that it's possible the acceptance tests are using v1 of the AWS SDK for some reason, while runtime uses v2, since I believe the "Invalid Configuration: FIPS and custom endpoint are not supported" check was only added to AWS SDK v2. Is that possible? If so, how is that controlled? If not, what else might I be missing?

[gdavison]

Thanks for submitting this, @danielgmyers-figma. We've been having a few discussions internally and with our AWS contacts around FIPS and endpoint overrides. It was mostly prompted by #33952.

Generally, not all services support FIPS, but AWS SDK implementation

generate[s] these endpoints for forward compatibility reasons so that new regions "will just work" when rolled out from the service side

(aws/aws-sdk-go-v2#2336 (comment)). This behaviour is consistent with the AWS CLI, so it seems to be an overall design decision from AWS.

I suspect that the inability to override endpoints when use_fips_endpoint (and/or use_dualstack_endpoint) is enabled is related to our current use of V1 endpoint resolution, which has been superseded by V2 resolution (see https://aws.github.io/aws-sdk-go-v2/docs/configuring-sdk/endpoints/).

Thanks for all of the work you put into this PR. We're going to investigate updating endpoint resolution to V2, but keep this PR open in case that doesn't address it.

[danielgmyers-figma]

Sounds good, thanks @gdavison!

[github-actions]

This functionality has been released in v5.53.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[github-actions]

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.