Document AWS_SDK_LOAD_CONFIG on aws provider

[fernandrone]

Community Note

• Please vote on this pull request by adding a 👍 reaction to the original pull request comment to help the community and maintainers prioritize this request
• Please do not leave "+1" comments, they generate extra noise for pull request followers and do not help prioritize the request

---

Changes proposed in this pull request:

• Document AWS_SDK_LOAD_CONFIG requirement when setting the S3 Backend AWS profile through the AWS_PROFILE variable.

Fixes:

There are multiple (closed) issues about this with calls for documentation, such as:

• Terraform does not read AWS profile from environment variable #233
• Using terraform with assumed roles defined in shared configuration are ineffective #1184
• Use of AWS non default credentials profile ignored terraform#18402
  Closes AWS_PROVIDER not working, explicit 'provider=xx' works fine #8779

I've run some tests and confirmed that this is an issue both when configuring the S3 backend and when configuring the AWS provider (see other PR hashicorp/terraform#21122). I can detail the results if required, but given the number of existing issues and the aws-sdk-go documentation I think this behavior is as "confirmed" as it gets.

Notes:

AWS_SDK_LOAD_CONFIG is a configuration specific to AWS Golang SDK (and JavaScript SDK as well -- and no other SDK, as far as I can tell).

Per the AWS documentation, this must be set when using the Go SDK, otherwise AWS_PROFILE is ignored:

By default NewSession only loads credentials from the shared credentials file (~/.aws/credentials). If the AWS_SDK_LOAD_CONFIG environment variable is set to a truthy value, the session is created from the configuration values from the shared configuration (~/.aws/config) and shared credentials (~/.aws/credentials) files. See Sessions with a Shared Configuration File for more information.

This variable is not particularly well documented and seems to be a source of confusion, being limited to only two SDKs (Boto does not have it and profiles work on the fly). Personally if there's approval from the maintainers I'd edit the code to act as if AWS_SDK_LOAD_CONFIG is set true, but documenting seems to be the next best thing.

[apparentlymart]

I left a comment about this over in hashicorp/terraform#21122 but want to note it over here for better visibility:

If this environment variable is not normally required by other AWS-integrated tools like the AWS CLI then I think the ideal path here would be to find a way to make Terraform's AWS provider and S3 backend not require it either, since I think our goal is that any user with a properly-configured and functioning AWS CLI should be able to run Terraform without any special further configuration to get their credentials to apply.

I don't know if that ideal can actually be achieved within the limitations of the SDK, but perhaps we should investigate that further (if we didn't already) and consider documenting this additional awkward extra step as a last resort?

[fernandrone]

Alright, thanks @apparentlymart !

I answered at hashicorp/terraform#21122 but I'm mostly copying the answer here, since it's relevant to both scenarios:

According to the aws golang sdk docs, it seems one should be able to override this using NewSessionWithOptions:

Sessions can be created using the method above that will only load the additional config if the AWS_SDK_LOAD_CONFIG environment variable is set. Alternatively you can explicitly create a Session with shared config enabled. To do this you can use NewSessionWithOptions to configure how the Session will be created. Using the NewSessionWithOptions with SharedConfigState set to SharedConfigEnable will create the session as if the AWS_SDK_LOAD_CONFIG environment variable was set.

---

The aws terraform provider uses hashicorp/aws-sdk-go-base. It actually defines the session.SharedConfigEnable options here: https://github.com/hashicorp/aws-sdk-go-base/blob/master/session.go#L60

However, it does so only if Config.Profile is set (https://github.com/hashicorp/aws-sdk-go-base/blob/master/session.go#L42). When setting AWS_PROFILE from the environment variable, Config.Profile is not set, and this setting is ignored.

So a possible fix is to configure the CLI to set Config.Profile when AWS_PROFILE is set:

https://github.com/hashicorp/terraform/blob/364c3e70b7cf5c1e6c6e4aea570d32dcc9f5ecc9/backend/remote-state/s3/backend.go#L130

			"profile": {
				Type:        schema.TypeString,
				Optional:    true,
				Description: "AWS profile name",
				Default:     "",
			},

To:

			"profile": {
				Type:        schema.TypeString,
				Optional:    true,
				Description: "AWS profile name",
				Default:     schema.EnvDefaultFunc("AWS_PROFILE", ""),,
			},

Maybe support AWS_DEFAULT_PROFILE as well, but I think it's deprecated.

---

Anyway, I would've proposed this change from the beginning but I worried this'd need a fair bit of testing, and may need to wait for the next major version release since this might be considered a breaking change (there might be some terraform scripts out there running in an environment with AWS_PROFILE set, but AWS_SDK_LOAD_CONFIG unset, so they're actually using - knowingly or not - the default ~/.aws/credentials instead).

[bflad]

The wording of "must" here is overly strong, since AWS_PROFILE can successfully be used today in situations without AWS_SDK_LOAD_CONFIG. For example, the usage of only the AWS_PROFILE environment variable, works when the credentials are in a credentials file and the profile does not have any advanced configurations, such as: https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html

See also: #8779 (comment)

My recommendation would be to call out specific usages that require the additional environment variable, e.g.

Suggested change

	must also set `AWS_SDK_LOAD_CONFIG` to a truthy value, e.g. `AWS_SDK_LOAD_CONFIG=1`
	may also need to set `AWS_SDK_LOAD_CONFIG` to a truthy value (e.g. `AWS_SDK_LOAD_CONFIG=1`) for advanced AWS client configurations, such as profiles that use the `source_profile` or `role_arn` configurations.

[bflad]

Merged with suggestion above. 👍

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!