provider: Lower retry threshold for DNS resolution failures

[bflad]

Add a session-wide retry handler that will lower the retry threshold when attempting to contact endpoints that do not exist or if there is a reasonably long DNS resolution issue.

Closes #4457
Closes #3760
Reference #1756
Reference #1354

Previously, the MaxRetries default of 25 would mean this RequestError for "no such host" could take upwards of 30 minutes to reveal (I gave up below):

export AWS_DEFAULT_REGION=eu-central-1
make testacc TEST=./aws TESTARGS='-run=TestAccAWSServiceDiscoveryPrivateDnsNamespace_basic'
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./aws -v -run=TestAccAWSServiceDiscoveryPrivateDnsNamespace_basic -timeout 120m
=== RUN   TestAccAWSServiceDiscoveryPrivateDnsNamespace_basic
^CFAIL	github.com/terraform-providers/terraform-provider-aws/aws	1559.376s
make: *** [testacc] Error 1

Now it will return in a reasonable timeframe:

export AWS_DEFAULT_REGION=eu-central-1
make testacc TEST=./aws TESTARGS='-run=TestAccAWSServiceDiscoveryPrivateDnsNamespace_basic'
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./aws -v -run=TestAccAWSServiceDiscoveryPrivateDnsNamespace_basic -timeout 120m
=== RUN   TestAccAWSServiceDiscoveryPrivateDnsNamespace_basic
--- FAIL: TestAccAWSServiceDiscoveryPrivateDnsNamespace_basic (38.19s)
	testing.go:518: Step 0 error: Error applying: 1 error(s) occurred:

		* aws_service_discovery_private_dns_namespace.test: 1 error(s) occurred:

		* aws_service_discovery_private_dns_namespace.test: RequestError: send request failed
		caused by: Post https://servicediscovery.eu-central-1.amazonaws.com/: dial tcp: lookup servicediscovery.eu-central-1.amazonaws.com: no such host
FAIL
FAIL	github.com/terraform-providers/terraform-provider-aws/aws	38.230s
make: *** [testacc] Error 1

#3760 case:

export AWS_DEFAULT_REGION=eu-central-1
make testacc TEST=./aws TESTARGS='-run=TestAccAWSSESConfigurationSet_basic'
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./aws -v -run=TestAccAWSSESConfigurationSet_basic -timeout 120m
=== RUN   TestAccAWSSESConfigurationSet_basic
--- FAIL: TestAccAWSSESConfigurationSet_basic (27.44s)
	testing.go:518: Step 0 error: Error applying: 1 error(s) occurred:

		* aws_ses_configuration_set.test: 1 error(s) occurred:

		* aws_ses_configuration_set.test: Error creating SES configuration set: RequestError: send request failed
		caused by: Post https://email.eu-central-1.amazonaws.com/: dial tcp: lookup email.eu-central-1.amazonaws.com: no such host
FAIL
FAIL	github.com/terraform-providers/terraform-provider-aws/aws	27.485s
make: *** [testacc] Error 1

#1756 case:

export AWS_DEFAULT_REGION=ap-southeast-1
make testacc TEST=./aws TESTARGS='-run=TestAccAWSEFSFileSystem_basic'
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./aws -v -run=TestAccAWSEFSFileSystem_basic -timeout 120m
=== RUN   TestAccAWSEFSFileSystem_basic
--- FAIL: TestAccAWSEFSFileSystem_basic (27.54s)
	testing.go:518: Step 0 error: Error applying: 1 error(s) occurred:

		* aws_efs_file_system.foo: 1 error(s) occurred:

		* aws_efs_file_system.foo: Error creating EFS file system: RequestError: send request failed
		caused by: Post https://elasticfilesystem.ap-southeast-1.amazonaws.com/2015-02-01/file-systems: dial tcp: lookup elasticfilesystem.ap-southeast-1.amazonaws.com: no such host
FAIL
FAIL	github.com/terraform-providers/terraform-provider-aws/aws	27.580s
make: *** [testacc] Error 1

[mbfrahry]

LGTM

[bflad]

This has been released in version 1.18.0 of the AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!