aws-sdk-go-v2 route53 failing IMDS region discovery log messages

[elliotdobson]

/kind bug

1. What kops version are you running? The command kops version, will display
this information.
Client version: 1.30.1 (git-v1.30.1)

2. What Kubernetes version are you running? kubectl version will print the
version if a cluster is running or provide the Kubernetes version specified as
a kops flag.
Server Version: v1.29.7

3. What cloud provider are you using?
AWS

4. What commands did you run? What is the simplest way to reproduce this issue?
When upgrading from kOps v1.29.2 -> v1.30.1 we are receiving some DEBUG messages from the aws-sdk when running command like kops update cluster from a laptop (outside of AWS).

5. What happened after the commands executed?

SDK 2024/09/16 08:54:03 DEBUG retrying request /, attempt 2
SDK 2024/09/16 08:54:04 DEBUG retrying request /, attempt 3
SDK 2024/09/16 08:54:05 DEBUG retrying request /, attempt 4
SDK 2024/09/16 08:54:06 DEBUG retrying request /, attempt 5
SDK 2024/09/16 08:54:06 DEBUG max retry attempts exhausted, max 5
SDK 2024/09/16 08:54:23 DEBUG retrying request /, attempt 2
SDK 2024/09/16 08:54:25 DEBUG retrying request /, attempt 3
SDK 2024/09/16 08:54:26 DEBUG retrying request /, attempt 4
SDK 2024/09/16 08:54:27 DEBUG retrying request /, attempt 5
SDK 2024/09/16 08:54:27 DEBUG request failed with unretryable error canceled, context deadline exceeded
SDK 2024/09/16 08:54:27 DEBUG request failed with unretryable error canceled, context deadline exceeded
I0916 08:55:26.635859   85354 executor.go:113] Tasks: 0 done / 211 total; 64 can run
I0916 08:55:29.085395   85354 executor.go:113] Tasks: 64 done / 211 total; 32 can run
I0916 08:55:29.706467   85354 executor.go:113] Tasks: 96 done / 211 total; 47 can run
I0916 08:55:30.975327   85354 executor.go:113] Tasks: 143 done / 211 total; 15 can run
I0916 08:55:33.756963   85354 executor.go:113] Tasks: 158 done / 211 total; 23 can run
I0916 08:55:35.424744   85354 executor.go:113] Tasks: 181 done / 211 total; 10 can run
I0916 08:55:36.377389   85354 executor.go:113] Tasks: 191 done / 211 total; 10 can run
I0916 08:55:36.549778   85354 executor.go:113] Tasks: 201 done / 211 total; 10 can run
I0916 08:55:36.704646   85354 executor.go:113] Tasks: 211 done / 211 total; 0 can run
SDK 2024/09/16 08:55:37 DEBUG retrying request /, attempt 2
SDK 2024/09/16 08:55:38 DEBUG retrying request /, attempt 3
SDK 2024/09/16 08:55:39 DEBUG retrying request /, attempt 4
SDK 2024/09/16 08:55:40 DEBUG retrying request /, attempt 5
SDK 2024/09/16 08:55:40 DEBUG max retry attempts exhausted, max 5
I0916 08:55:45.297521   85354 dns.go:235] Pre-creating DNS records
...

6. What did you expect to happen?
No failed requests, however it does not seem to be causing an error or preventing us from upgrading.

7. Please provide your cluster manifest. Execute
kops get --name my.example.com -o yaml to display your cluster manifest.
You may want to remove your cluster name and other sensitive information.

8. Please run the commands with most verbose logging by adding the -v 10 flag.
Paste the logs into this report, or in a gist and provide the gist link here.

I0916 08:27:37.810079   77227 populate_cluster_spec.go:389] Defaulted PodCIDR to 100.96.0.0/11
I0916 08:27:37.810097   77227 populate_cluster_spec.go:398] Defaulted ServiceClusterIPRange to 100.64.0.0/13
I0916 08:27:37.810120   77227 subnets.go:51] All subnets have CIDRs; skipping assignment logic
I0916 08:27:37.810169   77227 defaults.go:262] Not setting up Proxy Excludes
I0916 08:27:37.810196   77227 clientset.go:114] Using keystore path: "s3://example-k8s-kops/example.k8s.kx.gd/pki"
I0916 08:27:37.810962   77227 middleware.go:44] AWS request:  ?
SDK 2024/09/16 08:27:39 DEBUG retrying request /, attempt 2
I0916 08:27:39.064319   77227 middleware.go:44] AWS request:  ?
SDK 2024/09/16 08:27:40 DEBUG retrying request /, attempt 3
I0916 08:27:40.068135   77227 middleware.go:44] AWS request:  ?
SDK 2024/09/16 08:27:41 DEBUG retrying request /, attempt 4
I0916 08:27:41.071150   77227 middleware.go:44] AWS request:  ?
SDK 2024/09/16 08:27:42 DEBUG retrying request /, attempt 5
I0916 08:27:42.073258   77227 middleware.go:44] AWS request:  ?
SDK 2024/09/16 08:27:42 DEBUG max retry attempts exhausted, max 5
I0916 08:27:42.074464   77227 middleware.go:44] AWS request:  ?
I0916 08:27:42.811986   77227 route53.go:69] Unable to discover region by IMDS, using SDK defaults: operation error ec2imds: GetRegion, request canceled, context deadline exceeded

9. Anything else do we need to know?
This looks related to the aws-sdk-go-v2 migration in #16424, and it seems like the fallback in #16778 is working as expected but it would be nice to suppress these messages.

[elliotdobson]

@justinsb @rifelpet I've updated to kOps v1.30.2 (which should have the fix included) but the DEBUG messages are still present in kops update cluster:

$ kops version
Client version: 1.30.2 (git-v1.30.2)
$ kops update cluster
SDK 2024/11/29 10:39:33 DEBUG retrying request /, attempt 2
SDK 2024/11/29 10:39:34 DEBUG retrying request /, attempt 3
SDK 2024/11/29 10:39:35 DEBUG retrying request /, attempt 4
SDK 2024/11/29 10:39:36 DEBUG retrying request /, attempt 5
SDK 2024/11/29 10:39:36 DEBUG max retry attempts exhausted, max 5
SDK 2024/11/29 10:39:52 DEBUG retrying request /, attempt 2
SDK 2024/11/29 10:39:53 DEBUG retrying request /, attempt 3
SDK 2024/11/29 10:39:54 DEBUG retrying request /, attempt 4
SDK 2024/11/29 10:39:55 DEBUG retrying request /, attempt 5
SDK 2024/11/29 10:39:55 DEBUG max retry attempts exhausted, max 5
I1129 10:41:03.876340   37927 executor.go:113] Tasks: 0 done / 213 total; 64 can run
I1129 10:41:04.957852   37927 executor.go:113] Tasks: 64 done / 213 total; 32 can run
I1129 10:41:05.669357   37927 executor.go:113] Tasks: 96 done / 213 total; 47 can run
I1129 10:41:06.384940   37927 executor.go:113] Tasks: 143 done / 213 total; 16 can run
I1129 10:41:08.443814   37927 executor.go:113] Tasks: 159 done / 213 total; 24 can run
I1129 10:41:10.016082   37927 executor.go:113] Tasks: 183 done / 213 total; 10 can run
I1129 10:41:10.390071   37927 executor.go:113] Tasks: 193 done / 213 total; 10 can run
I1129 10:41:10.512488   37927 executor.go:113] Tasks: 203 done / 213 total; 10 can run
I1129 10:41:10.629467   37927 executor.go:113] Tasks: 213 done / 213 total; 0 can run
SDK 2024/11/29 10:41:11 DEBUG retrying request /, attempt 2
SDK 2024/11/29 10:41:13 DEBUG retrying request /, attempt 3
SDK 2024/11/29 10:41:14 DEBUG retrying request /, attempt 4
SDK 2024/11/29 10:41:15 DEBUG request failed with unretryable error canceled, context deadline exceeded
I1129 10:41:19.613060   37927 dns.go:235] Pre-creating DNS records