After upgrade from Kubernetes 1.29.2 to 1.30, dns-controller fails

[sridhar81]

/kind bug

1. What kops version are you running? The command kops version, will display
this information.
Client version: 1.30.0 (git-v1.30.0)

2. What Kubernetes version are you running? kubectl version will print the
version if a cluster is running or provide the Kubernetes version specified as
a kops flag.
Server Version: v1.30.3

3. What cloud provider are you using?
AWS

4. What commands did you run? What is the simplest way to reproduce this issue?
Upgraded the Kubernetes cluster from 1.29.2 to 1.30

5. What happened after the commands executed?
During the upgrade process, the DNS entries in route53 for kops-controller.internal.xxx and api.internal.xxx fails to be updated. The first 2 master nodes are upgraded, but the final one fails because the DNS still contains stale IPs and haven't been updated with the new IP's. If the IP's are manually updated to the new IP's, the upgrade process completes and the cluster is validated.

I found the following error in the dns-controller pod.
1 dnscontroller.go:134] Unexpected error in DNS controller, will retry in 2m40s: error querying for zones: error querying for DNS zones: error listing hosted zones: operation error Route 53: ListHostedZones, get identity: get credentials: failed to refresh cached credentials, failed to retrieve credentials, operation error STS: AssumeRoleWithWebIdentity, failed to resolve service endpoint, endpoint rule error, Invalid Configuration: Missing Region

Image:         registry.k8s.io/kops/dns-controller:1.30.0@sha256:c7b84dc295c2a4dd856e29d537eedf8d89f5af7648ef001111e36dafaa75ec95

6. What did you expect to happen?
The DNS entries for kops-controller.internal.xxx and api.internal.xxx are updated during the upgrade process

7. Please provide your cluster manifest. Execute
kops get --name my.example.com -o yaml to display your cluster manifest.
You may want to remove your cluster name and other sensitive information.

8. Please run the commands with most verbose logging by adding the -v 10 flag.
Paste the logs into this report, or in a gist and provide the gist link here.

9. Anything else do we need to know?

[sridhar81]

Duplicate of #16645
Hence, closing.