Migrate AWS Verifier to aws-sdk-go-v2

[rifelpet]

Closes #16424

From this comment:

While presigned requests are still supported in V2, the presign methods and types no longer provide access to the request body, only their url and headers. See aws/aws-sdk-go-v2#1137. Kops-controller currently reads the request body to perform some validation:

kops/upup/pkg/fi/cloudup/awsup/aws_verifier.go

Lines 153 to 157 in 1c24423

	requestBytes, _ := io.ReadAll(stsRequest.Body)
	_, _ = stsRequest.Body.Seek(0, io.SeekStart)
	if stsRequest.HTTPRequest.Header.Get("Content-Length") != strconv.Itoa(len(requestBytes)) {
	return nil, fmt.Errorf("incorrect content-length")
	}

In V1 the presigned request is a POST however in V2 it is converted to a GET request and the normal Action=GetCallerIdentity&Version=2011-06-15 body is moved to URL query parameters:
https://github.com/aws/aws-sdk-go-v2/blob/bc2a669d3241023e20194cdfe042b8c275887e51/service/sts/api_client.go#L641-L645

I've removed the validation that checks the Content-Length signed header matches the size of the request body, given the request body is now empty.

This thread on the original kops-controller PR discusses potential upgrade challenges. In this case I believe we'll have the normal race with this type of change:

1. Old node is launched
2. New kops update cluster --yes and kops rolling-update --yes is ran
3. New control plane node is launched, new kops-controller is running
4. Old node attempts to join and fails because the GetCallerIdentity requests dont match.

In reality I dont anticipate this being a problem because the old node would have needed to not join the cluster in the time it took the control plane to bootstrap and launch the kops-controller daemonset pod (a significantly longer process than normal k8s node bootstrap). Eventually the failed node will be cleaned up by cluster-autoscaler or karpenter.

My only concern would be disruption to kops rolling-update if the node never joins which causes cluster validation to fail until it is cleaned up.

[rifelpet]

/cc @hakman @justinsb

/hold for discussion

[justinsb]

We added WithSTSRegionalEndpoint in #12043 . Is it still supported? (Is it automatic now?)

[justinsb]

We don't have great unit test coverage here, so I figured it might be good to add some: #16487

TBH I'm struggling to get the test to pass with v2, trying to figure it out...

I think we were already very sensitive to the aws-sdk version, so I don't think this is a huge regression version-wise.

[justinsb]

I think I've figured it out ... the V2 API puts a lot more in the query URL. So we likely need to pass along the url/method/headers. The good thing is that eliminates the dependency on the aws-sdk-go version (going forwards!)

Here's my WIP / PoC branch ... let me know if you want me e.g. to send a PR to your branch or something like that. I'm happy to fix up the tests, the code change itself here is pretty simple (we encode an awsV2Token, including the url & method)

https://github.com/kubernetes/kops/compare/master...justinsb:kops:aws-sdk-go-v2-verifier?expand=1

[rifelpet]

I think I've figured it out ... the V2 API puts a lot more in the query URL. So we likely need to pass along the url/method/headers. The good thing is that eliminates the dependency on the aws-sdk-go version (going forwards!)

Here's my WIP / PoC branch ... let me know if you want me e.g. to send a PR to your branch or something like that. I'm happy to fix up the tests, the code change itself here is pretty simple (we encode an awsV2Token, including the url & method)

master...justinsb:kops:aws-sdk-go-v2-verifier?expand=1 (compare)

Yes, feel free to PR to my branch or just push directly to it. I just rebased to resolve conflicts. Adding the other data in to the token makes sense to me 👍🏻

[hakman]

/retest

[rifelpet]

/unhold

/cc @hakman

[hakman]

/lgtm
/assign @justinsb

[justinsb]

Thanks @rifelpet

/approve
/lgtm

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: justinsb

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [justinsb]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment