cannot create Kubernetes client: Unauthorized

[conclusionlogic]

What happened?

After upgrading to provider-aws@v0.20.0 (which is based on aws-sdk-go-v2), provider-kubernetes is no longer able to connect to the EKS cluster using provider-kubernetes ProviderConfig.

With existing EKS clusters (created before upgrading provider-aws) the previously working connection to the EKS clusters break.
Reason: CannotConnectToProvider.

If a new EKS cluster is launched after upgrading provider-aws, provider-kubernetes is similarly unable to make a connect to the EKS cluster to proceed with the setup.

Comparing the related kubeconfig Secret objects referenced by provider-kubernetes ProviderConfig, the kubeconfig user token value is slightly shorter when provider-aws@v0.20.0 is installed compared to older versions of the provider, likely due to aws-sdk-go-v2.

Based on the discussion in Crossplane Slack, the same problem affects provider-helm.

How can we reproduce it?

composition.txt

What environment did it happen in?

Crossplane version: v1.4.1
Provider AWS: v0.20.0
Provider Kubernetes: v0.1.0
Kubernetes distribution: AWS EKS v1.21

[blut]

Exeriencing the same issue

[doubliez]

Same issue here with Crossplane v1.5.0, provider-aws v0.20.0 and provider-kubernetes v0.1.0.

[haarchri]

we have the same Issue with v0.20.0 :/ with provider-aws, provider-helm

[negz]

Just hit this myself - it's not immediately obvious to me what change is responsible for this regression but my first guess would be #602.

https://github.com/crossplane/provider-aws/blob/v0.20.0/pkg/clients/eks/eks.go#L336

The above function is the code that should be handling ensuring the kubeconfig connection detail is up-to-date.

[haarchri]

@negz thanks for the code function

• so we have a token part with the following:
  token: k8s-aws-v1.<<base64>>

via crossplane (in aws-provider 0.20.0 result is unauthorized) we setup connection secret with kubeconfig removed base64 and also removed base64 after k8s-aws-v1.

https://sts.us-west-2.amazonaws.com/?
  Action=GetCallerIdentity&
  Version=2011-06-15&
  X-Amz-Algorithm=AWS4-HMAC-SHA256&
  X-Amz-Credential=xxxx%2F20211031%2Fus-west-2%2Fsts%2Faws4_request&
  X-Amz-Date=20211031T103835Z&
  X-Amz-SignedHeaders=host&
  X-Amz-Signature=ea57163ecb07f1154ac3d254b1aa2eb2247ad9d6aae5bc4ad3572d5246ad2d92

lets try with aws-cli
aws eks get-token --cluster-name platform-ref-aws-cluster --region us-west-2

via aws cli: (is working)

https://sts.us-west-2.amazonaws.com/?
  Action=GetCallerIdentity&
  Version=2011-06-15&
  X-Amz-Algorithm=AWS4-HMAC-SHA256&
  X-Amz-Credential=xxxx%2F20211031%2Fus-west-2%2Fsts%2Faws4_request&
  X-Amz-Date=20211031T125725Z&
  X-Amz-Expires=60&
  X-Amz-SignedHeaders=host%3Bx-k8s-aws-id&
  X-Amz-Signature=67af338d83a46f2deeb2f2d15b58b23f21301595336ec613cc2b130b0d9a9d4e

think the problem is around the X-Amz-SignedHeaders with x-k8s-aws-id here https://github.com/crossplane/provider-aws/blob/v0.20.0/pkg/clients/eks/eks.go#L342
looks like that getCallerIdentity.SignedHeader is not working as expected or something is missing
possible this helps: aws/aws-sdk-go-v2#1137 (comment)

[haarchri]

i handled to get kubeconfig working again in v0.20.0 - provider-helm & helm-chart rollout successful - who can take a review & test also?
#901

[muvaf]

provider-aws v0.20.1 has been released with the fix. Please test and confirm that the problem is fixed once you have a chance 🙂

[conclusionlogic]

This fixed it. Thanks!