Malformed response when requesting EKS token from gov-cloud (affects >=2.10.4)

[timfallmk]

Describe the bug

After updating to cli v 2.10.4 (and continuing in 2.11.0) we get the following response from our clusters running in Gov-Cloud

kubectl get pods
Unable to connect to the server: getting credentials: decoding stdout: couldn't get version/kind; json parse error: json: cannot unmarshal array into Go value of type struct { APIVersion string "json:\"apiVersion,omitempty\""; Kind string "json:\"kind,omitempty\"" }

We get the same response from every kubectl request.

Narrowed the problem down to the authentication step callout to aws eks get-token. With debug logs on, error occurs after the following lines:

host:sts.us-gov-west-1.amazonaws.com
x-k8s-aws-id:farpoint

host;x-k8s-aws-id
<earlier call stack>
<snip>
2023-03-03 13:43:15,726 - MainThread - botocore.auth - DEBUG - StringToSign:
AWS4-HMAC-SHA256
20230303T214315Z
20230303/us-gov-west-1/sts/aws4_request
<snip>
2023-03-03 13:43:15,726 - MainThread - botocore.auth - DEBUG - Signature:
<snip>

A few key details:

• Region: us-gov-west-1
• kubectl version:

Client Version: v1.25.6
Kustomize Version: v4.5.7
Server Version: v1.25.6-eks-48e63af

• Clusters are EKS running 1.25 and provisioned with eksctl
• We use aws sso for account auth and token generation

Confirmed a few things:

• Does not happen when using fixed IAM credentials, either via aws or aws-iam-authenticator
• Does not happen for cli <=2.10.3

I can't seem to get any more debug logs so I can't dig further, but feels like a malformed response somewhere? Naked aws eks get-token seems to return expected results.

Expected Behavior

kubectl command returns data

Current Behavior

kubectl get pods
Unable to connect to the server: getting credentials: decoding stdout: couldn't get version/kind; json parse error: json: cannot unmarshal array into Go value of type struct { APIVersion string "json:\"apiVersion,omitempty\""; Kind string "json:\"kind,omitempty\"" }

Happens for all subcommands checked (that interact with the server)

Reproduction Steps

Needs:

• EKS cluster 1.25 in Gov-Cloud region
• AWS SSO setup and used as login method

Doesn't work:

asdf shell awscli 2.10.4/2.11.0/latest
kubectl X
Unable to connect to the server: getting credentials: decoding stdout: couldn't get version/kind; json parse error: json: cannot unmarshal array into Go value of type struct { APIVersion string "json:\"apiVersion,omitempty\""; Kind string "json:\"kind,omitempty\"" }

Does work:

asdf shell awscli 2.10.3 (or before)
kubectl X
<expected output>

Possible Solution

Smells like a malformed HTML response is trying to get unmarshalled (just to my 👃 )
Downgrading to 2.10.3 or earlier is a viable workaround for now

Additional Information/Context

AWS SSO connected to Azure AD IdP

CLI version used

2.10.4+

Environment details (OS name and version, etc.)

macOS 13.2.1, kubectl and awscli via homebrew

[psemeniuk]

It seems that after 2.10.3 awscli relies on configured output format. Before it was json despite configured format:

aws --version; aws --region us-east-1 eks get-token --cluster-name REDACTED --role REDACTED aws-cli/2.10.3 Python/3.11.2 Darwin/21.6.0 source/arm64 prompt/off {"kind": "ExecCredential", "apiVersion": "client.authentication.k8s.io/v1beta1", "spec": {}, "status": {"expirationTimestamp": "2023-03-06T12:25:46Z", "token": "REDACTED"}}
vs
aws --version; aws --region us-east-1 eks get-token --cluster-name REDACTED --role REDACTED aws-cli/2.11.0 Python/3.11.2 Darwin/21.6.0 source/arm64 prompt/off client.authentication.k8s.io/v1beta1 ExecCredential STATUS 2023-03-06T12:28:05Z REDACTED

(i have configured text)

as a workaround you can set AWS_DEFAULT_OUTPUT env variable ( to json:

export AWS_DEFAULT_OUTPUT=json; aws --version; aws --region us-east-1 eks get-token --cluster-name REDACTED --role REDACTEED aws-cli/2.11.0 Python/3.11.2 Darwin/21.6.0 source/arm64 prompt/off { "kind": "ExecCredential", "apiVersion": "client.authentication.k8s.io/v1beta1", "spec": {}, "status": { "expirationTimestamp": "2023-03-06T12:29:08Z", "token": "REDACTED" } }

You can embed this variable by editing kubeconfig file.

[tim-finnigan]

This issue should be fixed via the PR linked above (#7726). Those changes will be available in the next CLI release.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.

[timfallmk]

Correct me if I'm wrong, but this would not fix the issue for current users (unless they (re)generated kubeconfig entry using the update-kubeconfig command.

[psemeniuk]

As i see from PR description, it just restores previous behaviour (before 2.10.4), so it should work out of the box.

[timfallmk]

Maybe I'm just misreading it. I can always re-open after release if it still happens. Thanks!

[tim-finnigan]

The fixed version for v1 will be out today and for v2 tomorrow. You will need to regenerate the config with update-kubeconfig again.

[timfallmk]

I'm not sure this is a complete fix then. New (greenfield) users would be covered, but anyone would else would still have this issue. Since the error message doesn't give any hints on cause they would have to search around, find this issue, and then apply fixes themselves. Unless I'm misunderstanding something.

I would classify this as still open.

Suggestion: Ignore user-set formatting variables when not in an interactive tty?

[tim-finnigan]

I'm not sure about that but recommend leaving feedback on that recent PR (#7726) to get visibility on your concerns from those who have directly been working on this. I can reference your most recent comment there.

[yuxiang-zhang]

I'm not sure this is a complete fix then. New (greenfield) users would be covered, but anyone would else would still have this issue. Since the error message doesn't give any hints on cause they would have to search around, find this issue, and then apply fixes themselves. Unless I'm misunderstanding something.

I would classify this as still open.

Suggestion: Ignore user-set formatting variables when not in an interactive tty?

The fix involves (1) getting the format from the flag --output before trying environment variables, and then (2) setting the flag to json in the generated config.

The behaviour before 2.10.4 was to use a default json when no flag was set, but since 2.10.4 (where --output support was added for eks get-token) the format was only set based on environment variable.

Yes the config file needs to be re-generated to add the flag --output json while getting user entries.

[timfallmk]

I still feel like this would leave most users out (given kubeconfig files are not touched frequently), but maybe adding something to the release notes about needing regeneration would be enough?