[EKS] [request]: Update EKS Kubernetes 1.21 release notes about a breaking change due to BoundServiceAccountTokenVolume

[anusha-ragunathan]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request
Accurately update EKS Kubernetes 1.21 release notes about the breaking change related to BoundServiceAccountTokenVolume.

Which service(s) is this request for?
EKS

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?

According to the EKS release notes for Kubernetes 1.21, BoundServiceAccountTokenVolume graduated to beta and is enabled by default. There's also mention of EKS adding an additional 90 days towards token expiration. This is all good. So we went ahead and updated our EKS clusters to 1.21, planning to update our applications in the next 90 days to the new SDKs. However, this plan doesn't work, because there's a breaking change in Kubernetes 1.21 around the struct of JWT tokens. Unrelated to the token expiration, the structure of JWT tokens has changed in 1.21 compared to 1.20.

Details of the change in service account token structure is clearly explained by this VMware Knowledge Base article:
https://kb.vmware.com/s/article/87079. We are experiencing the exact same issue. Specific snippets of interest from the article:

Here is what a ServiceAccount token looks like before the Bound Service Account Token Volume feature:
 
$ export TOKEN_BEFORE=$(kubectl exec -it deploy/test-token -n default -- cat /var/run/secrets/kubernetes.io/serviceaccount/token)
$ jwt $TOKEN_BEFORE
<<id_token redacted >>
To verify on jwt.io:
✻ Header
{
  "alg": "RS256",
  "kid": "5wZc-b_QlwN87zyYCAD3HVsxNJk2PRiXN1WfU0avdS4"
}
✻ Payload
{
  "iss": "kubernetes/serviceaccount",
  "kubernetes.io/serviceaccount/namespace": "default",
  "kubernetes.io/serviceaccount/secret.name": "test-token-m4pn6",
  "kubernetes.io/serviceaccount/service-account.name": "token-test",
  "kubernetes.io/serviceaccount/service-account.uid": "6a11d4bc-1a44-45c9-b025-4ba51df8dbcd",
  "sub": "system:serviceaccount:default:token-test"
}
 
Here is what a ServiceAccount token looks like with the Bound Service Account Token Volume feature (staring in Kubernetes 1.21):
 
$ export TOKEN_AFTER=$(kubectl exec -it deploy/tkgi-test-token -n default -- cat /var/run/secrets/kubernetes.io/serviceaccount/token)
$ jwt $TOKEN_AFTER
<<id_token redacted >>
To verify on jwt.io:
✻ Header
{
  "alg": "RS256",
  "kid": "wa08W8jGSrAi_df1T8vSM5X5CUch1MD9vB2qBv38C8Q"
}
✻ Payload
{
  "aud": [
  ],
  "exp": 1670718570,
  "iat": 1639182570,
  "kubernetes.io": {
    "namespace": "default",
    "pod": {
      "name": "tkgi-test-token-f9b48bbc8-4kxvm",
      "uid": "6d5119e8-8b85-41c0-8c13-a86f104b0990"
    },
    "serviceaccount": {
      "name": "tkgi-test",
      "uid": "10af3f93-ca29-4704-b888-bc7599bf0726"
    },
    "warnafter": 1639186177
  },
  "nbf": 1639182570,
  "sub": "system:serviceaccount:default:tkgi-test"
}

As a result, applications that read the serviceaccount token are breaking. So far, our dex containers are breaking and this seems to be a popular issue that the dex community is running into: dexidp/dex#2082

Are you currently working around this issue?
We are planning to stop the upgrade to 1.21 due to this breaking change.

Additional context
It would have been immensely helpful if we had clear release notes around the change in service account token structure.
That way, we could have planned to migrate our applications to the new SDKs and then performed the upgrades. This breaking change should get a lot more documentation and awareness. Hopefully, the release notes can be updated in a timely manner now, so that other EKS customers dont run into the same pain that we faced.

Attachments
If you think you might have additional information that you'd like to include via an attachment, please do - we'll take a look. (Remember to remove any personally-identifiable information.)