[k8s 1.21] BoundServiceAccountTokenVolume refresh token

[mo-saeed]

Kubernetes version 1.21 graduated BoundServiceAccountTokenVolume feature [1] to beta and enabled it by default. This feature improves security of service account tokens by requiring a one hour expiry time, over the previous default of no expiration. This means that applications that do not refetch service account tokens periodically will receive an HTTP 401 unauthorized error response on requests to Kubernetes API server with expired tokens.

In our kubernetes audit logs we see that kubernetes dashboard still using stale tokens

"authentication.k8s.io/stale-token":"subject: system:serviceaccount:kube-infrastructure:helm-operator, seconds after warning threshold: 1200715"

As per my understanding, that should be fixed by upgrading kubernetes client to latest release Go v0.15.7 and later

To Reproduce
install helm-operator in kubernetes cluster with version >= 1.21 and check the kubernetes audit logs for stale-token

Expected behavior

A clear and concise description of what you expected to happen.

Additional context

• Helm Operator version: latest
• Kubernetes version: 1.22

As per my understanding, that should be fixed by upgrading kubernetes client SDK to latest release Go v0.15.7 and later

[pjbgf]

This is a duplicate of fluxcd/flux#3610. I would recommend tuning in that thread for latest information around the issue.

[pjbgf]

We have recently upgraded most dependencies (including AWS SDK) and shall release a new version in the coming days. That is mostly to mitigate security vulnerabilities, but we are not completely sure that would suffice to fix this problem.

A workaround for this issue is to force the pod to be restarted before the token expiration period (in EKS' case that is 90 days). That could be achieved with a Kubernetes CronJob or by simply redeploying Flux within that time frame.

Unfortunately, we won't be able to fix the root cause of this issue as Flux v1 is in Migration and security support only. We recommend users to migrate to Flux 2 at their earliest convenience so they don't encounter other issues as newer Kubernetes versions may degrade Flux v1 operations.

More information about the Flux 2 transition timetable can be found at: https://fluxcd.io/docs/migration/timetable/.

[kingdonb]

It looks like there's some complicated nuance to this issue that explains why it did not solve anything to upgrade the client-go package: fluxcd/helm-controller#480

That being said, it should be noted that this issue has been resolved in Helm Controller since the 0.31 Flux v2 release in May.