unable to get kubeconfig kubernetes.io/serviceaccount/token: permission denied

[kingdonb]

Maybe fixed by:

• add fsgroup for securityContext helm-controller#435
• add fsgroup for securityContext notification-controller#342

-- someone who has experienced this issue can test and let us know if this is what's needed.

Describe the bug

Helm Controller and Notification Controller were reported by a user on EKS to be in crashloopbackoff.

I am not sure why I haven't seen this issue on any of my clusters before, but I noticed that error often surfaces when fsGroup setting is missing from the securityContext and when I checked my deployments, I noticed those two controllers were the only ones that did not have this block in their deployment config:

      securityContext:
        fsGroup: 1337

Seems too congruous to be a coincidence! Should we add those in the config of helm-controller and notification-controller or are they omitted on purpose?

Steps to reproduce

I don't have details for a reproduction, I only know that this issue was reported by an EKS user who said they didn't do anything special to their cluster. (Thread: https://cloud-native.slack.com/archives/CLAJ40HV3/p1647269384516989)

We tried some things to bisect the issue, but I believe this is likely the problem.

Expected behavior

Not crashloopbackoff

Screenshots and recordings

No response

OS / Distro

EKS

Flux version

0.27.3

Flux check

N/A

Git provider

No response

Container Registry provider

No response

Additional context

I'm happy to submit the PRs for this if we're agreed this is what's needed (I'll start them now, so they can be merged straight away if that's the case).

Code of Conduct

• I agree to follow this project's Code of Conduct

[kingdonb]

It is unsure if there is an issue we still need to address after merging those two, based on this thread:

fluxcd/notification-controller#342 (comment)

[mikestef9]

setting fsGroup is only required for EKS 1.18 and earlier clusters https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-technical-overview.html#pod-configuration

[stefanprodan]

What about Fargate on EKS? Last time I tried it didn’t worked without an fsGroup.

[stefanprodan]

Closing this but feel free to reply here if this is still an issue.