Kubernetes clusters should disable automounting API credentials

[SunilDSK]

What happened:

We deployed nginx ingress controller on AKS cluster. We need to disable automount serviceaccount token. We need to set this automountServiceAccountToken: false flag on pod spec, but, there is no parameter in to set this flag from values file.

What you expected to happen:

We want to disable automounting of serviceaccount token using automountServiceAccountToken: false flag. Please provide a way to set automountServiceAccountToken: false in pod spec.

NGINX Ingress controller version - 1.5.1

Kubernetes version - 1.24.6

Environment:

• Azure AKS:

• Basic cluster related info:

  • kubectl version -

Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.2", 
GitCommit:"5835544ca568b757a8ecae5c153f317e5736700e", GitTreeState:"clean", BuildDate:"2022-09-21T14:33:49Z", 
GoVersion:"go1.19.1", Compiler:"gc", Platform:"windows/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.6", 
GitCommit:"08d3594304660f86cfbd17bbb862041b4b75fe6c", GitTreeState:"clean", BuildDate:"2023-02-08T17:22:59Z", GoVersion:"go1.18.6", Compiler:"gc", Platform:"linux/amd64"}

• kubectl get nodes -o wide

NAME                              STATUS   ROLES   AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION     CONTAINER-RUNTIME
aks-apps-38117287-vmss000000      Ready    agent   16d   v1.24.6   11.1.2.95     <none>        Ubuntu 18.04.6 LTS   5.4.0-1103-azure   containerd://1.6.15+azure-1
aks-apps-38117287-vmss000001      Ready    agent   16d   v1.24.6   11.1.2.124    <none>        Ubuntu 18.04.6 LTS   5.4.0-1103-azure   containerd://1.6.15+azure-1
aks-default-70227024-vmss000000   Ready    agent   16d   v1.24.6   11.1.2.8      <none>        Ubuntu 18.04.6 LTS   5.4.0-1103-azure   containerd://1.6.15+azure-1
aks-default-70227024-vmss000001   Ready    agent   16d   v1.24.6   11.1.2.37     <none>        Ubuntu 18.04.6 LTS   5.4.0-1103-azure   containerd://1.6.15+azure-1
aks-default-70227024-vmss000002   Ready    agent   16d   v1.24.6   11.1.2.66     <none>        Ubuntu 18.04.6 LTS   5.4.0-1103-azure   containerd://1.6.15+azure-1

• How was the ingress-nginx-controller installed:
  • If helm was used then please show output of helm ls -A | grep -i ingress

NAME            NAMESPACE       REVISION        UPDATED                                 STATUS          CHART                   APP VERSION
ingress-nginx   tools           1               2023-02-27 13:50:38.035545 +0530 IST    deployed        ingress-nginx-4.4.0     1.5.1      

• If helm was used then please show output of helm -n tools get values ingress-nginx

USER-SUPPLIED VALUES:
controller:
  extraVolumeMounts:
  - mountPath: /mnt/secrets-store
    name: secrets-store-inline
    readOnly: true
  extraVolumes:
  - csi:
      driver: secrets-store.csi.k8s.io
      readOnly: true
      volumeAttributes:
        secretProviderClass: mytls
    name: secrets-store-inline
  replicaCount: 2
  service:
    annotations:
      service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path: /healthz
      service.beta.kubernetes.io/azure-load-balancer-resource-group: nonprod
    loadBalancerIP: XX.XX.XX.XX

• Current State of the controller:
  • kubectl describe ingressclasses

Name:         nginx
Labels:       app.kubernetes.io/component=controller  
              app.kubernetes.io/instance=ingress-nginx
              app.kubernetes.io/managed-by=Helm       
              app.kubernetes.io/name=ingress-nginx    
              app.kubernetes.io/part-of=ingress-nginx 
              app.kubernetes.io/version=1.5.1
              helm.sh/chart=ingress-nginx-4.4.0       
Annotations:  meta.helm.sh/release-name: ingress-nginx
              meta.helm.sh/release-namespace: tools   
Controller:   k8s.io/ingress

• kubectl -n tools get all -A -o wide

NAME                                            READY   STATUS    RESTARTS   AGE   IP           NODE                           NOMINATED NODE   READINESS GATES
pod/ingress-nginx-controller-59cffe56y9-gx5m4   1/1     Running   0          15d   11.1.2.124   aks-apps-38117287-vmss000001   <none>           <none>
pod/ingress-nginx-controller-59cffe56y9-tqxzc   1/1     Running   0          15d   11.1.2.95   aks-apps-38117287-vmss000000   <none>           <none>

NAME                                         TYPE           CLUSTER-IP    EXTERNAL-IP     PORT(S)                      AGE   SELECTOR
service/ingress-nginx-controller             LoadBalancer   11.1.65.249   xx.xx.xx.xx     80:30823/TCP,443:30969/TCP   15d   app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
service/ingress-nginx-controller-admission   ClusterIP      11.1.65.116   <none>          443/TCP                      15d   app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx

NAME                                       READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS   IMAGES                                                                                                                    SELECTOR
deployment.apps/ingress-nginx-controller   2/2     2            2           15d   controller   registry.k8s.io/ingress-nginx/controller:v1.5.1@sha256:4ba73c697770664c1e00e9f968de14e08f606ff961c76e5d7033a4a9c593c629   app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx

NAME                                                  DESIRED   CURRENT   READY   AGE   CONTAINERS   IMAGES                                                                                                                    SELECTOR
replicaset.apps/ingress-nginx-controller-59cffe56y9   2         2         2       15d   controller   registry.k8s.io/ingress-nginx/controller:v1.5.1@sha256:4ba73c697770664c1e00e9f968de14e08f606ff961c76e5d7033a4a9c593c629   app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx,pod-template-hash=59cffe56y9

• kubectl -n tools describe po ingress-nginx-controller-59cffe56y9-gx5m4

Name:             ingress-nginx-controller-59cffe56y9-gx5m4
Namespace:        tools
Priority:         0
Service Account:  ingress-nginx
Node:             aks-apps-38117287-vmss000001/10.0.0.124
Start Time:       Mon, 27 Feb 2023 13:51:37 +0530
Labels:           app.kubernetes.io/component=controller
                  app.kubernetes.io/instance=ingress-nginx
                  app.kubernetes.io/name=ingress-nginx
                  pod-template-hash=59cffe56y9
Annotations:      <none>
Status:           Running
IP:               11.1.2.124
IPs:
  IP:           11.1.2.124
Controlled By:  ReplicaSet/ingress-nginx-controller-59cffe56y9
Containers:
  controller:
    Container ID:  containerd://5d55b2eea9244115a9eb1c2774dd157758550f54bdca6efaba7a8da0d810d4b3
    Image:         registry.k8s.io/ingress-nginx/controller:v1.5.1@sha256:4ba73c697770664c1e00e9f968de14e08f606ff961c76e5d7033a4a9c593c629
    Image ID:      registry.k8s.io/ingress-nginx/controller@sha256:4ba73c697770664c1e00e9f968de14e08f606ff961c76e5d7033a4a9c593c629       
    Ports:         80/TCP, 443/TCP, 9443/TCP
    Host Ports:    0/TCP, 0/TCP, 0/TCP
    Args:
      /nginx-ingress-controller
      --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller
      --election-id=ingress-nginx-leader
      --controller-class=k8s.io/ingress-nginx
      --ingress-class=nginx
      --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
      --validating-webhook=:9443
      --validating-webhook-certificate=/usr/local/certificates/cert
      --validating-webhook-key=/usr/local/certificates/key
    State:          Running
      Started:      Mon, 27 Feb 2023 13:51:43 +0530
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:      100m
      memory:   90Mi
    Liveness:   http-get http://:8989/healthz delay=10s timeout=1s period=10s #success=1 #failure=5
    Readiness:  http-get http://:8989/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
    Environment:
      POD_NAME:       ingress-nginx-controller-59cffe56y9-gx5m4 (v1:metadata.name)
      POD_NAMESPACE:  tools (v1:metadata.namespace)
      LD_PRELOAD:     /usr/local/lib/libmimalloc.so
    Mounts:
      /mnt/secrets-store from secrets-store-inline (ro)
      /usr/local/certificates/ from webhook-cert (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-95hb7 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  webhook-cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  ingress-nginx-admission
    Optional:    false
  secrets-store-inline:
    Type:              CSI (a Container Storage Interface (CSI) volume source)
    Driver:            secrets-store.csi.k8s.io
    FSType:
    ReadOnly:          true
    VolumeAttributes:      secretProviderClass=mytls
  kube-api-access-95hb7:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              kubernetes.io/os=linux
Tolerations:                 node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason                  Age                  From                        Message
  ----    ------                  ----                 ----                        -------
  Normal  SecretRotationComplete  5s (x8731 over 12d)  csi-secrets-store-rotation  successfully rotated K8s secret mapstls

• kubectl -n tools describe svc ingress-nginx-controller

Name:                     ingress-nginx-controller
Namespace:                tools
Labels:                   app.kubernetes.io/component=controller
                          app.kubernetes.io/instance=ingress-nginx
                          app.kubernetes.io/managed-by=Helm
                          app.kubernetes.io/name=ingress-nginx
                          app.kubernetes.io/part-of=ingress-nginx
                          app.kubernetes.io/version=1.5.1
                          helm.sh/chart=ingress-nginx-4.4.0
Annotations:              meta.helm.sh/release-name: ingress-nginx
                          meta.helm.sh/release-namespace: tools
                          service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path: /healthz
                          service.beta.kubernetes.io/azure-load-balancer-resource-group: nonprod
Selector:                 app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
Type:                     LoadBalancer
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.0.65.249
IPs:                      10.0.65.249
IP:                       xx.xx.xx.xx
LoadBalancer Ingress:     xx.xx.xx.xx
Port:                     http  80/TCP
TargetPort:               http/TCP
NodePort:                 http  30823/TCP
Endpoints:                11.1.2.95:80,11.1.2.124:80
Port:                     https  443/TCP
TargetPort:               https/TCP
NodePort:                 https  30969/TCP
Endpoints:                11.1.2.95:443,11.1.2.124:443
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

[longwuyuan]

/remove-kind bug

[strongjz]

@SunilDSK The PR to add a value is straightforward, and there are many examples of how to add that looking at the current helm chart.

/triage accepted
/priority backlog
/kind feature
/area helm
/help-wanted
/good-first-issue

[k8s-ci-robot]

@strongjz:
This request has been marked as suitable for new contributors.

Guidelines

Please ensure that the issue body includes answers to the following questions:

• Why are we solving this issue?
• To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points?
• Does this issue have zero to low barrier of entry?
• How can the assignee reach out to you for help?

For more details on the requirements of such an issue, please see here and ensure that they are met.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-good-first-issue command.

In response to this:

@SunilDSK The PR to add a value is straightforward, and there are many examples of how to add that looking at the current helm chart.

/triage accepted
/priority backlog
/kind feature
/area helm
/help-wanted
/good-first-issue

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[KR411-prog]

@strongjz could you review my PR?

[strongjz]

If you disable auto mount are you adding the token a different way? The controller still needs to access the k9s api.

[github-actions]

This is stale, but we won't close it automatically, just bare in mind the maintainers may be busy with other tasks and will reach your issue ASAP. If you have any question or request to prioritize this, please reach #ingress-nginx-dev on Kubernetes Slack.

[rajagopalsom]

@longwuyuan @strongjz kubernetes documentation link provides option to opt out of automounting which is part of ServiceAccount template file. In the same link there is a option to manually create the token instead of automount. Guess similar option is expected by the OP as part of ingress-nginx Service account config.

[OmarHawk]

@longwuyuan @strongjz kubernetes documentation link provides option to opt out of automounting which is part of ServiceAccount template file. In the same link there is a option to manually create the token instead of automount. Guess similar option is expected by the OP as part of ingress-nginx Service account config.

yep, that is what is required so that the controller still has the token available through manual mount and e.g. mount mode can be adjusted so that there are no write permissions on that token (default mount mode is 644, but 444 is usually sufficient).
Also it would help resolving issues in corresponding policies from e.g. MS Defender for Cloud without having to whitelist the whole namespace...

[Saurabh12p]

@longwuyuan @strongjz: @rajagopalsom @SunilDSK
To work with the Ingress controller pod while setting 'automountServiceAccountToken' to 'false,' it is necessary to add a created service account token and mount it within the pod.
My question is, when will this flag be added under 'pod.spec'? Additionally, is there a need for a service token file to be included as part of the Ingress controller chart?
Summary what I did:

1. Added automountServiceAccountToken: false under pod.spec in both ingress controller and default backend pod
   
2. Created service token file:

apiVersion: v1
kind: Secret
metadata:
  name: {{ template "ingress-nginx.serviceAccountName" . }}-token
  annotations:
    kubernetes.io/service-account.name: {{ template "ingress-nginx.serviceAccountName" . }}

type: kubernetes.io/service-account-token
3. Mounted this secret on ingress controller pod

volumeMounts:
  - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
    name:<release-name>-ingress-nginx-token

volumes:
- name:<release-name>-ingress-nginx-token
        projected:
          defaultMode: 420
          sources:
          - serviceAccountToken:
              expirationSeconds: 3607
              path: token
          - secret:
              items:
              - key: ca.crt
                path: ca.crt
              name: release-name-ingress-nginx-token
          - downwardAPI:
              items:
              - fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.namespace
                path: namespace

[Gacko]

ATM there's a automountServiceAccountToken value on the ServiceAccount level. According to the Kubernetes docs, setting this to false already disables auto-mounting the secret in the pod. So no pod-level flag required, the desired behavior has already been implemented.

/close

[k8s-ci-robot]

@Gacko: Closing this issue.

In response to this:

ATM there's a automountServiceAccountToken value on the ServiceAccount level. According to the Kubernetes docs, setting this to false already disables auto-mounting the secret in the pod. So no pod-level flag required, the desired behavior has already been implemented.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[fuzail26]

@longwuyuan @strongjz: @rajagopalsom @SunilDSK To work with the Ingress controller pod while setting 'automountServiceAccountToken' to 'false,' it is necessary to add a created service account token and mount it within the pod. My question is, when will this flag be added under 'pod.spec'? Additionally, is there a need for a service token file to be included as part of the Ingress controller chart? Summary what I did:

1. Added automountServiceAccountToken: false under pod.spec in both ingress controller and default backend pod
   
2. Created service token file:

apiVersion: v1
kind: Secret
metadata:
  name: {{ template "ingress-nginx.serviceAccountName" . }}-token
  annotations:
    kubernetes.io/service-account.name: {{ template "ingress-nginx.serviceAccountName" . }}

type: kubernetes.io/service-account-token 3. Mounted this secret on ingress controller pod

volumeMounts:
  - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
    name:<release-name>-ingress-nginx-token

volumes:
- name:<release-name>-ingress-nginx-token
        projected:
          defaultMode: 420
          sources:
          - serviceAccountToken:
              expirationSeconds: 3607
              path: token
          - secret:
              items:
              - key: ca.crt
                path: ca.crt
              name: release-name-ingress-nginx-token
          - downwardAPI:
              items:
              - fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.namespace
                path: namespace

Is this solution working ?? I too face similar issue on my AKS with nginx ingress controller

[chandlerkent]

ATM there's a automountServiceAccountToken value on the ServiceAccount level. According to the Kubernetes docs, setting this to false already disables auto-mounting the secret in the pod. So no pod-level flag required, the desired behavior has already been implemented.

/close

@Gacko when automountServiceAccountToken is set to false on the ServiceAccount level, the ingress controller no longer functions:

-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       v1.5.1
  Build:         d003aae913cc25f375deb74f898c7f3c65c06f05
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.21.6

-------------------------------------------------------------------------------

W0221 18:44:07.278068       7 client_config.go:617] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
W0221 18:44:07.278138       7 client_config.go:622] error creating inClusterConfig, falling back to default config: open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
F0221 18:44:07.278251       7 main.go:267] Error while initiating a connection to the Kubernetes API server. This could mean the cluster is misconfigured (e.g. it has invalid API server certificates or Service Accounts configuration). Reason: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable
Refer to the troubleshooting guide for more information: https://kubernetes.github.io/ingress-nginx/troubleshooting/

What is the expectation on how users can make this work and disable auto-mounting the service account token?

[Gacko]

Apart from the fact that you're running a quite old version of the controller, Ingress NGINX needs to communicate with the API server and therefore requires a service account token. So you either auto-mount it or have it mounted manually, but you can not run Ingress NGINX without API access.

[chandlerkent]

@Gacko yes that makes sense, but I guess my question is what is the point of allowing that to be specified in the Helm chart if the controller does not work when it is set?

Thanks for the reminder on upgrading the ingress. Will do!

[Gacko]

Well, that's also something I can not explain. I only found the PR introducing the flag and it doesn't have any reasoning.

[Gacko]

Apparently you can also manually mount the service account token and other related information with custom settings. So in case one wants to do so using the extra volume mounts in the chart, they would need to disable the auto-mounting.

[chandlerkent]

Yeah that makes sense. However I am experiencing the same issue as the original issue on AKS with Azure Policy and the only way to satisfy the policy would be if the pod itself had auto-mounting disabled, not the ServiceAccount.

For now I’ve excluded ingress-nginx from Azure Policy as I see no way with the helm chart to satisfy its requirements.

Thanks for the help!

[Gacko]

That's an interesting insight, thank you for sharing this! Actually I don't think you can simply disable that and guess this Azure Policy in particular is targeted for workload not requiring API server access, which probably matches the most of their customer workloads.

[germanattanasio]

For those that came here with the Azure Policy complaining about automount token....

Update your Helm chart values to include the following:

# Disable automount for the ServiceAccount
serviceAccount:
  automountServiceAccountToken: false

controller:  
  # Add extra volumes to mount the ServiceAccount token
  extraVolumes:
    - name: serviceaccount-token
      projected:
        defaultMode: 0444
        sources:
          - serviceAccountToken:
              expirationSeconds: 3607
              path: token
          - configMap:
              name: kube-root-ca.crt
              items:
                - key: ca.crt
                  path: ca.crt
          - downwardAPI:
              items:
                - path: namespace
                  fieldRef:
                    apiVersion: v1
                    fieldPath: metadata.namespace
  # Mount the volume to the appropriate directory
  extraVolumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: serviceaccount-token
      readOnly: true

Notes:

This configuration ensures the ServiceAccount token is not automounted, and manual mounting of the token allows the controller to maintain API server access with restricted permissions (defaultMode: 0444).

[WaitingForGuacamole]

This is all well and good for user deployments. Problem is, DfC complains about NMI and MIC pods in the default namespace. Are we supposed to be patching Microsoft-deployed pods now?