v23.01.1: Error: container has runAsNonRoot and image will run as root #817

temirg · 2023-03-30T14:53:23Z

Describe the bug
Upgrade of trident in Rancher UI via helmchart from 22.10.0 to 23.01.1 failed

Environment
Rancher 2.7.1, RKE2 v1.24.9+rke2r2

Trident version: 23.01.1
Trident installation flags used:
** yaml **
affinity: {}
deploymentAnnotations: {}
imagePullPolicy: IfNotPresent
imagePullSecrets: null
imageRegistry: registry.k8s.io/sig-storage
kubeletDir: ''
nodeSelector: {}
operatorDebug: true
operatorImage: docker.io/netapp/trident-operator:23.01.1
operatorImageTag: ''
podAnnotations: {}
tolerations: null
tridentAutosupportImage: docker.io/netapp/trident-autosupport:23.01
tridentAutosupportImageTag: '23.01'
tridentAutosupportProxy: ''
tridentDebug: false
tridentEnableNodePrep: false
tridentHttpRequestTimeout: 90s
tridentIPv6: false
tridentImage: docker.io/netapp/trident:23.01.1
tridentImageTag: ''
tridentK8sTimeout: 0
tridentLogFormat: text
tridentProbePort: ''
tridentSilenceAutosupport: false
tridentSkipK8sVersionCheck: false
windows: false
** yaml end **
Container runtime: containerd
Kubernetes version: v1.24.9+rke2r2
Kubernetes orchestrator: Rancher v2.7.1
OS: SLES 15.4

To Reproduce
Steps to reproduce the behavior:

push images to local registry, upload trident helmchart 23.01.1 to registry
Rancher => downstream cluster (explorer) => Apps => installed Apps => trident => Upgrade/Edit => choose 23.01.1 => apply using YAML above

Expected behavior
All trident pods run without errors

Additional Events:
Type Reason Age " ---- ------ ---- Normal Scheduled 37s Normal Pulling 37s Normal Pulled 35s Normal Pulling 35s Normal Pulling 34s Normal Pulled 34s Normal Pulling 33s Normal Pulled 33s Normal Pulled 32s Normal Pulling 32s Warning Failed 32s Warning Failed 31s Normal Pulled 31s Normal Pulling 31s Normal Pulled 31s Warning Failed 31s Warning Failed Warning Failed Warning Failed Normal Pulled 30s Normal Pulled 30s Normal Pulled 30s Normal Pulled 30s context => Error:
From Message
---- -------"
default-scheduler Successfully assigned trident/trident-controller-89d5d9c5f-pn7tr to worker-xxx
kubelet Pulling image "docker.io/netapp/trident:23.01.1"
kubelet Successfully pulled image "docker.io/netapp/trident:23.01.1" in 1.634133777s
kubelet Pulling image "docker.io/netapp/trident-autosupport:23.01"
kubelet Pulling image "registry.k8s.io/sig-storage/csi-provisioner:v3.4.0"
kubelet Successfully pulled image "docker.io/netapp/trident-autosupport:23.01" in 1.048826948s
kubelet Pulling image "registry.k8s.io/sig-storage/csi-attacher:v4.1.0"
kubelet Successfully pulled image "registry.k8s.io/sig-storage/csi-provisioner:v3.4.0" in 759.625294ms
kubelet Successfully pulled image "registry.k8s.io/sig-storage/csi-attacher:v4.1.0" in 881.271892ms
kubelet Pulling image "registry.k8s.io/sig-storage/csi-resizer:v1.7.0"
kubelet Error: container has runAsNonRoot and image will run as root (pod: "trident-controller-89d5d9c5f-pn7tr_trident(6e9ac407-9040-4af6-ae1e-eea99b92d08e)", container: csi-attacher)
kubelet Error: container has runAsNonRoot and image will run as root (pod: "trident-controller-89d5d9c5f-pn7tr_trident(6e9ac407-9040-4af6-ae1e-eea99b92d08e)", container: csi-resizer)
kubelet Successfully pulled image "registry.k8s.io/sig-storage/csi-resizer:v1.7.0" in 819.988585ms
kubelet Pulling image "registry.k8s.io/sig-storage/csi-snapshotter:v6.2.1"
kubelet Successfully pulled image "registry.k8s.io/sig-storage/csi-snapshotter:v6.2.1" in 812.559916ms
kubelet Error: container has runAsNonRoot and image will run as root (pod: "trident-controller-89d5d9c5f-pn7tr_trident(6e9ac407-9040-4af6-ae1e-eea99b92d08e)", container: csi-snapshotter)
30s (x2 over 35s) kubelet Error: container has runAsNonRoot and image will run as root (pod: "trident-controller-89d5d9c5f-pn7tr_trident(6e9ac407-9040-4af6-ae1e-eea99b92d08e)", container: trident-main)
30s (x2 over 33s) kubelet Error: container has runAsNonRoot and image will run as root (pod: "trident-controller-89d5d9c5f-pn7tr_trident(6e9ac407-9040-4af6-ae1e-eea99b92d08e)", container: csi-provisioner)
30s (x2 over 34s) kubelet Error: container has runAsNonRoot and image will run as root (pod: "trident-controller-89d5d9c5f-pn7tr_trident(6e9ac407-9040-4af6-ae1e-eea99b92d08e)", container: trident-autosupport)
kubelet Container image "docker.io/netapp/trident:23.01.1" already present on machine
kubelet Container image "docker.io/netapp/trident-autosupport:23.01" already present on machine
kubelet Container image "registry.k8s.io/sig-storage/csi-provisioner:v3.4.0" already present on machine
kubelet (combined from similar events): Container image "registry.k8s.io/sig-storage/csi-attacher:v4.1.0" already present on machine

temirg · 2023-03-31T07:03:41Z

Hello,

attached is a diff/compare between v22.10.0 and v23.01.1 helmcharts:
charts-diff_v22.10.0_vs_v23.01.1.txt

v23.01.1 - roles/rolebindings added before clusteroles/clusterrolebindings.

Regards, temir.

temirg · 2023-04-03T08:05:22Z

Possibly very important info: all rke2 cluster with rke2 (k8s) <1.25.x have "cis-profile: 1.6" configured.

In the future, when rke2 is updated to version 1.25.x and newer, "cis-profile: 1.23" will be used.

Regards, temir.

gnarl · 2023-04-03T13:23:38Z

Hi @temirg,

We haven't tested Trident against RKE 2 and it seems that you are the first customer reporting an issue while using RKE 2. Please open a NetApp Support case for this issue so that additional information can be collected. This will help us to address this issue faster than if we try to just collect the information via GitHub Issues.

temirg · 2023-04-03T15:44:39Z

Hello @gnarl,

Netapp Case with prio 2 is closed today, because the bugfix wills take a while.
Should the Issue be closed too?

gnarl · 2023-04-03T16:40:29Z

Hi @temirg,

I found out about the NetApp support case a bit ago. We are working on reopening the support case.

antwynne · 2023-04-04T05:05:46Z

The case has now been re-opened

antwynne · 2023-04-17T05:21:04Z

Hi @gnarl

Do you need any more information about this issue. Please let me know if you do.

mdekoster · 2023-05-15T08:55:22Z

Hi,

We have the same issue with the Trident Operator (on k8s version 1.24). The Operator creates the deployment for the trident-controller, but it does not specify the securityContext on pod level and container level in the deployment. If an another PSP is specified in the cluster ~~with a alphabetic name for the ClusterRoleBinding that is before the ClusterRoleBinding fleet-controller~~ with least permissions, the mutating admission psp controller will add securityContexts that do not conform the persmissons the pod/container(s) will need.

antwynne · 2023-05-16T07:17:24Z

@mdekoster
we have the output of kubectl describe ns trident:
@test:~> kubectl describe ns trident
Name: trident
Labels: kubernetes.io/metadata.name=trident

pod-security.kubernetes.io/enforce=privileged
Annotations: cattle.io/status:

{"Conditions":[{"Type":"ResourceQuotaInit","Status":"True","Message":"","LastUpdateTime":"2022-01-28T11:03:35Z"},{"Type":"InitialRol
esPopu...
lifecycle.cattle.io/create.namespace-auth: true
Status: Active

Resource Quotas
Name:
trident-csi
Resource Used Hard

No LimitRange resource.

mdekoster · 2023-05-16T07:59:31Z

@antwynne that's for the Pod Security Admission (PSA) in k8s 1.25+. We are still on 1.24 and use the (deprecated) Pod Security Policy feature.

But, you still have to specify the needed securityContext in the pod specs of the deployment, daemonset or statefulset. The triden-operator does not specify the securityContext in the trident-controller deployment.

temirg · 2023-06-06T13:25:54Z

@antwynne / @gnarl
there is a new trident release available: v23.04: https://github.com/NetApp/trident/releases/tag/v23.04.0
Does the new release solve the problem of version 23.01 with RKE2?

Regards, temirg.

zlmitchell · 2023-06-07T04:06:48Z

I just attempted the same with 23.04 and still have the same issue. RKE2 1.24.12+rke2r1, cis 1.6

Error: container has runAsNonRoot and image will run as root

zlmitchell · 2023-06-07T04:54:01Z

It appears that the Trident-Controller does not have a securityContext set, and is not configurable via the helm chart. Updating the deployment for the trident-controller and adding follow got it running. But as the chart/operator has no way to configure this we should probably get this bug report submitted.

securityContext:
  runAsNonRoot: false

antwynne · 2023-06-07T06:59:19Z

@zlmitchell
In helm chart values exists a line with "deploymentAnnotations: {}" .

There are two deployments in total: trident-csi and trident-operator.
What is the correct syntax to pass the rigth securityContext to the trident-csi?

== As a sample ==
deploymentAnnotations:
metadata:
name: trident-csi
spec:
spec:
securityContext:
runAsNonRoot: false

temirg · 2023-06-07T10:43:20Z

@zlmitchell The workaround works, but: is it safe for production?
Or does the orchestrator eventually realize that the deployment has been changed and reset it again?

zlmitchell · 2023-06-07T17:30:17Z

If you are running PSA in your RKE2 cluster you will also need to add the following to the trident-operator ClusterRole

  - apiGroups:
      - management.cattle.io
    resources:
      - projects
    verbs:
      - updatepsa

temirg · 2023-06-13T15:43:55Z

Hello all,

the workaround with deployment edit works fine.
Issue wills be closed as solved by workaround.

Many thanks to all involved!

Best Regards, temirg.

gnarl · 2023-07-14T17:52:04Z

This issue is also addressed with commit 6d30a16 and will be included in the Trident 23.07 release.

temirg added the bug label Mar 30, 2023

temirg closed this as completed Jun 13, 2023

alloydsa mentioned this issue Jul 23, 2024

Capability SYS_ADMIN is published as removed in changelog, but is still in factory method #862

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

v23.01.1: Error: container has runAsNonRoot and image will run as root #817

v23.01.1: Error: container has runAsNonRoot and image will run as root #817

temirg commented Mar 30, 2023

temirg commented Mar 31, 2023

temirg commented Apr 3, 2023 •

edited

Loading

gnarl commented Apr 3, 2023

temirg commented Apr 3, 2023

gnarl commented Apr 3, 2023

antwynne commented Apr 4, 2023

antwynne commented Apr 17, 2023

mdekoster commented May 15, 2023 •

edited

Loading

antwynne commented May 16, 2023

mdekoster commented May 16, 2023 •

edited

Loading

temirg commented Jun 6, 2023 •

edited

Loading

zlmitchell commented Jun 7, 2023

zlmitchell commented Jun 7, 2023

antwynne commented Jun 7, 2023

temirg commented Jun 7, 2023

zlmitchell commented Jun 7, 2023

temirg commented Jun 13, 2023

gnarl commented Jul 14, 2023

v23.01.1: Error: container has runAsNonRoot and image will run as root #817

v23.01.1: Error: container has runAsNonRoot and image will run as root #817

Comments

temirg commented Mar 30, 2023

temirg commented Mar 31, 2023

temirg commented Apr 3, 2023 • edited Loading

gnarl commented Apr 3, 2023

temirg commented Apr 3, 2023

gnarl commented Apr 3, 2023

antwynne commented Apr 4, 2023

antwynne commented Apr 17, 2023

mdekoster commented May 15, 2023 • edited Loading

antwynne commented May 16, 2023

mdekoster commented May 16, 2023 • edited Loading

temirg commented Jun 6, 2023 • edited Loading

zlmitchell commented Jun 7, 2023

zlmitchell commented Jun 7, 2023

antwynne commented Jun 7, 2023

temirg commented Jun 7, 2023

zlmitchell commented Jun 7, 2023

temirg commented Jun 13, 2023

gnarl commented Jul 14, 2023

temirg commented Apr 3, 2023 •

edited

Loading

mdekoster commented May 15, 2023 •

edited

Loading

mdekoster commented May 16, 2023 •

edited

Loading

temirg commented Jun 6, 2023 •

edited

Loading