Can't integrate rocks to securityContext.runAsNonRoot: true

[misohu]

Bug Description

While working on net-istio-webhook extension rock for knative we had encountered a problem where we can't run rocks in securityContext.runAsNonRoot: true Kubernetes deployment.

Rocks like net-istio-webhook are deployed as part of KnativeServing CR.

This is how securityContext for the webhook deployment looks like.

...
securityContext:
  allowPrivilegeEscalation: false
  capabilities:
    drop:
    - ALL
  runAsNonRoot: true
  seccompProfile:
    type: RuntimeDefault
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
...

This problem is known problem in rockcraft project tracked here: canonical/rockcraft#659

To Reproduce

Use (or build) the rock presented in this PR.

Use the rock in the config file for knative-serving. One way to do it is just change the linked field in the config before deploying. Simplest way is to run bundle integration test from the root folder of knative-operators.

tox -e integration -- --model kubeflow --keep-models -vv -s

After some time the test should be green. After that you can check the pods in the knnative-serving namespace

kubectl get po -n knative-serving

The pod for net-istio-webhook is in ConfigurationError state. You can inspect the events of the pod

kubectl describe pod net-istio-webhook-<pod-id> -n knative-serving

You should see simillar error message.

Events:
  Type     Reason     Age               From               Message
  ----     ------     ----              ----               -------
  Normal   Scheduled  32s               default-scheduler  Successfully assigned knative-serving/net-istio-webhook-86cddb56c6-b89bl to hyrule
  Normal   Pulling    32s               kubelet            Pulling image "misohu/net-istio-webhook:1.12.12"
  Normal   Pulled     28s               kubelet            Successfully pulled image "misohu/net-istio-webhook:1.12.12" in 3.804s (3.804s including waiting)
  Warning  Failed     5s (x4 over 28s)  kubelet            Error: container has runAsNonRoot and image has non-numeric user (_daemon_), cannot verify user is non-root (pod: "net-istio-webhook-86cddb56c6-b89bl_knative-serving(7ceb92aa-cd32-49e8-9fb0-a83b1987779d)", container: webhook)
  Normal   Pulled     5s (x3 over 27s)  kubelet            Container image "misohu/net-istio-webhook:1.12.12" already present on machine

Environment

Juju: 3.4.5
Kubernetes: 1.29

Relevant Log Output

Events:
  Type     Reason     Age               From               Message
  ----     ------     ----              ----               -------
  Normal   Scheduled  32s               default-scheduler  Successfully assigned knative-serving/net-istio-webhook-86cddb56c6-b89bl to hyrule
  Normal   Pulling    32s               kubelet            Pulling image "misohu/net-istio-webhook:1.12.12"
  Normal   Pulled     28s               kubelet            Successfully pulled image "misohu/net-istio-webhook:1.12.12" in 3.804s (3.804s including waiting)
  Warning  Failed     5s (x4 over 28s)  kubelet            Error: container has runAsNonRoot and image has non-numeric user (_daemon_), cannot verify user is non-root (pod: "net-istio-webhook-86cddb56c6-b89bl_knative-serving(7ceb92aa-cd32-49e8-9fb0-a83b1987779d)", container: webhook)
  Normal   Pulled     5s (x3 over 27s)  kubelet            Container image "misohu/net-istio-webhook:1.12.12" already present on machine

Additional Context

No response

[syncronize-issues-to-jira]

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-6537.

This message was autogenerated

[misohu]

I have also explored the option to disable the runAsNonRoot option for net-istio-weebhook deployment in KnativeServing CR by adding the:

workloads:
    - name: net-istio-webhook
      podSpec:
        securityContext:
          runAsUser: 0

Sadly this option is not supported for net-istio workloads.

[mvlassis]

I encounter the same issue while integrating the net-istio-controller rock, see the logs:

Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  73s                default-scheduler  Successfully assigned knative-serving/net-istio-controller-5896777f8b-pkjcx to dev
  Normal   Pulling    73s                kubelet            Pulling image "mvlassis/net-istio-controller:v1.12.3"
  Normal   Pulled     68s                kubelet            Successfully pulled image "mvlassis/net-istio-controller:v1.12.3" in 5.012s (5.012s including waiting)
  Warning  Failed     12s (x6 over 68s)  kubelet            Error: container has runAsNonRoot and image has non-numeric user (_daemon_), cannot verify user is non-root (pod: "net-istio-controller-5896777f8b-pkjcx_knative-serving(f27196b4-4f99-4dcf-b234-9a587b905c8c)", container: controller)
  Normal   Pulled     12s (x5 over 68s)  kubelet            Container image "mvlassis/net-istio-controller:v1.12.3" already present on machine

[mvlassis]

Adding more context to this issue:

The problem occurs due to a known limitation of Pebble, described in canonical/pebble#462. In short, Pebble needs write access to the disk for storing information about it's current state.

Note that the above issue has been closed not because it now works on a readonly filesystem, but because it outputs an appropriate error message. A read-only filesystem still produces errors.

According to the Pebble team, needing write access seems to be a strong constraint for Pebble, and a workaround is not actively being worked on by the team.

[kimwnasptd]

To summarize the current state, initially we couldn't integrate rocks because of runAsNonRoot, as described above, but this got unblocked by the rockcraft team in canonical/rockcraft#659.

Although to include it we had to use latest rockcraft for some time canonical/knative-rocks#12.

---

The last comment above from @mvlassis is about readOnlyRootFilesystem and pebble, and we'll tackle it in #291