cannot use Bidirectional mountPropagation

[junhuihuang]

What happened:
Error from server: error when creating "helloworld.yaml": admission webhook "validatejob.volcano.sh" denied the request: spec.task[0].template.spec.containers[1].volumeMounts.mountPropagation: Forbidden: Bidirectional mount propagation is available only to privileged containers.
What you expected to happen:
Can be created
How to reproduce it (as minimally and precisely as possible):

securityContext:
  privileged: true
volumeMounts:
  mountPropagation: Bidirectional

Anything else we need to know?:
pkg/webhooks/admission/jobs/validate/admit_job.go#validateTaskTemplate() reset privileged to nil

// Skip verify container SecurityContex.Privileged as it depends on
// the kube-apiserver `allow-privileged` flag.
for i, container := range coreTemplateSpec.Spec.Containers {
	if container.SecurityContext != nil && container.SecurityContext.Privileged != nil {
		coreTemplateSpec.Spec.Containers[i].SecurityContext.Privileged = nil
	}
}

Environment:

• Volcano Version:
• Kubernetes version (use kubectl version):
• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Others:

[Thor-wl]

/cc @william-wang Please help take a look. THX.

[stale]

Hello 👋 Looks like there was no activity on this issue for last 90 days.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity for 60 days, this issue will be closed (we can always reopen an issue if we need!).

[LuBingtan]

We also meet the same problem in our production environment. It seems the reason is the validation in admission controller, i.e this block in admit_job.go:

// Skip verify container SecurityContex.Privileged as it depends on
// the kube-apiserver `allow-privileged` flag.
for i, container := range coreTemplateSpec.Spec.Containers {
    if container.SecurityContext != nil && container.SecurityContext.Privileged != nil {
        coreTemplateSpec.Spec.Containers[i].SecurityContext.Privileged = nil
    }
}

It will will return an error at k8scorevalid.ValidatePodTemplate, because it will check if the container is privileged when container's mount propagation is Bidirectional (in the vendor validation.go)

privileged := container.SecurityContext != nil && container.SecurityContext.Privileged != nil && *container.SecurityContext.Privileged
if *mountPropagation == core.MountPropagationBidirectional && !privileged {
    allErrs = append(allErrs, field.Forbidden(fldPath, "Bidirectional mount propagation is available only to privileged containers"))
}

And we have temporarily solved this problem in our internal environment.
Firsrt, we removed the reassignment for Privileged, and then enabled the AllowPrivileged capability in admission init func, e.g.

func init() {
	capabilities.Initialize(capabilities.Capabilities{
		AllowPrivileged: true,
		PrivilegedSources: capabilities.PrivilegedSources{
			HostNetworkSources: []string{},
			HostPIDSources:     []string{},
			HostIPCSources:     []string{},
		},
	})
}

But I'm not sure about is there any risk in this solution. Why volcano would skip verify container SecurityContex.Privileged? Can we using a flag (e.g. SkipVerifyPrivileged) to control the behavior?

Is there any suggestions ? @Thor-wl

[yuyue9284]

Same problem encountered, temporarily set --enabled-admission=/jobs/mutate,/podgroups/mutate,/pods/validate,/pods/mutate,/queues/mutate,/queues/validate in volcano admission to bypass the job validation.

[Thor-wl]

Thanks for the report. I've added important label for this issue. Will take a priority to do a check.
/assign @Thor-wl

[stale]

Hello 👋 Looks like there was no activity on this issue for last 90 days.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity for 60 days, this issue will be closed (we can always reopen an issue if we need!).

[Thor-wl]

Hi, I'm sorry for reply late. I've taken a look at releated history of the commits. Just as @LuBingtan mentioned above, it takes a check about priviledged containers. It depends on the configuration of kube-apiserver. I agree with @LuBingtan 's advice. Perhaps we can add a flag such as SkipVerifyPrivileged. Let me give a fix about that.

[Thor-wl]

I've added a parameter enabled-privilege-container for admission webhook to decide whether allows privilege containers, whose default value is false.

[Thor-wl]

Besides, I notice that issue2123 has discussed about this problem. Another approach is to ignore privilege container check in volcano admission webhook for it has already been checked by kube-apiserver before. What do you think about the 2 solutions? @junhuihuang @LuBingtan @yuyue9284

[william-wang]

As the kube-apiserver already check it. No need to check it in Volcano. It's inconvient for some users to check the api-server configuration and then keep volcano the same with apiserver, e.g. user use the k8s managed by cloud provider.

[k82cn]

I'm a little confiused about this issue? Did we skip "privilege" in Volcano right now? If skipped, why do we still have such kind of erro?

[william-wang]

We skiped "privilege" by setting coreTemplateSpec.Spec.InitContainers[i].SecurityContext.Privileged = nil, however mountPropagation: Bidirectional in pod depends on the "priviledge" which is changed to nil by us.

[Thor-wl]

I'm a little confiused about this issue? Did we skip "privilege" in Volcano right now? If skipped, why do we still have such kind of erro?

Because of the the force convert from privilege containers to common containers, which are introduced from PR411 and PR2125, Bidirectional mountPropagation cannot be used for it can only with privilege containers. It is very unreasonable.