Support Pod tolerations for DevWorkspace Pods

[metlos]

Description

To support node selection for workspace pods, it would be great if we supported pod tolerations and node selector for DevWorkspace pods.

It would be best if this was a noop but we need to investigate if the same can be done on namespace level - e.g. by labeling namespaces such that addmission controllers can select the nodes automatically, see https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#podnodeselector

[sleshchenko]

As far I as understand @l0rd will provide a better requirement for this feature, like should it be defined on namespace object/on a specific workspace level.

[l0rd]

I have looked at past issues and here is the original where it's mentioned that a central config is required first. NodeSelector per workspace was considered a successive iteration.

I also found a customer request:

node selector would be great to have. If that can be defined in dev file e.g, that would be best.
So being able to specify that at workspace level is important too (I don't think that we should do that at devfile spec level though as that would make the devfile less portable).

Summing it up:

• Higher priority: being able to have current, global, che-server configuration in devworkpace world too
• Lower priority: being able to specify NodeSelector at the workspace level

[max-cx]

Hi, a question to the assignee of this issue:

Will the outcome require any changes to the relevant content of the Installation Guide or Administration Guide or End-user Guide?

Yes/No?

[amisevsk]

Supporting this makes sense as a part of the main Devfile API; I've created devfile/api#689 and will follow up.

[skabashnyuk]

Now pod teleration look like this

# Optionally configures tolerations for workspace Pod. Format is a string representing a JSON Array of taint tolerations,
# or `NULL` to disable it. The objects contained in the array have to follow the
# link:https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.20/#toleration-v1-core[toleration v1 core specifications].
# Example: `[{"effect":"NoExecute","key":"aNodeTaint","operator":"Equal","value":"aValue"}]`
che.workspace.pod.tolerations_json=NULL

@amisevsk can you tell us how this config should look like so devworkspace controller can understand it?

[amisevsk]

We had a discussion in a devfile/api call today, and it sounds like we want to hold off on setting per-workspace tolerations and node selectors for now. As a result, it's not fully clear how this will be implemented on the DWO side for the time being, though it will likely be as a global config option.

Formatting-wise, what's in Che would work. The initial proposal (#681) uses pod tolerations matching the k8s object (we literally deserialize into []corev1.PodTolerations). For node selectors, it's just a map[string]string.

[amisevsk]

PR #696 adds support for setting tolerations and node selectors at a namespace level. @skabashnyuk would it be possible to make Che apply the appropriate annotations when syncing a user's namespace?

[skabashnyuk]

@amisevsk I think yes. eclipse-che/che#20884