Support for namespace-level configuration of pod tolerations and node selector

[skabashnyuk]

Is your task related to a problem? Please describe

As a Eclipse Che admin, I want to be able to set up the configuration of pod tolerations and node selector. Devworkspace operator expected them as k8s namespace annotations:

    metadata:
      annotations:
        controller.devfile.io/node-selector: |
          {
            "test-node-selector": "test"
          }
        controller.devfile.io/pod-tolerations: |
          [
            {
              "key": "key1",
              "operator": "Equal",
              "value": "value1",
              "effect": "NoSchedule"
            }
          ]

see more devfile/devworkspace-operator#696

Describe the solution you'd like

Che-operator should grab existing configuration from che-server's env var configuration CHE_WORKSPACE_POD_NODE__SELECTOR and CHE_WORKSPACE_POD_TOLERATIONS__JSON and put them in a format expected by devworkspace operator.

Describe alternatives you've considered

n/a

Release Notes Text

It is now possible to specify pod tolerations and node selector for workspaces of a Che instance setting a CustomCheProperty in CheCluster CR. It's not possible to specify that for just one workspace or for one user: it's a global config.

[ibuziuk]

@skabashnyuk we probably need to add the issue to #20830

[skabashnyuk]

The source of configuration now is che-server configuration: CHE_WORKSPACE_POD_NODE__SELECTOR and CHE_WORKSPACE_POD_TOLERATIONS__JSON. @l0rd do you think we should keep this source or it should be configured somehow differently?

CC @amisevsk @tolusha

[l0rd]

If I look at the new CheCluster draft I would say:

spec:
  server:
    workspacePodNodeSelector:
    workspacePodTolerationsJson:

@amisevsk @tolusha what's your take on that?

[tolusha]

What about moving workspace related configuration to a dedicated section?

spec:
   server:
     workspace:
       podNodeSelector:
       podTolerationsJson:

or:

spec:
   workspace:
     podNodeSelector:
     podTolerationsJson:

[amisevsk]

podTolerationsJson

If we're doing configuration on a CR level, we can simply reuse the k8s types in the configurations struct, i.e. make podNodeSelector a map[string]string and podTolerations a []corev1.Toleration. Defining these things as strings is error-prone (as it has to be valid json/yaml and a missed comma breaks things).

I do think that configuring it in a CR is more straightforward and user-friendly. For DWO, moving to a config CRD instead of a configmap has given us

1. Autocomplete in VSCode/the OpenShift Console editor
2. Easily accessible documentation through oc explain

In terms of scoping, I don't have any real opinion on workspace vs server, but it is worth noting that storing this config in the CR binds it to forwards/backwards compatibiliity, so whatever we choose here we're stuck with for a long time.

[skabashnyuk]

@l0rd do you still want to put configuration under /spec/server/ as described here #20884 (comment) or maybe /spec/workspace/ as @tolusha #20884 (comment) suggested is better?

[l0rd]

I am ok with both. The important thing though is to be consistent: currently workspaces related fields are under spec.server. If we create a new spec.workspaces those fields should be moved to there too.

[metlos]

For CheCluster v1, I'd definitely avoid any restructuring, because that would be breaking the compatibility with CheCluster resources preexisting in the cluster. So for v1, I'd slot these under spec.server. For v2, we can pick any place we want.

[devstudio-release]

sync'd to Red Hat JIRA https://issues.redhat.com/browse/CRW-2711