Investigate how Che can propagate SelfSigned certificates and proxy settings to devworkspaces

[sleshchenko]

Is your task related to a problem? Please describe.

Che is typically run with self-signed certificates which should to propagated to devworkspace to make Che Theia request plugin registry.

In addition Che has a feature to configure custom self-signed certificates, which then can be used withing DevWorkspace for custom git servers and other stuff.

In addition, Che supports http proxy, so Che should propagate HTTP_PROXY, HTTPS_PROXY, HTTP_NO_PROXY, HTTPS_NO_PROXY (I know pretty nothing about proxy, so maybe something is missing) env vars into DevWorkspace containters.

Describe the solution you'd like

The way how we're going to implement it is not defined yet.
It may be a dedicated controller that propagate some configuration into user's namespaces and then DevWorkspace operator just mount, inject it into DevWorkspace containers.

[amisevsk]

Serhii and I talked through a bit of the potential options for implementing this recently; I'm copy-pasting the contents of a Google doc (that is otherwise inaccessible except by Red Hat employees) below.

Propagating Che settings into DevWorkspaces

This document outlines options for how we can propagate global configuration settings (e.g. proxy settings, self-signed certificates) into DevWorkspaces when running with Che.

Overall goal

The Che server allows configuring settings common to all workspaces e.g. for self-signed certificates and proxy settings. With the usual architecture (with the Che server creating/managing workspace deployments) it can provision the required settings directly into the deployment on top of a provided devfile. However, once creation of devworkspace deployments is delegated to the DevWorkspace Operator (DWO), this will no longer be possible.

There’s a plan to add functionality to the DevWorkspace Operator to automatically mount configmaps/secrets with specific labels into any workspace that starts in the same namespace, e.g.

kind: ConfigMap
labels:
  controller.devfile.io/automount: [true|false]
  controller.devfile.io/automountPath: “/path/to/files”
  controller.devfile.io/automountAsEnv: [true|false]

...etc. Note the design of this is still undefined, but it would be useful to have

1. Mount keys in configmap as files in a given directory
2. Use envFrom to set all values in configmap as environment variables

Option 1: Che server/operator/dashboard creates and manages configmaps in each user namespace

If some Che-side component can manage creating and maintaining configmaps/secrets as required for this functionality (and reusing the automount functionality), then this would solve the problem. However, the issues we would face with this approach would be

1. Whoever creates the configmaps would have to track configmaps in all user namespaces, and keep them up to date when the global setting is changed.
2. The creator would also need to track cleanup where appropriate, as these configmaps wouldn’t be owned by the DevWorkspace
3. The creator would need to re-create configmaps if they happen to be deleted

The issues around maintaining configmaps lead to Option 2:

Option 2: Add functionality to the DevWorkspace Operator to automatically propagate configmaps with annotations from its namespace to all DevWorkspaces

The basic idea here is that Che (server/operator/dashboard) maintains a configmap inside the devworkspace-operator’s namespace (likely openshift-operators if installed via operatorhub) with annotations as above. For every DevWorkspace, the DWO creates copies of these configmaps in the DevWorkspace’s namespace, owned by the DevWorkspace, and reuses the automount functionality. This solves the lifecycle problems above, as Che only needs to maintain configmaps in one location, but would also mean that every workspace gets these configmaps automatically mounted (which would include Web Terminals, for instance).

Option 3: Add attribute/annotation to DevWorkspaces that references a namespace to pull global configmaps from

The idea here is similar to option 2, but instead of automatically grabbing configmaps for all workspaces, we require an annotation on created DevWorkspaces, e.g.

annotations:
  controller.devfile.io/automount-from: ‘<namespace>’

If a DevWorkspace is created with this annotation, the DevWorkspace controller looks inside <namespace> for any configmaps with the special automount annotations described above, and copies them to the current namespace. This would avoid the issue of propagating the information needlessly while keeping the lifecycle relatively simple.

[metlos]

Is this discussing only the support for things that are truly managed "globally", such as the mentioned certificates or proxy settings, or is it meant as a generic feature for applying external settings to devworkspaces?

If it is the latter then I think we cannot simply rule out option 1, because some settings might contain sensitive information, like credentials (in which case we even should not store them as configmaps, but rather secrets). These should IMHO be dealt with on a per-user (i.e. per-namespace) basis as suggested in option 1.

[amisevsk]

This is targetted at the former -- settings you select on the Che level that should apply to every single workspace managed by Che.

For workspace-specific settings, we're planning on adding some mechanism to automount configmaps/secrets in DevWorkspaces, so we would just have to create the correct secret/configmap in the workspace namespace and DWO would handle it. The issue for a global setting is that choosing option 1 means having to maintain state on n configmaps in case the user changes e.g. proxy settings.

[l0rd]

I understand that option 1 is required anyway for user specific settings (gitconfig, ssh keys etc...) but options 2/3 are an optimization for Che global configs (proxy, trusted tls bundles etc...).

A few considerations:

• as we need to implement option 1 for user settings I would initially use the same mechanism for global configs too
• in the case of multi cluster Che we will need to manage n configmaps anyway, one for every cluster (with a smaller n in this case).
• when switching to the devworkspace engine we only support per user namespace strategy (I am saying that because I am not sure if "in each user namespace" above meant that a user can have multiple namespaces)
• between option 2 and 3 I prefer option 2 as one central configmap is simpler to manage for an admin. Those configs/secrets will be probably managed by an admin rather than by the final user that creates devworkspaces.

[amisevsk]

A few comments:

• as we need to implement option 1 for user settings I would initially use the same mechanism for global configs too

The main issue here is that changing global settings would require iterating though each user's namespace. @sleshchenko and I discussed this and decided it might be hard to do, especially since some maintenance of these config settings would be required (e.g. what happens if I delete my local configmap? Will it get recreated immediately?)

• in the case of multi cluster Che we will need to manage n configmaps anyway, one for every cluster (with a smaller n in this case).

I agree, but the difference is n configmaps in defined namespaces (e.g. always che-global-settings) vs n*m namespaces, all with different names.

• between option 2 and 3 I prefer option 2 as one central configmap is simpler to manage for an admin. Those configs/secrets will be probably managed by an admin rather than by the final user that creates devworkspaces.

I think my description isn't entirely clear -- both option 2 and 3 have one central configmap. The only difference is the namespace (alongside DWO -- i.e. in openshift-operators -- vs any namespace the admin chooses). With option 2, DWO looks in the operator namespace for the configmap, with option 3, DWO looks in an annotation-defined namespace for the same configmap. In option 3 it's assumed that Che will provision DevWorkspaces with the correct namespace in an annotation.

[sleshchenko]

as we need to implement option 1 for user settings I would initially use the same mechanism for global configs too

The main issue here is that changing global settings would require iterating though each user's namespace. @sleshchenko and I discussed this and decided it might be hard to do, especially since some maintenance of these config settings would be required (e.g. what happens if I delete my local configmap? Will it get recreated immediately?)

Yeah, the difference is that global settings and user's specific (container registries, ssh keys) has different lifecycle.
Making clients like dashboard managing user's specific - container registries, is a right thing, because, user uses client to modify that stuff.
With global settings, I don't see it as way to go(making clients in charge to propagate it), because when clients are supposed to propagate these changes? On tab refresh? Before workspace start?
And how clients will get that info?

Comparing 2 and 3, I like more 2, since 3 has additional complexity in RBAC perspective.
Which namespaces we allows for the users to mount? Some fixed configured?

openshift-operators -- vs any namespace the admin chooses

Using openshift-operators can be not good enough, since only real cluster admins has access to it.
But it should not be an issue since it's Che operator who is supposed to manage that configmap, not a real user. While user is supposed to manage CheCluster.
If we want real users to has direct access to that secrets and configmap - we may also support automount from default namespace (where regular users should not have access to, I believe) but it's less restricted than openshift-operators.

[amisevsk]

Which namespaces we allows for the users to mount? Some fixed configured?

It's a combo of namespace + the appropriate automount labels, so I think the risk is minimal even if we use "any namespace", but I see your point.

Edit: though, thinking for a moment more, this could allow users to steal ssh keys from other namespaces, so that probably means option 3 is a no-go.

[metlos]

I would like to point out that we should think about if these kinds of settings are ever going to differ between different usages of devworkspace (i.e. wto or che).

Therefore the implementation should reflect what we decide in #19326, too.

[sleshchenko]

The investigation is done.
It mostly duplicates with #19326, so closing this and dedicated issues will be created after we make a decision and choose the way.