Use a reverse proxy to avoid routes/ingress creation at workspace startup

[l0rd]

Description

In the past 2 years of running Che in production we have seen that OpenShift routes do not always fit our needs (we need to bring-up 3 or more routes in a few seconds for every user workspace). The same applies to Kubernetes Ingresses (still in Beta).

Thus the need to investigate alternatives. One proposal is to pre-create one workspace route/ingress for Che server and link it to a reverse proxy that will route all the workspaces traffic (e.g. re-use JWT proxy, envoy or traefik).

That would allow us to:

• limit the number or route/ingress Che needs
• create the route before a user ask to start a workspace

If this approach is validated we could divide the work in 4 steps

• Analyse if the already existent JWT Proxy would satisfy our needs
• All workspaces inbound and outbound traffic should be routed through the JWT Proxy
• Share one JWT Proxy instance amongst all the workspaces of a single user
• Share one JWT Proxy instance amongst all the Che users of a Kube cluster

UPDATE Another important use case is associated to this issue: allow running Che workspaces on OpenShift (where single-host strategy is not available) when using wildcard SSL certificates is not possible.

Implementation

• [Study] Possible implementations for single-host strategy on OpenShift [Study] Possible implementations for single-host strategy on OpenShift #16702
• Implement a single host version of the happy-path test Implement a single host version of the happy-path test #16842
• Reuse openshift-router - Impossible
• POC for Traefik - https://github.com/metlos/che-singlehost-poc
• POC for Traefik using CRDs - https://github.com/skabashnyuk/openshift-traefik
• POC for HAProxy - https://github.com/sparkoo/che-singlehost-haproxy-POC
• POC for nginx - Single-host for OpenShift - POC for nginx #16883
• Investigate a controller able to sync configmaps and send a signal to a gateway process - https://github.com/metlos/cm-bump
• Investigate the best way for hot-reloading the configuration in gateway
  • External controller - Single-host for OpenShift - Hot-reload the gateway using external controller #16886
  • In-container controller - Single-host for OpenShift - Hot-reload the gateway using in-container controller #16887
  • Sidecar controller - not working on default Openshift 3.x due to shared process namespace being switched off by default
• Testable POCs
  • Traefik - Single-host for OpenShift - Traefik testable POCs #16888
  • nginx - should be part of Single-host for OpenShift - POC for nginx #16883
  • HAProxy - the tests are being implemented using this POC
  • Envoy - Single-host for OpenShift - Envoy testable POCs #17182
• Do performance tests - Single-host for OpenShift - Performance tests #16889
  • Run perf tests for Envoy Run performance tests on Envoy proxy singlehost POC #17243
• Che Server implementation
  • Enable singlehost strategy to not be ingress-bound Enable singlehost strategy to not be ingress-bound #17059
  • Implement gateway-based singlehost strategy for Kubernetes Implement gateway-based singlehost strategy for Kubernetes #17060
  • Implement gateway-based singlehost strategy for OpenShift Implement gateway-based singlehost strategy for OpenShift #17061
• Implement cookie path rewriting in Traefik ❓ Implement cookie path rewriting in Traefik #17062
• Finalize our config-fetching controller Finalize gateway config fetching #17063
• Update Helm chart to support gateway-based singlehost mode Update Helm chart to support gateway-based singlehost mode #17064
• Update the che-operator to support gateway-based singlehost mode Update the che-operator to support gateway-based singlehost mode #17065
• Document how to deploy gateway singlehost with Chectl Document how to deploy gateway singlehost with Chectl #17525
• Keycloak behind the gateway [Singlehost on Openshift] Keycloak behind the gateway #17809
• Ensure Configubump tool has all necessary CQs open | Ensure Configbump tool has all necessary CQs open #17568
• Configubump tool produces incorrect configuration in some cases | Configubump tool produces incorrect configuration in some cases #17567

[l0rd]

cc @gorkem

[che-bot]

Issues go stale after 180 days of inactivity. lifecycle/stale issues rot after an additional 7 days of inactivity and eventually close.

Mark the issue as fresh with /remove-lifecycle stale in a new comment.

If this issue is safe to close now please do so.

Moderators: Add lifecycle/frozen label to avoid stale mode.

[skabashnyuk]

/remove-lifecycle stale

[benoitf]

should it go into a backlog ?

[l0rd]

We should put it into a backlog during priotization

[che-bot]

Issues go stale after 180 days of inactivity. lifecycle/stale issues rot after an additional 7 days of inactivity and eventually close.

Mark the issue as fresh with /remove-lifecycle stale in a new comment.

If this issue is safe to close now please do so.

Moderators: Add lifecycle/frozen label to avoid stale mode.

[l0rd]

/remove-lifecycle stale

[metlos]

We've started implementing the performance tests for the individual POCs. Take a look at https://github.com/che-incubator/che-gateway-poc.

[metlos]

In the above mentioned POC repository, we now have 3 POCs implemented:

• haproxy-scripted - this is a vanilla haproxy image operated by oc commands from the test scripts
• nginx-custom-image - this is a custom image using our cm-bump utility and nginxinc/nginx-unprivileged official image of nginx
• traefik-sidecar - this is combination of traefik and cm-bump (we don't require a custom image in this case because traefik can watch for config changes on its own)

We're working on haproxy-custom-image which is very similar to nginx-custom-image only with haproxy as the gateway solution. This is to be able to quantify the effect of a custom controller vs externally executed commands.

We're also working on the testsuite. We're developing a number of load test scenarios (https://github.com/che-incubator/che-gateway-poc/tree/master/test#testcases). We have not yet started websocket and cookie handling tests which we are going to start once the haproxy-custom-image poc is implemented.

[sparkoo]

@jfaltermeier as a side-effect, yes it will help. We will have only one Ingress for whole Che and will do routing to workspaces ourselves, so we can do it more effectively than cluster.

[metlos]

We have concluded the performance tests. I have created a number of subtasks to guide us through the implementation and referenced them in the description of this epic.

We have not yet chosen the gateway solution though, because there was no clear winner. I have sent out an email to the Che-dev mailing list detailing our current thinking and progress.

[metlos]

Note that we have concluded our testing of the candidate solutions for a reverse proxy. We chose Traefik and will commence the implementation with #17063 - making our Rust-based POC a fully maintained controller written in Go.

To read more about the selection process and reasoning behind the choice of Traefik, please read through https://www.eclipse.org/lists/che-dev/msg03828.html

[sparkoo]

all issues in the scope of this epic closed. related single-host issues will be solved separately.