[che-auth] secure workspace services

[sparkoo]

Is your task related to a problem? Please describe.

With new authentication, we need 2 levels of workspace protection. 1st is in the auth gateway (#19707), that will prevent unauthorized access from outside of the cluster through the ingress/route. 2nd level must be in workspace service, to prevent in-cluster attackers that could potentially access workspace service directly. This issue is about the latter case.

Describe the solution you'd like

I can think of 2 designs:

kube-rbac-proxy per workspace port

• put kube-rbac-proxy container in front of each opened workspace port and set proper RBAC rules
  

This should work and we did POC of this solution at #19297.

Big disadvantage of this solution is high overhead as we would need one extra container per open port, because kube-rbac-proxy can have only one upstream.

reverse proxy + kube-rbac-proxy per workspace

• put reverse proxy, like traefik, behind the service listen on all ports we need (https://doc.traefik.io/traefik/routing/entrypoints/) and do the routing to correct port, with forwardAuth (https://doc.traefik.io/traefik/middlewares/forwardauth/) to kube-rbac-proxy. We're doing something similar in Che gateway, but all tihs would live in workspace pod.

We did not tested this solution. It has better and constant footprint (+2 containers per workspace), but more complex setup.

Describe alternatives you've considered

• We could prepare some custom solution, something like kube-rbac-proxy that can do the routing to multiple upstreams by port.
• We could not care about in-cluster attacks. While this might be ok for onprem installations, it is not ok for SAAS, where anyone from the internet can access.

Additional context

epic: #19182

[skabashnyuk]

My two cents.

2nd level must be in workspace service, to prevent in-cluster attackers that could potentially access workspace service directly.

Is this SAAS only problem? Can we say that we trust all users by default and if not we will have to enable some "optional heavy" thing?

Big disadvantage of this solution is high overhead as we would need one extra container per open port, because kube-rbac-proxy can have only one upstream.

Can/should we explore the opportunity to improve this by contributing to kube-rbac-proxy?

CC @l0rd @benoitf

[benoitf]

yes exploring multiple stream might be better (more upstream solution than a custom one)

about the in-cluster attack, it could be interesting to be have a flag to toggle this

[sparkoo]

While contributing with multiple streams to kube-rbac-proxy may be an option brancz/kube-rbac-proxy#142, there is another issue. Kube-rbac-proxy needs cluster permissions to perform auth check https://github.com/brancz/kube-rbac-proxy/blob/master/examples/resource-attributes/deployment.yaml#L19. That would mean that the every workspace ServiceAccount would need ClusterRoleBinding to this ClusterRole. With devworkspaces, new SA is created for each workspace, so that would mean ClusterRoleBinding per workspace.
Currently creating ServiceAccount and everything related to it is in control of DevWorkspace controller, without any mechanism how to include ClusterRoleBinding to it.

So we have 2 issues with kube-rbac-proxy:

• multiple streams
• cluster permissions

Because of time-consuming possible contribution to kube-rbac-proxy upstream + needed downport to openshift port https://github.com/openshift/kube-rbac-proxy and new mechanism of assigning ClusterRoles to workspace ServiceAccount, I think custom solution is cleaner and simpler at this moment. I've put together simple multi-stream auth proxy that does not require any cluster permissions https://github.com/sparkoo/kube-svcauth-proxy.

• 👍🏻 only 1 extra sidecar per workspace
• 👍🏻 no additional permissions or cluster permissions needed
• 👍🏻 simple configuration
• 👎🏻 custom component with all the stuff around it (productization, CQ, ...)

I have 2 alternatives to completely custom solution, but both solves just 1st issue with multiple streams:

• use 2nd sidecar responsible for multiple stream routing. Something like traefik with forwardAuth to kube-rbac-proxy
  • 👎🏻 2 extra sidecars in the workspace
  • 👎🏻 still needs cluster permissions for kube-rbac-proxy
  • 👎🏻 complicated configuration of another component
  • 👍🏻 no custom component
• custom component responsible for multiple streams and using kube-rbac-proxy as a library
  • 👎🏻 custom component with all the stuff around it (productization, CQ, ...)
  • 👎🏻 still needs cluster permissions
  • 👍🏻 only 1 extra sidecar

[sparkoo]

I've came up with different solution that will need only 1 extra container per workspace, no cluster permissions and no custom component. Additional sidecar is Traefik (which we already have in main che-gateway), which will take care of routing inside the workspace pod AND with forwardAuth will authorize request through kube-rbac-proxy in main che-gateway. Hopefully diagram is explains better how it works.

ℹ️ dotted line is forwardAuth check https://doc.traefik.io/traefik/middlewares/http/forwardauth/

[sparkoo]

fixed by eclipse-che/che-operator#1045