Devworkspace native auth on Kubernetes - Unauthorized! Configured service account doesn't have access #21049
Comments
nils-mosbach
added
the
kind/bug
che-bot
added
the
status/need-triage
ibuziuk
added
severity/P1
|
@sparkoo could you please take a look? |
|
@nils-mosbach It's not only about Dex being more lightweight then Keycloak. With devworkspaces being Kubernetes objects, we need somehow control access to them. So we need to set Kubernetes RBAC rules so that only devworkspace owner has an access. For that we need Kubernetes to know the user, thus we require Kubernetes to be configured with OIDC. So for devworkspace engine (default since 7.42 I think), OIDC Kubernetes is mandatory. For che-workspace engine, you can still use built-in keycloak as before. However, I can't guarantee for how long this will be supported. |
|
@sparkoo Thanks! Makes sense. As soon as I got this working, I'll drop a summary what was necessary for Rancher deployed clusters here. Maybe that helps others as well. Do you have an idea what could be the issue? |
|
It looks like kubernetes is not properly configured with oidc. If you see login page and can actually login and you're redirected to dashboard page, oidc server and che configuration should be fine.
Please note that you have to use same client (id and secret) for both che and kubernetes apiserver. Here's the doc how to configure kubernetes apiserver with oidc https://kubernetes.io/docs/reference/access-authn-authz/authentication/#configuring-the-api-server Here's chectl code that is checking oidc configuration of kubernetes (I'm not sure it's 100% bulletproof) https://github.com/che-incubator/chectl/blob/main/src/commands/server/deploy.ts#L443 |
|
Looking at the code, woud this check fail when the oidc args are listed under args and not commands? not sure exactly how the go api works for that. in our deployment the command is just the command and everything else, including oidc settings are under args, and we fail the check. |
|
Getting closer... :) We've tested kubectl login with a cluster admin account and tried to use "regular"-user accounts for che. While digging a little deeper, after we created a clusterrolebinding for the user that should access che, dashboard worked as expected. Question is: Should this be done by all users manually or is this a process that should be handled by che? For now granting access rights before users are going to login is fine, but later on it'll be nice if che will grant all required rights for accessing the users namespace. After we granted cluster-admin rights there are at least roles and bindings created for the workspace. Removing the clusterrole admin binding will cause the dashboard to crash again. We're still struggling with tokens that expire after ~5 mins but that might be due to the token lifespan (keycloak/gitlab). I'll have a look on this tomorrow. In our case chectl still deploys devworkspace operator 0.9. Is this causing the issue? chectl check |
That's great information for us. Can you please create an issue so we can improve the check? Or you can propose PR yourself. cc: @tolusha |
|
@nils-mosbach looks promising. Seems like you're logged with the user that kubernetes knows. I would not recommend to set cluster-admin permissions for the user. In general, Che should set all needed permissions for user during the namespace provisioning (che-dashboard request to che-server). Do you create namespace yourself or is it created by Che? The permissions are created in https://github.com/eclipse-che/che-server/blob/main/infrastructures/kubernetes/src/main/java/org/eclipse/che/workspace/infrastructure/kubernetes/namespace/configurator/UserPermissionConfigurator.java. As you can see there, Che does not know any particular cluster roles, but it get's it from So in the final, in user's namespace you should have something like |
|
I think we finally found the issue which caused it. Kubernetes uses It seems that in the users table there's is a colum that contains the After we deleted the database and started fresh, the @sparkoo: Thanks a lot! If you want to improve chectl checks, one thing that would be nice is if |
|
For others reading this, setting everything up on rancher was quite simple. Deploy keycloak
Note that this client must be used by che and kubernetes and must be protected. Otherweise OAuth2-Proxy will throw an error that secret must not be empty. Setup OIDC for kube-api-serverOn Rancher provisioned clusters the api server can be configured using extra_args.
After saving the changes it takes a couple of minutes until all ctrl plane nodes are restarted. If using a self signed certificate, which we don't, it gets a little tricky since kube api server must know the CAs certificate. Configure CheSet the external identifty provider and oidc username claim in the cr. |
|
@nils-mosbach May I know how did you configure kubernetis with OIDC provider? I mean, which yaml file needs to be modified? I'm not able to narrown down to which piece thru Rancher. |
|
Rancher does not deploy the api server as a Kubernetes resource. That’s why the official Kubernetes documentation doesn’t work. for rancher: when you edit your Cluster there’s an option to switch to yaml mode. That of course only works with rancher provisioned clusters. In other cases that needs to be set in the api-server/kube-System namespace as described by the K8s docs. |
|
@nils-mosbach Thank you!! Let me play around and see. |
|
I ran into this same issue with Che 7.43 deployed via the che-operator 7.43 onto a GKE cluster with a separate Keycloak instance that I manage myself. In case someone finds this helpful, here are the steps I took to finally get a working installation. On GKE (probably EKS and AKS too) you cannot just change Kubernetes API server settings since this is being managed by the cloud provider.
Lots of effort was required before I was "CodeReady" :P Bonne chance! |
Hi... Thanks & regards |
|
Hi @nils-mosbach , could you please share with me how to set up keycloak client? Is there any option I have to change beside using the default ones when create the client. Thank alot. |
Describe the bug
We have a keycloak that authenticate agains GitLabs OIDC provider. (Informationen on our setup can be found here: #20962 (comment)) That actually served us well using Eclipse Che 7.40.
Upgrading to 7.42 seems to require native auth, so we tried to setup OAuth on Kubernetes using keycloak based on information provided by @sparkoo #21036 (comment). Are we missing something?
Dashboard just shows a blank page and shows the following errors.
It seems roles, bindings and serviceaccounts are not created. Should this be done by eclipse che in the provision api call?
Since chectl fails if oauth wasn't properly setup: Is the "non-native-auth" way of handling users deprecated with the release of Che 7.42?
I get the idea of using OpenShifts buildin authentication and understand that some might find keycloak quite heavy compared to dex. But I think setting up OAuth on the Kubernetes API server isn't really something a lot of users do on a regular basis. I'm not even sure if all managed kubernetes services allow changing these settings. Troubleshooting this tends to be quite difficult as well. (#21036 (comment))
Is setting
nativeUserMode: falsestill a supported option?Che version
7.42@latest
Steps to reproduce
I've created a new client
kubernetesin keycloak and set the following settings on our kube-api-server:Chectl does not verify this setting as valid and throws a warning. Not sure what's the issue, so tried bypassing the check
--skip-oidc-provider-check. The chectl log file unfortunately does not contain aditional information.Che configuration contains the following configuration.
Additional Info:
Expected behavior
User native mode on kubernetes should work.
Runtime
Kubernetes (vanilla)
Screenshots
No response
Installation method
chectl/latest
Environment
Linux
Eclipse Che Logs
Additional context
Kubernetes: v1.20.4
Provisioned by Rancher
No errors in Kuberentes API Server logs.
OAuth2 Proxy Logs
Since Che log tries to find a secret in the workspaces' name:
The text was updated successfully, but these errors were encountered: