Auth via Keycloak

[Fareroo7]

Summary
I want to use Keycloak as IDP and the auth should be working but I get an error inside the kubeappsapi container...
In the logs I get an AuthSuccess message, but I get redirected back to the login page.

Background and rationale
I installed kubeapps via the bitnami helm chart.

Version: bitnami/kubeapps 8.1.11

Used custom values:

ingress:
  enabled: true
  hostname: kubeapps.k8s.local
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
  tls: true
authProxy:
  enabled: true
  provider: oidc
  clientID: kubeapps
  clientSecret: #secret
  cookieSecret: #secret
  skipKubeappsLoginPage: false
  extraFlags:
    - --ssl-insecure-skip-verify
    - --cookie-secure=false
    - --scope=openid email groups
    - --oidc-issuer-url=#url
postgresql:
  primary:
    persistence:
      enabled: true

Description
After successful (?!) login, I get redirected back to the login page.

Additional context
kubeapps auth-proxy log:

10.244.2.93:48308 - d6adbcced0916375182d2691e44bcf75 - dominik.simon@fs-soft.at [2022/06/20 12:42:25] [AuthSuccess] Authenticated via OAuth2: Session{email:dominik.simon@fs-soft.at user: PreferredUsername:d.simon token:true id_token:true created:2022-06-20 12:42:25.996248618 +0000 UTC m=+130.787952188 expires:2022-06-20 12:47:25.995288077 +0000 UTC m=+430.786991647 refresh_token:true groups:[admin developer]}
10.244.2.93:48308 - d6adbcced0916375182d2691e44bcf75 - - [2022/06/20 12:42:25] kubeapps.k8s.local GET - "/oauth2/callback?state=0eSqzb5278PLY6GKsSFOzQ2phWzazPwcZuy9MtujKlw%3A%2F&session_state=4b95465a-a474-4397-bcfa-f533e5f479f4&code=d6e2ef99-af52-4f49-ab68-7a93af85eeec.4b95465a-a474-4397-bcfa-f533e5f479f4.63c9b0c1-9a77-4f72-b711-4b6ab8799d18" HTTP/1.1 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.0.0 Safari/537.36" 302 24 0.017
10.244.2.93:48308 - 53e335928db2b80db5036b0188741f87 - - [2022/06/20 12:42:26] kubeapps.k8s.local GET - "/oauth2/start" HTTP/1.1 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.0.0 Safari/537.36" 302 323 0.000

kubeapps nginx log:

127.0.0.1 - - [20/Jun/2022:13:08:49 +0000] "GET /config.json HTTP/1.1" 200  445 "https://kubeapps.k8s.local/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.0.0 Safari/537.36" "10.244.2.1, 10.244.2.93"
127.0.0.1 - - [20/Jun/2022:13:08:49 +0000] "POST /apis/kubeappsapis.plugins.resources.v1alpha1.ResourcesService/CheckNamespaceExists HTTP/1.1" 200  5 "https://kubeapps.k8s.local/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.0.0 Safari/537.36" "10.244.2.1, 10.244.2.93"
127.0.0.1 - - [20/Jun/2022:13:08:49 +0000] "POST /apis/kubeappsapis.plugins.resources.v1alpha1.ResourcesService/CheckNamespaceExists HTTP/1.1" 200  5 "https://kubeapps.k8s.local/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.0.0 Safari/537.36" "10.244.2.1, 10.244.2.93"
10.244.2.1 - - [20/Jun/2022:13:08:57 +0000] "GET / HTTP/1.1" 200  1920 "-" "kube-probe/1.24" "-"
10.244.2.1 - - [20/Jun/2022:13:09:07 +0000] "GET / HTTP/1.1" 200  1920 "-" "kube-probe/1.24" "-"
10.244.2.1 - - [20/Jun/2022:13:09:17 +0000] "GET / HTTP/1.1" 200  1920 "-" "kube-probe/1.24" "-"
10.244.2.1 - - [20/Jun/2022:13:09:27 +0000] "GET / HTTP/1.1" 200  1920 "-" "kube-probe/1.24" "-"

kubeapps api log:

I0620 13:01:36.764881       1 namespaces.go:24] +resources CheckNamespaceExists (cluster: "default", namespace="default")
I0620 13:01:36.769786       1 server.go:59] Unauthenticated 4.927107ms /kubeappsapis.plugins.resources.v1alpha1.ResourcesService/CheckNamespaceExists
I0620 13:08:49.410141       1 namespaces.go:24] +resources CheckNamespaceExists (cluster: "default", namespace="default")
I0620 13:08:49.424461       1 server.go:59] Unauthenticated 14.339612ms /kubeappsapis.plugins.resources.v1alpha1.ResourcesService/CheckNamespaceExists
I0620 13:08:50.580987       1 namespaces.go:24] +resources CheckNamespaceExists (cluster: "default", namespace="default")
I0620 13:08:50.586011       1 server.go:59] Unauthenticated 5.036115ms /kubeappsapis.plugins.resources.v1alpha1.ResourcesService/CheckNamespaceExists

[antgamdia]

Hi,

it seems you are successfully authenticated, but the token is not authorized somehow. There is a variety of reasons why it can fail; I'd recommend you have a look at the OIDC debugging docs as well as the guide for configuring Keycloak in Kubeapps.

[Fareroo7]

Hi @antgamdia,

thank you a lot for the response.
The OIDC debugging docs helped me to solve my issue.
The authentication against Kubeapps was successful, but I was not authenticated against Kubernetes.

[ghost]

Hi @antgamdia, I get the same redirect loop because I use a single keycloak kubeapps client.

Here, the environment consisted of the kubeapps web app and two kubernetes clusters. So we need to create three clients.

Why do you enforce an additional kubernetes client in keycloak when it's running on the same node?
I guess I'm forced to use an extra client, but which id and secret should i use?

Please add a global admin user and password and replace step 2 in the getting-started tutorial.

How to remove unused scopes like email and groups?

[absoludity]

Hi @kllp . The redirect loop was solved above by the original poster by following the linked debugging instructions. Were you able to work through those to see where the issue is? The normal case is that you are authenticating fine against your identity provider (keycloak) but the credential isn't valid for your cluster (usually a config issue), as was the case for the original poster.

We don't enforce an additional OIDC client-id per kubernetes cluster - that's just recommended. You can choose to use the same client for both of your clusters.

guess I'm forced to use an extra client, but which id and secret should i use?

I'm not sure what you mean here, sorry. I thought you were talking about the OIDC client-ids, but here it sounds like you're talking about user login details?

Please add a global admin user and password and replace step 2 in the getting-started tutorial.

The getting started documentation intentionally does not include instructions for an integrated authentication system (such as Keycloak, or Dex) for the very reason that those authentication systems can be quite complex to setup, debug and integrate. That's why the getting started focuses on a demo setup using a service account, because it is simple to try out. We have documentation for the more complicated authentication integrations.

How to remove unused scopes like email and groups?

Do you mean in Keycloak? You would have to check their documentation, but generally I'm not sure why you would not be using those in the sense that, your cluster will be presented with a signed id_token from your identity provider - keycloak - which usually, in a normal configuration, uses those scopes to link that id_token with matching RBAC on your cluster for user and groups.

[ghost]

Thank you. I didn't read the entire debugging page.

One common issue is that the Kubernetes cluster’s api server is not configured for oidc (some people don’t realise this is necessary).

Can you please explain why this is necessary and other cluster management tools don't need it?

The getting started documentation intentionally does not include instructions for an integrated authentication system (such as Keycloak, or Dex) for the very reason that those authentication systems can be quite complex to setup, debug and integrate. That's why the getting started focuses on a demo setup using a service account, because it is simple to try out. We have documentation for the more complicated authentication integrations.

My request to use a plain admin password is about not having to use integrated authentication systems. The 'quick' alternative involves a kubernetes token (you have to create a serviceaccount, a clusterrolebinding and a secret) which is not as easy as using predefined admin password or token.

I tried to base64 encode a short custom token as serviceaccount secret, but obviously the token must have a fixed length.
kubectl get --namespace default secret kubeapps-operator-token -o go-template='{{.data.token | base64decode}}' returns the correct custom token, but kubeapps returns There was an error connecting to the Kubernetes API. Please check that your token is valid.

Do you mean in Keycloak? You would have to check their documentation, but generally I'm not sure why you would not be using those in the sense that, your cluster will be presented with a signed id_token from your identity provider - keycloak - which usually, in a normal configuration, uses those scopes to link that id_token with matching RBAC on your cluster for user and groups.

I asked to keep the Keycloak configuration as simple as possible because it prevents configuration errors.

[absoludity]

Thank you. I didn't read the entire debugging page.

👍

One common issue is that the Kubernetes cluster’s api server is not configured for oidc (some people don’t realise this is necessary).

Can you please explain why this is necessary and other cluster management tools don't need it?

Remember, Kubeapps isn't a cluster management tool - we don't manage clusters, we allow users to browse a catalog of apps that they (the user, with whatever RBAC they've been assigned on the cluster - see users in Kubernetes) can install on the cluster (according to their RBAC on that cluster). I don't know which cluster management tool you might mean, but at a guess, perhaps those tools run with a privileged service account that already has the required permissions to do privileged actions. Kubeapps, on the other hand, allows users to only do what their (already defined) RBAC on the cluster allows. If you want to use OIDC for user authentication for Kubeapps, then the cluster must be able to trust the id_tokens that your identity provider provides (that's the bit that requires configuration of your cluster).

My request to use a plain admin password is about not having to use integrated authentication systems. The 'quick' alternative involves a kubernetes token (you have to create a serviceaccount, a clusterrolebinding and a secret) which is not as easy as using predefined admin password or token.

Yes, that would be nice to be able to have available to make it easier to try out. I just don't know how we could do that safely - since I think it would require either:

• the cluster itself to be configured in some way to validate the user's credential (this is the OIDC config) and match them to a privileged cluster role, or
• kubeapps to include a privileged account that is used when the user logs in (very much against this out of the box because it is privilege escalation, but this is also exactly what the getting started gets people to do manually with warnings that it is for demo only)

Since we don't control the authentication systems trusted by the cluster, Kubeapps can't really do this. (We do do this in our development environment because we control the cluster creation and configuration).

I tried to base64 encode a short custom token as serviceaccount secret, but obviously the token must have a fixed length. kubectl get --namespace default secret kubeapps-operator-token -o go-template='{{.data.token | base64decode}}' returns the correct custom token, but kubeapps returns There was an error connecting to the Kubernetes API. Please check that your token is valid.

That means that the Kubernetes cluster did not think your token is valid. I'd recommend letting Kubernetes create a valid serviceaccount secret for you (I've never tried or wanted to manually create one).

Do you mean in Keycloak? You would have to check their documentation, but generally I'm not sure why you would not be using those in the sense that, your cluster will be presented with a signed id_token from your identity provider - keycloak - which usually, in a normal configuration, uses those scopes to link that id_token with matching RBAC on your cluster for user and groups.

I asked to keep the Keycloak configuration as simple as possible because it prevents configuration errors.

Sorry, I don't think I'm following what you mean. It's one thing to want to keep your Identity Provider configuration simple, but at the very least, it needs to be able to communicate to the cluster a user id and group id for an authenticated user in some form that the cluster can trust. All I meant was that normally the user email and group scopes of a signed id_token are used for this, so it's not clear to me why or how you could remove them without having some alternative.

I think you're saying that you wish authentication could be simpler: if so, I agree :) If you have implementation ideas for how it could be simpler for a demo Kubeapps instance that talks to a cluster (simpler than the service-account instructions we provide), I'm happy to discuss it :)

[ghost]

Thank you very much for your quick and extensive reply!

Sorry, I don't think I'm following what you mean.

You got it right and I now understand what you mean.

kubeapps to include a privileged account that is used when the user logs in (very much against this out of the box because it is privilege escalation, but this is also exactly what the getting started gets people to do manually with warnings that it is for demo only)

This is how I expected it to be because it's so common amongst different applications. Just use a plain admin user and password. Other than leaking credentials, there's no danger.

[ghost]

Finally both keycloak clients work fine. Now I'm stuck at the last step where I need to allow users or groups access to namespaces.
https://kubeapps.dev/docs/latest/howto/oidc/oauth2oidc-keycloak/#users
In keycloak I have a user admin so I run

kubectl create clusterrolebinding root-cluster-admin-binding --clusterrole=cluster-admin --user=admin

but in kubeapps I still get the error The current account does not have access to any namespaces.
https://kubernetes.io/docs/reference/access-authn-authz/rbac/#kubectl-create-clusterrolebinding

Can you please tell me if this is the right way to provide the user admin access to all namespaces?

[absoludity]

Hi @kllp . That depends. Normally you would use an email claim for user identity (better uniqueness guarantees) rather than a name claim. Given that you say your user is called admin, I'm guessing that you're expecting to use the name claim on the JWT id_token. You will need to check yourself by inspecting the JWT id_token (as per the debugging instructions).

Once you know what info is on your signed id_token, you can then verify that your cluster is configured both to trust your keycloak identity provider and recognises the same claim that you are expecting. I'd recommend using an email claim, because, as per https://kubernetes.io/docs/reference/access-authn-authz/authentication/#openid-connect-tokens

By default sub [claim is used for user identity], which is expected to be a unique identifier of the end user. Admins can choose other claims, such as email or name, depending on their provider. However, claims other than email will be prefixed with the issuer URL to prevent naming clashes with other plugins.

[ghost]

Thank you very much. I replaced admin with the email and it works! 👍

kubectl create clusterrolebinding root-cluster-admin-binding --clusterrole=cluster-admin --user=admin@example.com

I prefer groups so I granted group access

kubectl delete clusterrolebinding root-cluster-admin-binding
kubectl create clusterrolebinding root-cluster-admin-binding --clusterrole=cluster-admin --group=oidc:admin

The JWT token includes

  "groups": [
    "admin"
  ],

so group-based access should work? How to debug this one? The default oidc-groups-prefix should be oidc:

[ghost]

My browser dev tools show

grpc-message: Forbidden to get the Namespace 'default' due to 'namespaces "default" is forbidden: User "admin@example.com" cannot get resource "namespaces" in API group "" in the namespace "default"'

[absoludity]

That message means exactly what it says: the user admin@example.com cannot get namespaces (which you know, since you deleted the RBAC for that above). So it appears the group check fails silently and the user check is the error reported to the user.

Why do you have the oidc: prefix in your RBAC though? Did you try adding the RBAC without the oidc: prefix? (your user one that worked before deleting was without the prefix). Unless you've specified the --oidc-groups-prefix=oidc arg to your k8s api server, your k8s api server won't be adding a prefix, so:

kubectl create clusterrolebinding root-cluster-admin-binding --clusterrole=cluster-admin --group=admin

If you did already try without a prefix, all I can suggest is comparing your configuration with other tutorials about keycloak with k8s oidc... as I've not used KeyCloak. If you paste a screenshot of your keycloak group, your k8s api settings (if any) your RBAC, and then the output of kubectl auth can-i --as-group ... commands showing the issue, it might be easier to help too! Let us know how you go.

[ghost]

It finally works. Thank you for pointing me to the auth can-i command.

I'm sorry, I misread the api server configuration options table. The third column says Example but I thought these are the default values. Turns out I only needed to add oidc-groups-claim=groups to the api server arguments.

oidc-issuer-url=REDACTED
oidc-client-id=REDACTED
oidc-username-claim=email
oidc-groups-claim=groups
oidc-username-prefix=
oidc-groups-prefix=

Thank you for your support!

[absoludity]

Excellent! Glad you got it working :)