Concierge seems to try WebhookAuthentication even though it's not supposed to

[isihu]

Hi everybody,

I'm trying to deploy the Concierge by itself on AKS.
I have deployed a JWTAuthenticator.
According to the status, it is valid:

status: 
  conditions:         
    - lastTransitionTime: "2024-06-20T09:04:04Z"    
      message: authenticator initialized                  
      observedGeneration: 3                            
      reason: Success                                
      status: "True"                        
      type: AuthenticatorValid

I'm sending requests to the Concierge via curl putting my JWT in the Authorization header.

curl --cacert impersonation-proxy-ca.pem -X GET \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <MY_JWT>" \
  https://<SERVICE_IP>/api/v1/namespaces

It returns

E0620 13:34:47.802797   40445 memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials
E0620 13:34:47.830089   40445 memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials
E0620 13:34:47.857041   40445 memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials
E0620 13:34:47.885679   40445 memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials
E0620 13:34:47.911855   40445 memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials
error: You must be logged in to the server (the server has asked for the client to provide credentials)

I have raised the log level to all and found these 2 entries stating that a request to the TokenReview API fails:

{"level":"all","timestamp":"2024-06-20T10:49:07.541122Z","caller":"k8s.io/client-go@v0.30.1/rest/request.go:1212$rest.glogBody","message":"Response Body: {\"kind\":\"TokenReview\",\"apiVersion\":\"authentication.k8s.io/v1\",\"metadata\":{\"creationTimestamp\":null,\"managedFields\":[{\"manager\":\"pinniped-concierge\",\"operation\":\"Update\",\"apiVersion\":\"authentication.k8s.io/v1\",\"time\":\"2024-06-20T10:49:07Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:spec\":{\"f:token\":{}}}}]},\"spec\":{\"token\":\"<MY_JWT>\"},\"status\":{\"user\":{},\"error\":\"[invalid bearer token, the server has asked for the client to provide credentials]\"}}","stacktrace":"k8s.io/client-go/rest.glogBody\n\tk8s.io/client-go@v0.30.1/rest/request.go:1212\nk8s.io/client-go/rest.(*Request).transformResponse\n\tk8s.io/client-go@v0.30.1/rest/request.go:1124\nk8s.io/client-go/rest.(*Request).Do.func1\n\tk8s.io/client-go@v0.30.1/rest/request.go:1064\nk8s.io/client-go/rest.(*Request).request.func3.1\n\tk8s.io/client-go@v0.30.1/rest/request.go:1039\nk8s.io/client-go/rest.(*Request).request.func3\n\tk8s.io/client-go@v0.30.1/rest/request.go:1046\nk8s.io/client-go/rest.(*Request).request\n\tk8s.io/client-go@v0.30.1/rest/request.go:1048\nk8s.io/client-go/rest.(*Request).Do\n\tk8s.io/client-go@v0.30.1/rest/request.go:1063\nk8s.io/apiserver/plugin/pkg/authenticator/token/webhook.(*tokenReviewV1Client).Create\n\tk8s.io/apiserver@v0.30.1/plugin/pkg/authenticator/token/webhook/webhook.go:252\nk8s.io/apiserver/plugin/pkg/authenticator/token/webhook.(*WebhookTokenAuthenticator).AuthenticateToken.func1\n\tk8s.io/apiserver@v0.30.1/plugin/pkg/authenticator/token/webhook/webhook.go:138\nk8s.io/apiserver/pkg/util/webhook.WithExponentialBackoff.func1\n\tk8s.io/apiserver@v0.30.1/pkg/util/webhook/webhook.go:125\nk8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtectionWithContext\n\tk8s.io/apimachinery@v0.30.1/pkg/util/wait/wait.go:154\nk8s.io/apimachinery/pkg/util/wait.ExponentialBackoffWithContext\n\tk8s.io/apimachinery@v0.30.1/pkg/util/wait/backoff.go:485\nk8s.io/apiserver/pkg/util/webhook.WithExponentialBackoff\n\tk8s.io/apiserver@v0.30.1/pkg/util/webhook/webhook.go:124\nk8s.io/apiserver/plugin/pkg/authenticator/token/webhook.(*WebhookTokenAuthenticator).AuthenticateToken\n\tk8s.io/apiserver@v0.30.1/plugin/pkg/authenticator/token/webhook/webhook.go:133\nk8s.io/apiserver/pkg/authentication/token/cache.(*cachedTokenAuthenticator).doAuthenticateToken.func1\n\tk8s.io/apiserver@v0.30.1/pkg/authentication/token/cache/cached_token_authenticator.go:206\ngolang.org/x/sync/singleflight.(*Group).doCall.func2\n\tgolang.org/x/sync@v0.7.0/singleflight/singleflight.go:198\ngolang.org/x/sync/singleflight.(*Group).doCall\n\tgolang.org/x/sync@v0.7.0/singleflight/singleflight.go:200"}

{"level":"error","timestamp":"2024-06-20T10:49:07.541275Z","caller":"k8s.io/apiserver@v0.30.1/pkg/endpoints/filters/authentication.go:73$filters.withAuthentication.func1","message":"Unable to authenticate the request","error":"[invalid bearer token, the server has asked for the client to provide credentials]","errorCauses":[{"error":"[invalid bearer token, the server has asked for the client to provide credentials]"}],"stacktrace":"k8s.io/apiserver/pkg/endpoints/filters.withAuthentication.func1\n\tk8s.io/apiserver@v0.30.1/pkg/endpoints/filters/authentication.go:73\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2166\nk8s.io/apiserver/pkg/endpoints/filterlatency.trackStarted.func1\n\tk8s.io/apiserver@v0.30.1/pkg/endpoints/filterlatency/filterlatency.go:84\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2166\nk8s.io/apiserver/pkg/server.DefaultBuildHandlerChain.WithWarningRecorder.func11\n\tk8s.io/apiserver@v0.30.1/pkg/endpoints/filters/warning.go:35\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2166\nk8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1\n\tk8s.io/apiserver@v0.30.1/pkg/server/filters/timeout.go:115"}

I could recreate the error from the TokenReview API via kubectl create -o yaml -f tokenreview.yaml using this as the tokenreview.yaml:

apiVersion: authentication.k8s.io/v1
kind: TokenReview
spec:
  token: <MY_JWT> 

# error
apiVersion: authentication.k8s.io/v1
kind: TokenReview
metadata:
  creationTimestamp: null
spec:
  token: <MY_JWT>
status:
  error: '[invalid bearer token, the server has asked for the client to provide credentials]'
  user: {}

Following the docs, it seems like the Concierge is trying to authenticate the requests via Webhook even though no WebhookAuthenticator is required. Maybe the Kubernetes API Server is used as a default WebhookAuthenticator?

Why doesn't it try to authenticate the JWT via the JWTAuthenticator configured? Do I have to put the JWT somewhere else?

Kind regards

[cfryanr]

Hi @isihu, thanks for asking your question here.

When you configure a JWTAuthenticator, you are configuring it for use by the TokenCredentialRequest API. When you call that API, you send your JWT in the request, and if it successfully authenticates you, then it gives you back an mTLS client certificate in the response (see API docs for TokenCredentialRequest). Then you can use that mTLS client certificate to make calls to the Kubernetes APIs.

This level of detail is usually abstracted by the Pinniped CLI, which will take care of calling the TokenCredentialRequest API on your behalf at all the right times. Or, if you are instead writing a webapp, then the docs for how to formulate these requests in this doc.

If you could share a little more about the bigger picture of what you are trying to accomplish, then we might be able to provide more specific help. What was your original problem that caused you to start debugging at this level of detail?

What you saw in the logs about token reviews is something else. The impersonation proxy will automatically perform token reviews sometimes as part of doing its job. It is not related to WebhookAuthenticators.

[isihu]

Hi @cfryanr,

thanks for your detailed response.

Our situation

We are currently operating a multi-tenant Kubernetes Cluster on AWS.
On AWS, you can enable these flags so the API Server trusts a OIDC provider. Our users use the kubelogin plugin to easily get the right tokens into their kubeconfig. But it's also possible to curl the API Server with the jwt in the Authorization header.

Now we want to add an AKS multitenant cluster and hoped that we can authenticate people in the same way. But for some reason, Azure decided to disable the feature that is literally already implemented in the API Server🥲 . They said that they were going to implement it though which was 2 years ago.
We started looking for alternatives and found the kube-oidc-proxy project and pinniped. The former doesn't seem to be super well maintained, so we decided to try out Pinniped first.
And I did interpret the Pinniped docs in the way that JWTAuthenticators were a perfect replacement for the standard oidc behavior as they are even linking to the API Server documentation:

The JWT authenticator provides the same functionality as the Kubernetes OIDC authentication mechanism, but it is configurable at cluster runtime instead of requiring flags to be set on the kube-apiserver process.

Our tests

When trying it out first on AWS, it worked just as we expected. We were able to send a request to the service of the impersonation proxy with the JWT in the Auth Header and we got a response. We could also change the behavior by changing the RoleBindings. That's why I dug so deep in AKS. It simply worked on AWS.

After your response I tested it out on AWS again. And sending requests to the Concierge worked only when the OIDC provider was configured in the API Server. I didn't even have to configure JWTAuthenticators or WebhookAuthenticators. As stated in my original question, that does make me guess that the Concierge authenticates request by using the TokenReview API of the API Server in some cases. That would also match the behavior of the TokenReview API which only validates a TokenReview with a JWT if the API server trusts the issuer.

Our expectations from Pinniped

I thought that the Authenticators are

• authenticating the user (by checking if the issuer of the JWT is trusted or by Webhook)
• authorizing the user by checking the Roles and RoleBindings themselves or generate a token/cert for that user
• forward the request to the API Server

So basically, we expected the functionality of the Pinniped CLI to be server-side.

How to proceed

There probably isn't a way to configure JWT Authenticators in the way I imagined, is there?
What would probably be possible is to write a little Server that checks the JWT itself, gets the mtls cert from the Concierge and enhances the request that way.

Other than that, I'm not sure how to proceed but I hope I could give you a more precise understanding of what we are trying to accomplish.

[cfryanr]

Thanks for explaining your use case in detail.

The JWT authenticator provides the same functionality as the Kubernetes OIDC authentication mechanism, but it is configurable at cluster runtime instead of requiring flags to be set on the kube-apiserver process.

While this statement from the Pinniped docs is conceptually true, and the Pinniped code for JWTAuthenticator actually uses the same Golang code from Kubernetes to implement this behavior, it is unfortunately not literally exactly the same, in the ways that I mentioned above. The Pinniped CLI will hide most of the differences to allow the end user's experience to be fairly seamless, but there are some key differences under the hood.

When trying it out first on AWS, it worked just as we expected. We were able to send a request to the service of the impersonation proxy with the JWT in the Auth Header and we got a response.

At that point, the impersonation proxy is just passing through your bearer token, and isn't really adding any value. If you can configure the OIDC settings on the API server directly, then you don't really need the Concierge on that cluster.

There probably isn't a way to configure JWT Authenticators in the way I imagined, is there?

No, there isn't. Not at this time.

What would probably be possible is to write a little Server that checks the JWT itself, gets the mtls cert from the Concierge and enhances the request that way.

That could be possible. It could basically do what the Pinniped CLI does, but do it in a server. However, that server would then be holding everyone's mTLS client credentials, presumably in some sort of session storage. It would take some work to write that code, get it correct from a security point of view, and protect those credentials.

Alternatively, you could have your end users install the Pinniped CLI. That seems like a much simpler solution. :) As the cluster owner, you can use the Pinniped CLI to generate kubeconfigs for your end users. Those kubeconfigs have no particular identity or credentials in them, so every end user can use the same kubeconfig for a cluster, which makes them easy to distribute. It's easy for end users to install the Pinniped CLI, e.g. brew install pinniped-cli, or download from GitHub, or distribute your own way. Once installed, they often don't even realize that they're using it, because it gets invoked automatically kubectl and similar client-side tools.

I'll also make a quick plug for the Pinniped Supervisor, in case you had not considered it yet. By using a Pinniped Supervisor (you only need one), you could improve the user experience of your end users (e.g. login once per day and then access all clusters of all types for the rest of the day, support for logging in from jump hosts that have no web browser, etc.) and improve security (e.g. short-lived cluster-specific tokens, user's group memberships are updated frequently, user's permission to continue their sessions are checked frequently, kubeconfigs never contain credentials, etc.). On your AWS clusters, you could simply configure the API server to trust JWTs issued by the Pinniped Supervisor. On your AKS clusters, you could use the Concierge with a JWTAuthenticator configured to trust JWTs issued by the Pinniped Supervisor. Your users would need the Pinniped CLI, which would help make everything seamless and appear to work the same across all these cluster types. And the Supervisor always gets user authentication and identities from an external provider of your choice (OIDC, LDAP, AD, or GitHub).

[isihu]

At that point, the impersonation proxy is just passing through your bearer token, and isn't really adding any value. If you can configure the OIDC settings on the API server directly, then you don't really need the Concierge on that cluster.

Ahh, of course. That's why it worked 😄

I'll also make a quick plug for the Pinniped Supervisor

For now, I'll just try to replicate the behavior on Azure with as little effort as possible. But when there's time for a redesign, maybe to streamline authentication to both cloud providers, Pinniped seems like a very good choice.

Thanks for all the information!