PIP-257: Add Open ID Connect Support to Server Components

[michaeljmarshall]

Discussion: https://lists.apache.org/thread/ttdh74t6v5nznmtw436otk8qdymjvrfk
Vote: https://lists.apache.org/thread/h2d48g7s58w1n6426hxlv9kwrj09kczz
Status: accepted

Motivation

Apache Pulsar does not yet support a server side AuthenticationProvider that implements the Open ID Connect spec for a relying party as defined by https://openid.net/connect/. The only token based authentication is provided via the AuthenticationProviderToken class. Given that we already have clients that implement the OAuth2.0 protocol, which integrates easily with an Open ID Connect AuthenticationProvider, it would be very helpful to add this support to the Pulsar Server components.

Goal

In implementing the OIDC spec, we will fulfill both the core (https://openid.net/specs/openid-connect-core-1_0.html) and the discovery (https://openid.net/specs/openid-connect-discovery-1_0.html) portions of the spec in the AuthenticationProvider implementation.

The end result will be a plugin that:

• supports multiple token issuers

• retrieves the JWKS uri for each issuer from the token issuer's /.well-known/openid-configuration endpoint

• retrieves and caches the JKWS when a client attempts to connect using a token issued by one of the trusted issuers

• refreshes the JWKS after a configured amount of time, which allows for seamless key rotation without needing to restart the proxy, broker, function worker, websocket proxy. (Restarts are still needed to mitigate problems like leaked private keys.)

• verifies that a token's signature and claims are valid

API Changes

There will be two new public classes:

1. Implement AuthenticationProvider with AuthenticationProviderOpenIDConnect.
2. Implement KubernetesFunctionAuthProvider with KubernetesSecretsAuthProvider.

Implementation

Add a module to the apache/pulsar repo named pulsar-openid-connect where we will implement the logic for AuthenticationProviderOpenIDConnect. The core additions will include the ability to discovery the JWKS URI for each token issuer, following the Open ID Connect Discovery protocol, then retrieving and caching the JWKS.

Create a class named KubernetesSecretsAuthProvider that mounts a pre-existing secret into the Pulsar Function pod. This class serves two purposes. First, the function worker does not need to have the credentials used by the function pod. Second, the current authentication flow in the function worker is to forward the authentication data used to create the function. Because OpenID Connect supports short lived tokens, it is not valid to assume that the function pod can operate with the supplied authentication data. Instead, a user will be able to configure the function pod with the client id and the client secret necessary to retrieve the authentication token from the OAuth 2 Authorization Server.

Alternatives

There was an initial attempt to implement this feature here: #11794. The PR was never completed, so it is closed now.

Anything else?

We will need to address the proxy authentication handling in order for users to avoid encountering some of the issues documented here #10816. I plan to follow up with a fix for this issue.

[EronWright]

One of the practical use cases for this is in combination with Kubernetes Service Account Token Projection. We could configure Pulsar to accept OIDC tokens from one or more Kubernetes clusters, based on the discovery feature.

This would allow a Pulsar function pod to authenticate based on their Kubernetes Service Account (KSA) using the standard token plugin. Kubernetes automatically mounts an OIDC token file into the pod, and rotates the token to keep it fresh. I believe that the token plugin reloads the token file automatically. In this scenario, there's no need for KubernetesSecretsAuthProvider.

@michaeljmarshall may I suggest that the new plugin be tested for compatibility with this feature of Kubernetes. In practice, it's very easy to use. There's one caveat that I'm aware of: the token's iss claim should be validated against the value of the issuer field of the disco document, as opposed to a configured value.

Here's a snippet showing how the Kubernetes API server exposes an OIDC discovery endpoint:

$ curl -k https://kubernetes.default.svc/.well-known/openid-configuration

{
    "issuer": "https://container.googleapis.com/v1/projects/xyz/locations/us-central1/clusters/test-oidc",
    "jwks_uri": "https://35.223.170.12:443/openid/v1/jwks",
    "response_types_supported": [
        "id_token"
    ],
    "subject_types_supported": [
        "public"
    ],
    "id_token_signing_alg_values_supported": [
        "RS256"
    ]
}

Here's how a function pod may acquire an OIDC token for a certain audience (i.e. the Pulsar cluster):

apiVersion: v1
kind: Pod
metadata:
  name: my-pulsar-function-0
  namespace: pulsar
spec:
  serviceAccountName: my-pulsar-function
  containers:
  - name: default
    volumeMounts:
    - name: pulsar-serviceaccounttoken
      mountPath: /var/run/secrets/pulsar/serviceaccount
  volumes:
   - name: pulsar-serviceaccounttoken
     projected:
       sources:
       - serviceAccountToken:
         audience: my-pulsar-cluster
         expirationSeconds: 3600
         path: token

[michaeljmarshall]

@EronWright - great suggestions, thank you for sharing! I was actually just looking into the k8s service accounts feature earlier today, and I completely agree that we should add support to make the function pods easily integrate with them. It seems like a great way to make it easier to leverage OIDC without needing to run your Authorization Server.

One reason I chose not to reference the k8s service account feature in this proposal is because these docs (https://kubernetes.io/docs/concepts/security/service-accounts/#authenticating-in-code) indicate that it is recommended to use the TokenReview API (https://kubernetes.io/docs/reference/kubernetes-api/authentication-resources/token-review-v1/) instead of OIDC because the TokenReview API prevents the usage of tokens generated for pods that are terminated.

That being said, relying on the TokenReview API seems like it would introduce latency and could couple the broker to the API Server in an unnecessary way. A short time to live could also limit how useful a token is after a pod terminates.

Another option could be to use the TokenReview API and then to fall back to the OIDC Authentication Provider to allow for better availability in the event of network connectivity issues when connecting to the api server.

What do you think?

[EronWright]

I agree that first-class support for "kubernetes" authentication on both the client and broker would be cool, but the beauty of the OIDC route is three-fold: it teaches the broker to interoperate with many identity providers in a common way, it works well with the token authentication client plugin, which is universally supported in the Pulsar ecosystem, and it works with long-running applications (e.g. functions and flink jobs).

Also, one can control the token TTL in the function pod spec, so you can limit the exposure.

[michaeljmarshall]

Great points about the integration. At this point, I think it makes sense to skip the integration with the TokenReview API. The design could be such that a future addition could add it if deemed necessary/valuable.

Also, one can control the token TTL in the function pod spec, so you can limit the exposure.

Makes sense. Are you thinking we should let the function worker create the service accounts? I hadn't considered that option. In your opinion, how configurable should that be? Creating service accounts requires an increased permission on the function worker within the k8s cluster, which introduces a new risk because a function worker could then potentially be used to create service accounts with excessive permissions.

[michaeljmarshall]

Note: for functions, they will use the SerializableURITokenSupplier for the AuthenticationProviderToken. This supplier is automatically used when the file: prefix starts to the token path, and it results in reading the file from the filesystem each time the token is needed (including when the broker challenges the client upon token expiration).

[michaeljmarshall]

While testing this out in Azure, I discovered that the azure provider requires authentication to retrieve the public keys while AWS does not. That will mean we need to modify #19849 to allow an issue to have authentication. Is it common to protect this kind of endpoint with authentication?

[michaeljmarshall]

For future reference, I checked out some implementations, and below are the claims sections of the JWT created when you specify the expiration and not the audience.

They all appear the same, which is good. I haven't found any k8s documentation that they will appear this way, but it makes sense. One challenge we'll face by integrating with k8s is making sure that we only give permission to the correct pods. This might be challenging. I'll be thinking about this some more.

A sample token from Azure Kubernetes service:

{
  "aud": [
    "https://some.info.azmk8s.io",
    "\"some.info.azmk8s.io\""
  ],
  "exp": 1679444659,
  "iat": 1679437459,
  "iss": "https://some.info.azmk8s.io",
  "kubernetes.io": {
    "namespace": "michael-test",
    "pod": {
      "name": "nginx",
      "uid": "6ad4f05d-64a9-44c7-9acd-fa2f34eb6c96"
    },
    "serviceaccount": {
      "name": "default",
      "uid": "4e618f02-2ce3-45a4-b623-fcc806a16cdf"
    }
  },
  "nbf": 1679437459,
  "sub": "system:serviceaccount:michael-test:default"
}

Sample from EKS:

{
  "aud": [
    "https://kubernetes.default.svc"
  ],
  "exp": 1710969822,
  "iat": 1679433822,
  "iss": "https://oidc.eks.us-east-2.amazonaws.com/id/some-id",
  "kubernetes.io": {
    "namespace": "michael-test",
    "pod": {
      "name": "nginx",
      "uid": "fbac8f9e-a47d-4ad7-a8f0-cc9a65d1331c"
    },
    "serviceaccount": {
      "name": "default",
      "uid": "5964f9d3-3dce-467c-8dbe-d0f463063d7a"
    },
    "warnafter": 1679437429
  },
  "nbf": 1679433822,
  "sub": "system:serviceaccount:michael-test:default"
}

GKE:

{
  "aud": [
    "https://container.googleapis.com/v1/projects/some-info"
  ],
  "exp": 1710973269,
  "iat": 1679437269,
  "iss": "https://container.googleapis.com/v1/projects/some-info",
  "kubernetes.io": {
    "namespace": "michael-test",
    "pod": {
      "name": "nginx",
      "uid": "b75e88ae-753b-4eb8-854e-488c04fc12bc"
    },
    "serviceaccount": {
      "name": "default",
      "uid": "32732d79-3203-4f4a-a343-69215edf1a2a"
    },
    "warnafter": 1679440876
  },
  "nbf": 1679437269,
  "sub": "system:serviceaccount:michael-test:default"
}

[EronWright]

@michaeljmarshall sorry for the delayed response.

About the function pod's service account, I would like to share my thoughts about that. The pod's KSA can be understood as the 'workload identity' of the function. It is the identity that the function will have when talking to Pulsar, and to other cloud services too. For example, a "Cloud Storage Sink" connector (which is a type of function) would naturally authenticate to S3 using its workload identity (see IRSA).

It seems important that the function submitter not be able to escalate their privileges by selecting an arbitrary service account. I feel that the function worker service (e.g. the function mesh) should assign a service account. I would hesistate to automatically create service accounts, because of the complexity of setting up the workload identity.

[EronWright]

On the topic of authorization: as you've shown, the sub claim conveys the name of the KSA, and that would assumedly be used as the Pulsar role name. One would then set Pulsar permissions based on the name, for example in broker.conf:

authorizationEnabled=true
superUserRoles=system:serviceaccount:michael-test:default

Using the KSA name in this way is actually quite convenient, but may benefit from an authorization plugin that would allow for group-based policies, e.g. based on the namespace claim.

Here's how it would work in a multi-k8s-cluster environment. Imagine that the broker was configured to trust the tokens from a number of Kubernetes clusters, collectively known as a cluster-set. By the principle of namespace sameness, a service account with a given name would be considered equivalent across the whole cluster-set. That's a Good Thing, in my opinion.

[michaeljmarshall]

It seems important that the function submitter not be able to escalate their privileges by selecting an arbitrary service account. I feel that the function worker service (e.g. the function mesh) should assign a service account. I would hesistate to automatically create service accounts, because of the complexity of setting up the workload identity.

I completely agree. I had asked the question about this above, but it cannot be correct to let the client select the service account. Further, we

may benefit from an authorization plugin that would allow for group-based policies, e.g. based on the namespace claim.

Yes, I've been thinking that because the tokens have other informative claims, it might make sense to provide a more nuanced way to check authorization with a provider that understands the claims.

That cluster example is very interesting, thanks for sharing!

[michaeljmarshall]

After additional checks, looks like this call consistently fails when no authentication is provided:

$ curl -k https://kubernetes.default.svc/.well-known/openid-configuration
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "forbidden: User \"system:anonymous\" cannot get path \"/.well-known/openid-configuration\"",
  "reason": "Forbidden",
  "details": {},
  "code": 403
}

(The -k is simply used for convenience.)

[michaeljmarshall]

The following works:

curl --cacert /run/secrets/kubernetes.io/serviceaccount/ca.crt \
    -H "Authorization: Bearer $(cat /run/secrets/kubernetes.io/serviceaccount/token)"  \
    https://kubernetes.default.svc/.well-known/openid-configuration

[RobertIndie]

This PIP has been implemented in #19849.
@michaeljmarshall I am closing it now. Feel free to reopen it and move it to the 3.1.0 milestone if any work is left for this PIP.