OIDC authentication and JWT authorization is unstable

[plnordquist]

Description:

When attempting to use OIDC authentication and pass that down to the new JWT authorization feature in v1.2.0, Envoy Gateway seems to fail to program the Envoy Proxy in a stable way. If I apply my Security Policy for an HTTP Route as a whole unit, most of the time I see errors around fetching the JWT public key with the message OAuth flow failed in the browser. Another time I only saw the OAuth flow failed without a corresponding reason in the Envoy Proxy log. If I apply the Security Policy with the oidc section and then add the jwt section and finally add the authorization section, I am able to create a working configuration that appropriately authenticates and authorizes the returned JWT token.

If more logs are required, I can attempt to deploy a minikube and grab logs there.

Repro steps:

Sample resources:

---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: prometheus
  namespace: monitoring
spec:
  hostnames:
  - prometheus.example.com
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: example
  rules:
  - backendRefs:
    - group: ""
      kind: Service
      name: prometheus
      port: 9090
      weight: 1
    matches:
    - path:
        type: PathPrefix
        value: /
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: SecurityPolicy
metadata:
  name: prometheus
  namespace: monitoring
spec:
  authorization:
    rules:
    - action: Allow
      name: allow
      principal:
        jwt:
          claims:
          - name: groups
            valueType: StringArray
            values:
            - foobar
          provider: exjwt
  jwt:
    providers:
    - claimToHeaders:
      - claim: email
        header: x-user-email
      extractFrom:
        cookies:
        - IdToken
      issuer: https://oidc.example.com/auth/realms/example
      name: exjwt
      remoteJWKS:
        uri: https://oidc.example.com/auth/realms/example/protocol/openid-connect/certs
  oidc:
    clientID: prometheus
    clientSecret:
      group: ""
      kind: Secret
      name: prometheus-oidc
    cookieNames:
      idToken: IdToken
    provider:
      issuer: https://oidc.example.com/auth/realms/example
    scopes:
    - openid
    - email
    - profile
  targetRefs:
  - group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: prometheus

Environment:

Envoy Gateway v1.2.1
Envoy Proxy v1.32.1

Relevant Logs:

In Envoy Proxy when deploying as a whole unit:

[2024-11-11 21:42:19.638][1][error][filter] [source/extensions/filters/http/common/jwks_fetcher.cc:51] fetch: fetch pubkey [uri = https://oidc.example.com/auth/realms/example/protocol/openid-connect/certs] failed: [cluster = oidc_example_com_443] is not configured
[2024-11-11 21:42:19.638][1][warning][jwt] [source/extensions/filters/http/jwt_authn/jwks_async_fetcher.cc:115] Jwks async fetching url=https://oidc.example.com/auth/realms/example/protocol/openid-connect/certs: failed

In Envoy Gateway when deploying as a whole unit:

2024-11-11T21:42:19.625Z	ERROR	xds-translator	runner/runner.go:85	failed to translate xds ir	{"runner": "xds-translator", "error": "xds cluster exists\nxds cluster exists"}
2024-11-11T21:42:19.625Z	ERROR	watchable	message/watchutil.go:56	observed an error	{"runner": "xds-translator", "error": "xds cluster exists\nxds cluster exists"}

[arkodg]

cc @zhaohuabing