JWT Validation configuration broken in RC 1.16

[mr-miles]

Overview of the Issue

I am very excited about the JWT validation in the upcoming release, and tried out the feature based on the docs.

I have an existing service which is configured to validate tokens from Azure AD and that is working fine. I wanted to move the validation into envoy and out of the service but have hit some hurdles.

1. I added jwt-provider config entry and service-intentions but got this error

2023-06-26T19:20:23.779Z+00:00 [warning] envoy.config(14) DeltaAggregatedResources gRPC config stream to consul-dataplane closed since 58s ago: 14, failed to generate all xDS resources from the snapshot: failed to generate xDS resources for "type.googleapis.com/envoy.config.listener.v3.Listener": provider specified in intention does not exist. Provider name: aad

This was solved by manually attaching a policy with mesh:read permission to the token set up for the account.

My setup is running in k8s via the helm chart, and the policies are automatically attached to the service token - how do I go about attaching this permission to a service token by default, rather than as a one-off?

2. I configured the provider to use remote jwks which it should be able to download but I then got this error:

2023-06-26T19:22:20.525Z+00:00 [error] envoy.filter(14) fetch: fetch pubkey [uri = https://login.microsoftonline.com/<guid>/discovery/v2.0/keys] failed: [cluster = jwks_cluster] is not configured
2023-06-26T19:22:20.525Z+00:00 [warning] envoy.jwt(14) Jwks async fetching url=https://login.microsoftonline.com/<guid>/discovery/v2.0/keys: failed

JWTProvider config looks like this:

{
	"Kind": "jwt-provider",
	"Name": "aad",
	"JSONWebKeySet": {
		"Remote": {
			"URI": "https://login.microsoftonline.com/<guid>/discovery/v2.0/keys",
			"RequestTimeoutMs": 1500,
			"RetryPolicy": {
				"NumRetries": 5
			}
		}
	},
	"Issuer": "https://login.microsoftonline.com/<guid>/v2.0",
	"Audiences": [
		"<audience>"
	],
	"Locations": [
		{
			"Header": {
				"Name": "Authorization",
				"ValuePrefix": "Bearer"
			}
		}
	],
	"Forwarding": {
		"HeaderName": "X-Consul-Jwt",
		"PadForwardPayloadHeader": false
	},
	"ClockSkewSeconds": 30,
	"CacheConfig": {
		"Size": 100
	}
}

Generated envoy config from a config dump looks like this:

{
              "name": "envoy.filters.http.jwt_authn",
              "typed_config": {
               "@type": "type.googleapis.com/envoy.extensions.filters.http.jwt_authn.v3.JwtAuthentication",
               "providers": {
                "aad": {
                 "issuer": "https://login.microsoftonline.com/<guid>/v2.0",
                 "audiences": [
                  "<audience>"
                 ],
                 "remote_jwks": {
                  "http_uri": {
                   "uri": "https://login.microsoftonline.com/<guid>/discovery/v2.0/keys",
                   "cluster": "jwks_cluster",
                   "timeout": "1s"
                  },
                  "async_fetch": {},
                  "retry_policy": {
                   "num_retries": 5
                  }
                 },
                 "from_headers": [
                  {
                   "name": "Authorization",
                   "value_prefix": "Bearer"
                  }
                 ],
                 "forward_payload_header": "X-Consul-Jwt",
                 "payload_in_metadata": "jwt_payload_aad"
                }
               },
               "rules": [
                {
                 "match": {
                  "prefix": "/"
                 },
                 "requires": {
                  "provider_name": "aad"
                 }
                }
               ]
              }
             },

However jwks_cluster does not exist in the envoy configuration at all.

I have not tried out a local jwks as a workaround.

I have verified that the jwks url is trusted and does return a valid jwks.

Do you expect that cluster to be statically configured in proxy-defaults (say) or should it be being received via xds by default?
Relatedly, what happens when there is more than one jwt provider since I would have expected the cluster name to look less constant

Operating system and Environment details

AWS EKS
Consul 1.16-rc1 via helm chart

[pglass]

Hey, thanks for trying this out and sorry you aren't able to use this feature right now. It looks like you are configuring things correctly, but we have oversights in the implementation.

We're working on the following fixes for the next Consul patch release:

• Allow reading JWTProvider config if the token has any service:write permission. This means mesh:read will not be necessary for sidecar proxies to get the necessary configuration over xDS.
• Generate an Envoy cluster for each remote JWKS URI over xDS. This means you will not need to manually configure clusters, and Consul will support multiple remote JWKS clusters.
• Add additional settings to configure the Envoy clusters (particularly TLS settings).
• Add additional thorough testing

Until the next patch release, you should be able to use a local JWKS and add the mesh:read permission to continue evaluating the feature. We appreciate any additional feedback you have!

[roncodingenthusiast]

This issue has now been fixed in consul 1.16.1