Skip to content

Commit 9207cf0

Browse files
mstrukmstruk
committed
OAuth metrics
Signed-off-by: Marko Strukelj <marko.strukelj@gmail.com>
1 parent ac8ee9c commit 9207cf0

File tree

87 files changed

+4719
-331
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

87 files changed

+4719
-331
lines changed

‎.travis.yml

+1
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,7 @@ addons:
2929
- keycloak
3030
- hydra
3131
- hydra-jwt
32+
- mockoauth
3233
env:
3334
global:
3435
- PULL_REQUEST=${TRAVIS_PULL_REQUEST}

‎README.md

+167
Original file line numberDiff line numberDiff line change
@@ -49,6 +49,7 @@ Strimzi Kafka OAuth modules provide support for OAuth2 as authentication mechani
4949
- [Configuring the Kafka client with SASL/PLAIN](#configuring-the-kafka-client-with-saslplain)
5050
- [Configuring the TLS truststore](#configuring-the-tls-truststore)
5151
- [Configuring the network timeouts for communication with authorization server](#configuring-the-network-timeouts-for-communication-with-authorization-server)
52+
- [Configuring the metrics](#configuring-the-metrics)
5253
- [Demo](#demo)
5354

5455
<!-- /TOC -->
@@ -347,6 +348,11 @@ You can control the minimum pause between two consecutive scheduled keys refresh
347348

348349
All access tokens can be invalidated by rotating the keys on authorization server and expiring old keys.
349350

351+
During the Kafka broker startup, a request to the JWKS endpoint immediately tries to load the keys.
352+
If JWKS keys can not be loaded or can not be successfully parsed during startup, the Kafka broker will exit.
353+
That behaviour can be turned off:
354+
- `oauth.fail.fast` (e.g.: "false" - it is "true" by default)
355+
350356
###### Validation using the introspection endpoint
351357

352358
When your authorization server is configured to use opaque tokens (not JWT) or if it does not expose JWKS endpoint, you have no other option but to use the introspection endpoint.
@@ -1109,6 +1115,167 @@ You may want to set these options globally as system properties or env vars to a
11091115

11101116
NOTE: These options are available since version 0.10.0. Before, one could only apply JDK network options `sun.net.client.defaultConnectTimeout`, and `sun.net.client.defaultReadTimeout` as described [here](https://docs.oracle.com/javase/8/docs/technotes/guides/net/properties.html), and the default was `no timeout`.
11111117

1118+
Configuring the metrics
1119+
-----------------------
1120+
1121+
By default, the gathering and exporting of metrics is disabled. Metrics are available to get an insight into the performance and failures during token validation, authorization operations and client authentication to the authorization server. You can also monitor the authorization server requests by background services such as refreshing of JWKS keys and refreshing of grants when `KeycloakRBACAuthorizer` is used.
1122+
1123+
You can enable metrics for token validation on the Kafka broker or for client authentication on the client by setting the following JAAS option to `true`:
1124+
- `oauth.enable.metrics` (e.g.: "true")
1125+
1126+
You can enable metrics for `KeycloakRBACAuthorizer` by setting an analogous option in Kafka broker's `server.properties` file:
1127+
- `strimzi.authorization.enable.metrics` (e.g.: "true")
1128+
1129+
If `OAUTH_ENABLE_METRICS` env variable is set or if `oauth.enable.metrics` system property is set, that will both also enable the metrics for `KeycloakRBACAuthorizer`.
1130+
1131+
If `oauth.config.id` is specified in JAAS configuration of the listener or the client, it will be available in MBean / metric name as `contextId` attribute. If not specified, it will be calculated from JAAS configuration for the validator or default to `client` in client JAAS config, or `keycloak-authorizer` for KeycloakRBACAuthorizer metrics.
1132+
1133+
Metrics are exposed through JMX managed beans. They can also be exposed as Prometheus metrics by using the Prometheus JMX Exporter agent, and mapping the JMX metrics names to prometheus metrics names.
1134+
1135+
When metrics are enabled, managed beans are registered on demand, containing the attributes that are easily translated into Prometheus metrics.
1136+
1137+
Each registered MBean contains two variables - `count`, and `timeTotal`.
1138+
The `count` contains the number of requests of the specific context (as represented by attributes).
1139+
The `totalTime` contains the cumulative time in milliseconds spent in the requests of the specific context.
1140+
1141+
Two reads of `count` and `timeTotal` allow to calculate the average request time within the time interval as `delta Time / delta Count`.
1142+
1143+
The following MBeans are registered, depending on which parts of `strimzi-kafka-oauth` are in use.
1144+
1145+
For fast local JWT token based validation there are:
1146+
1147+
- The metrics for validation requests which occur as part of the authentication:
1148+
- `strimzi.oauth:name=validation_requests,context=$CONFIG_ID,mechanism=$MECHANISM,type=jwks,host="$HOST:$PORT",path="$JWKS_ENDPOINT_PATH",outcome=success`
1149+
- `strimzi.oauth:name=validation_requests,context=$CONFIG_ID,mechanism=$MECHANISM,type=jwks,host="$HOST:$PORT",path="$JWKS_ENDPOINT_PATH",outcome=error,error_type=$ERROR_TYPE`
1150+
1151+
- The metrics for http requests to JWKS endpoint of the authorization server that are periodically performed in the background:
1152+
- `strimzi.oauth:name=http_requests,context=$CONFIG_ID,type=jwks,host="$HOST:$PORT",path="$JWKS_ENDPOINT_PATH",outcome=success,status=200`
1153+
- `strimzi.oauth:name=http_requests,context=$CONFIG_ID,type=jwks,host="$HOST:$PORT",path="$JWKS_ENDPOINT_PATH",outcome=error,error_type=http,status=$STATUS`
1154+
1155+
For introspection based validation there are:
1156+
1157+
- The metrics for validation requests which occur as part of the authentication:
1158+
- `strimzi.oauth:name=validation_requests,context=$CONFIG_ID,mechanism=$MECHANISM,type=introspect,host="$HOST:$PORT",path="$INTROSPECTION_ENDPOINT_PATH",outcome=success`
1159+
- `strimzi.oauth:name=validation_requests,context=$CONFIG_ID,mechanism=$MECHANISM,type=introspect,host="$HOST:$PORT",path="$INTROSPECTION_ENDPOINT_PATH",outcome=error,error_type=$ERROR_TYPE`
1160+
1161+
- The metrics for http requests to introspect endpoint during validation requests:
1162+
- `strimzi.oauth:name=http_requests,context=$CONFIG_ID,type=introspect,host="$HOST:$PORT",path="$INTROSPECTION_ENDPOINT_PATH",outcome=success,status=200`
1163+
- `strimzi.oauth:name=http_requests,context=$CONFIG_ID,type=introspect,host="$HOST:$PORT",path="$INTROSPECTION_ENDPOINT_PATH",outcome=error,error_type=http,status=$STATUS`
1164+
1165+
- The metrics for http requests to userinfo endpoint during validation requests (if configured):
1166+
- `strimzi.oauth:name=http_requests,context=$CONFIG_ID,type=userinfo,host="$HOST:$PORT",path="$USERINFO_ENDPOINT_PATH",outcome=success,status=200`
1167+
- `strimzi.oauth:name=http_requests,context=$CONFIG_ID,type=userinfo,host="$HOST:$PORT",path="$USERINFO_ENDPOINT_PATH",outcome=error,error_type=http,status=$STATUS`
1168+
1169+
For validation performed using OAuth over PLAIN there are additionally:
1170+
1171+
- The metrics for http requests to obtain the access token in client's name:
1172+
- `strimzi.oauth:name=http_requests,context=$CONFIG_ID,type=plain,host="$HOST:$PORT",path="$TOKEN_ENDPOINT_PATH",outcome=success,status=200`
1173+
- `strimzi.oauth:name=http_requests,context=$CONFIG_ID,type=plain,host="$HOST:$PORT",path="$TOKEN_ENDPOINT_PATH",outcome=error,error_type=http,status=$STATUS`
1174+
1175+
For Keycloak authorization there are:
1176+
1177+
- The metrics for authorization requests:
1178+
- `strimzi.oauth:name=authorization_requests,context=$CONFIG_ID,type=keycloak-authorization,host="$HOST:$PORT",path="$TOKEN_ENDPOINT_PATH",outcome=success`
1179+
- `strimzi.oauth:name=authorization_requests,context=$CONFIG_ID,type=keycloak-authorization,host="$HOST:$PORT",path="$TOKEN_ENDPOINT_PATH",outcome=error,error_type=$ERROR_TYPE`
1180+
1181+
- The metrics for http requests to retrieve or refresh grants for the authenticated user:
1182+
- `strimzi.oauth:name=http_requests,context=$CONFIG_ID,type=grants,host="$HOST:$PORT",path="$TOKEN_ENDPOINT_PATH",outcome=success,status=200`
1183+
- `strimzi.oauth:name=http_requests,context=$CONFIG_ID,type=grants,host="$HOST:$PORT",path="$TOKEN_ENDPOINT_PATH",outcome=error,error_type=http,status=$STATUS`
1184+
1185+
For client-side authentication there are:
1186+
1187+
- The metrics for client authentication requests:
1188+
- `strimzi.oauth:name=authentication_requests,context=$CONFIG_ID,type=client-auth,host="$HOST:$PORT",path="$TOKEN_ENDPOINT_PATH",outcome=success`
1189+
- `strimzi.oauth:name=authentication_requests,context=$CONFIG_ID,type=client-auth,host="$HOST:$PORT",path="$TOKEN_ENDPOINT_PATH",outcome=error,error_type=$ERROR_TYPE`
1190+
1191+
- The metrics for http requests to obtain the access token:
1192+
- `strimzi.oauth:name=http_requests,context=$CONFIG_ID,type=client-auth,host="$HOST:$PORT",path="$TOKEN_ENDPOINT_PATH",outcome=success,status=200`
1193+
- `strimzi.oauth:name=http_requests,context=$CONFIG_ID,type=client-auth,host="$HOST:$PORT",path="$TOKEN_ENDPOINT_PATH",outcome=error,error_type=http,status=$STATUS`
1194+
1195+
1196+
1197+
### Using the metrics with Prometheus
1198+
1199+
The [metrics-config.json](testsuite/docker/kafka/config/metrics-config.json) file contains the definitions for mapping the metrics from MBeans to Prometheus metrics names.
1200+
1201+
The following metrics are exposed as the result of the mapping.
1202+
1203+
For fast local JWT token based validation there are:
1204+
1205+
- The metrics for validation requests which occur as part of the authentication:
1206+
- `strimzi_oauth_validation_requests_count{type="jwks"}`
1207+
- `strimzi_oauth_validation_requests_timetotal{type="jwks"}`
1208+
1209+
- The metrics for http requests to JWKS endpoint of the authorization server that are periodically performed in the background:
1210+
- `strimzi_oauth_http_requests_count{type="jwks"}`
1211+
- `strimzi_oauth_http_requests_timetotal{type="jwks}`
1212+
1213+
For introspection based validation there are:
1214+
1215+
- The metrics for validation requests which occur as part of the authentication:
1216+
- `strimzi_oauth_validation_requests_count{type="introspect"}`
1217+
- `strimzi_oauth_validation_requests_timetotal{type="introspect"}`
1218+
1219+
- The metrics for http requests to introspect endpoint during validation requests:
1220+
- `strimzi_oauth_http_requests_count{type="introspect"}`
1221+
- `strimzi_oauth_http_requests_timetotal{type="introspect"}`
1222+
1223+
- The metrics for http requests to userinfo endpoint during validation requests (if configured):
1224+
- `strimzi_oauth_http_requests_count{type="userinfo"}`
1225+
- `strimzi_oauth_http_requests_timetotal{type="userinfo"}`
1226+
1227+
For validation performed using OAuth over PLAIN there are additionally:
1228+
1229+
- The metrics for http requests to obtain the access token in client's name:
1230+
- `strimzi_oauth_http_requests_count{type="plain"}`
1231+
- `strimzi_oauth_http_requests_timetotal{type="plain"}`
1232+
1233+
For Keycloak authorization there are:
1234+
1235+
- The metrics for authorization requests:
1236+
- `strimzi_oauth_authorization_requests_count{type="keycloak-authorization"}`
1237+
- `strimzi_oauth_authorization_requests_timetotal{type="keycloak-authorization"}`
1238+
1239+
- The metrics for http requests to retrieve or refresh grants for the authenticated user:
1240+
- `strimzi_oauth_http_requests_count{type="keycloak-authorization"}`
1241+
- `strimzi_oauth_http_requests_timetotal{type="keycloak-authorization"}`
1242+
1243+
For client-side authentication there are:
1244+
1245+
- The metrics for client authentication requests:
1246+
- `strimzi_oauth_authentication_requests_count{type="client-auth"}`
1247+
- `strimzi_oauth_authentication_requests_timetotal{type="client-auth"}`
1248+
1249+
- The metrics for http requests to obtain the access token:
1250+
- `strimzi_oauth_http_requests_count{type="client-auth"}`
1251+
- `strimzi_oauth_http_requests_timetotal{type="client-auth"}`
1252+
1253+
1254+
### Some examples of PromQL queries
1255+
1256+
- Get the average request time in ms during the last minute across all http requests to a specific authorization server:
1257+
```
1258+
rate(strimzi_oauth_http_requests_timetotal{host="sso:443"}[1m] / rate(strimzi_oaouth_http_requests_count{host="sso:443"}[1m]
1259+
```
1260+
Note, that this gives reliable results if scrape period is 15 seconds. If scrape period is longer, you should use a multiple of 4 as a minimum time span to average across. For example, if scrape period is 30 seconds, use 2m span instead of 1m.
1261+
1262+
- Get the number of http requests in the last minute:
1263+
```
1264+
idelta(strimzi_oauth_http_requests_count[1m])
1265+
```
1266+
1267+
- Get the network error counts across all http requests in the last minute:
1268+
```
1269+
idelta(strimzi_oauth_http_requests_count{outcome="error",error_type="connect"}[1m])
1270+
```
1271+
1272+
- Get the TLS error counts across all http requests in the last minute:
1273+
```
1274+
idelta(strimzi_oauth_http_requests_count{outcome="error",error_type="tls"}[1m])
1275+
```
1276+
1277+
1278+
11121279
Demo
11131280
----
11141281

‎audience-realm-composed.json

+197
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,197 @@
1+
{
2+
"realm": "scope-test",
3+
"accessTokenLifespan": 2592000,
4+
"ssoSessionMaxLifespan": 32140800,
5+
"ssoSessionIdleTimeout": 32140800,
6+
"enabled": true,
7+
"sslRequired": "external",
8+
"users": [
9+
{
10+
"username": "admin",
11+
"enabled": true,
12+
"email": "admin@example.com",
13+
"credentials": [
14+
{
15+
"type": "password",
16+
"value": "admin-password"
17+
}
18+
],
19+
"realmRoles": [
20+
"admin"
21+
],
22+
"clientRoles": {
23+
"realm-management": [
24+
"realm-admin"
25+
],
26+
"kafka": [
27+
"kafka-admin"
28+
]
29+
}
30+
},
31+
{
32+
"username": "service-account-kafka-broker",
33+
"enabled": true,
34+
"email": "service-account-kafka-broker@placeholder.org",
35+
"serviceAccountClientId": "kafka-broker",
36+
"clientRoles": {
37+
"kafka" : ["kafka-admin"]
38+
}
39+
},
40+
{
41+
"username": "service-account-kafka-client",
42+
"enabled": true,
43+
"serviceAccountClientId": "kafka-client",
44+
"realmRoles": [
45+
"default-roles-audience"
46+
]
47+
}
48+
],
49+
"roles": {
50+
"realm": [
51+
{
52+
"name": "user",
53+
"description": "User privileges"
54+
},
55+
{
56+
"name": "admin",
57+
"description": "Administrator privileges"
58+
}
59+
],
60+
"client": {
61+
"kafka": [],
62+
"kafka-client": []
63+
}
64+
},
65+
"scopeMappings": [
66+
{
67+
"client": "kafka-broker",
68+
"roles": [
69+
"offline_access"
70+
]
71+
},
72+
{
73+
"client": "kafka-client",
74+
"roles": [
75+
"offline_access"
76+
]
77+
},
78+
{
79+
"clientScope": "offline_access",
80+
"roles": [
81+
"offline_access"
82+
]
83+
}
84+
],
85+
"clientScopeMappings": {
86+
"kafka": [
87+
{
88+
"client": "kafka-broker",
89+
"roles": [
90+
"kafka-admin"
91+
]
92+
}
93+
]
94+
},
95+
"clientScopes": [
96+
{
97+
"name": "test",
98+
"description": "Custom test scope",
99+
"protocol": "openid-connect",
100+
"attributes": {
101+
"include.in.token.scope": "true",
102+
"display.on.consent.screen": "false"
103+
}
104+
},
105+
{
106+
"name": "offline_access",
107+
"description": "OpenID Connect built-in scope: offline_access",
108+
"protocol": "openid-connect",
109+
"attributes": {
110+
"consent.screen.text": "${offlineAccessScopeConsentText}",
111+
"display.on.consent.screen": "true"
112+
}
113+
},
114+
{
115+
"name": "profile",
116+
"description": "OpenID Connect built-in scope: profile",
117+
"protocol": "openid-connect",
118+
"attributes": {
119+
"include.in.token.scope": "true",
120+
"display.on.consent.screen": "true",
121+
"consent.screen.text": "${profileScopeConsentText}"
122+
},
123+
"protocolMappers": [
124+
{
125+
"name": "username",
126+
"protocol": "openid-connect",
127+
"protocolMapper": "oidc-usermodel-property-mapper",
128+
"consentRequired": false,
129+
"config": {
130+
"userinfo.token.claim": "true",
131+
"user.attribute": "username",
132+
"id.token.claim": "true",
133+
"access.token.claim": "true",
134+
"claim.name": "preferred_username",
135+
"jsonType.label": "String"
136+
}
137+
}
138+
]
139+
}
140+
]
141+
"clients": [
142+
{
143+
"clientId": "kafka",
144+
"enabled": true,
145+
"publicClient": true,
146+
"bearerOnly": false,
147+
"standardFlowEnabled": false,
148+
"implicitFlowEnabled": false,
149+
"directAccessGrantsEnabled": false,
150+
"serviceAccountsEnabled": false,
151+
"consentRequired" : false,
152+
"fullScopeAllowed" : false
153+
},
154+
{
155+
"clientId": "kafka-broker",
156+
"enabled": true,
157+
"clientAuthenticatorType": "client-secret",
158+
"secret": "kafka-broker-secret",
159+
"publicClient": false,
160+
"bearerOnly": false,
161+
"standardFlowEnabled": false,
162+
"implicitFlowEnabled": false,
163+
"directAccessGrantsEnabled": true,
164+
"serviceAccountsEnabled": true,
165+
"consentRequired" : false,
166+
"fullScopeAllowed" : false,
167+
"attributes": {
168+
"access.token.lifespan": "32140800"
169+
}
170+
},
171+
{
172+
"clientId": "kafka-client",
173+
"enabled": true,
174+
"clientAuthenticatorType": "client-secret",
175+
"secret": "kafka-client-secret",
176+
"bearerOnly": false,
177+
"consentRequired": false,
178+
"standardFlowEnabled": false,
179+
"implicitFlowEnabled": false,
180+
"directAccessGrantsEnabled": true,
181+
"serviceAccountsEnabled": true,
182+
"publicClient": false,
183+
"attributes": {
184+
"access.token.lifespan": "32140800"
185+
},
186+
"fullScopeAllowed": true,
187+
"nodeReRegistrationTimeout": -1,
188+
"defaultClientScopes": [
189+
"profile"
190+
],
191+
"optionalClientScopes": [
192+
"test",
193+
"offline_access"
194+
]
195+
}
196+
]
197+
}

0 commit comments

Comments
 (0)