PoC: Metrics Access Control

[david-martin]

Use Case

As a platform administrator, I want to allow Team A to access only the metrics and views for their own APIs/HTTPRoutes.

• Enables secure App/API Developer dashboards and visualizations
• Will likely be a component that encompasses/manages the kube-rbac-proxy and prom-label-proxy components. It would be configured with the location of metrics & permissions to read all metrics.

Tools:

• Kube-rbac-proxy - Authenticates requests and enforces Kubernetes RBAC - https://github.com/brancz/kube-rbac-proxy
• Prom-label-proxy - Enforces label-based filtering on Prometheus queries - https://github.com/prometheus-community/prom-label-proxy

Approach:

• kube-rbac-proxy in front of Thanos Query to handle authentication and authorization
  prom-label-proxy between kube-rbac-proxy and Thanos to enforce label-based access control
• Add namespace labels to metrics emitted by shared components (Istio Gateway, Kuadrant) to identify the originating namespace.
• In istio metrics, this would be the destination_service_namepsace label on istio_requests_total metric
• Verify that we can expose a Prometheus data source that filters shared Istio metrics based on user permissions

[david-martin]

Update from 13th December

Myself, @adam-cattermole and @R-Lawton were investigating some metrics access control to satisfy this use case
"As a platform administrator, I want to allow Team A to access only the metrics and views for their own APIs/HTTPRoutes."
After getting a better understanding of kube-rbac-proxy, and prom-label-proxy, we had some open questions about how we'd pass auth info to the rbac proxy, particularly if the request comes from within an openshfit console for the console plugin.
Openshift uses these components already for metrics tenancy, but the api is served from the same url as the console, so the same auth for the console can be used by the rbac proxy.
If we run our our rbac proxy, then we'd need an auth step in the browser (which would make it not seamless at all).
It should be possible in grafana to use the same auth provider as the rbac proxy, then use a 'Forward oauth identity' option to pass auth headers to the data source (i.e. the rbac proxy in front of thanos/prometheus), but I think we should try solve for the console first.
After gaining more understanding of how this all works, we agreed that some good next steps would be to learn more about how the openshift console metrics proxy works, in particular if we have 2 different users with their own app in separate namespaces, and no perms on each others namespace.
Like, would they even be able to query the istio_requests_total metric, and what kind of label replacing would happen.
We have options to do label replacing ourself (for the destination_service_namespace label) if that means we can just leverage what's there in openshift to give tenancy of metrics.

[david-martin]

Notes from some follow up work on 16th Dec

I set up a couple users in a cluster, and a new gateway.
There's 2 users: testuser01 & testuser02
They are htpasswd users, with just admin permissions at the moment in their own namespace/project (called the same as their username).
To generate some traffic (no auth or rl):

curl -k --write-out '%{http_code}\n' --silent --output /dev/null https://testuser01.dm.hcpapps.net/cars

curl -k --write-out '%{http_code}\n' --silent --output /dev/null https://testuser02.dm.hcpapps.net/cars

Few things I've noticed so far.

• Even if logged in as an admin, you can use the Developer view, then go to the Observe > Metrics menu to get namespace filtered metrics. This means you can login as testuser01, testuser02 or even as yourself (a cluster-admin) to test the Observe behaviour. The project dropdown determines what namespace label is applied to queries (by the label proxy).
• While logged in as myself (cluster-admin), i can see results for istio_requests_total when the project is set to that of the gateway api-gateway-testusers. If i change that to any other project, I get no results.
• While logged in as testuser01, i only have access to the testuser01 project. Therefore I don't see any results for that metric.

The label proxy replaces the namespace label on queries with whatever is set in the project dropdown.
The istio_requests_total metric has a namespace label of the gateway it's in i.e. api-gateway-testusers
That gets set automatically based on the PodMonitor being in the same namespace as the gateway pod (and user workload monitoring only allows monitors to scrape things from the same namespace as them).
I was thinking about relabeling to allow testuser01 to see the metrics.
I'm not sure yet if that's possible, or if user workload monitoring would even allow changing the namespace label.

...

if you try set the namespace differently than the project selector, you get this error

If we have a label proxy that enforces an exported_namespace label,
combined with a relabel config on the istio pod monitor i.e.

      metricRelabelings:
        - sourceLabels: ["destination_service_namespace"]
          targetLabel: "exported_namespace"
          action: replace

we might be able to get some sort of tenancy working.
Luckily, the majority of other metrics used (the state ones like gatewayapi_httproute_labels and kuadrant_dnspolicy_created ) already have an exported_namespace set, which is the original namespace rather than the namespace of the prometheus instance.
I'm not sure how useful this is just yet, as we still would have the auth issue as we'd have to direct any openshift console requests to our proxy(ies).
However, it might be useful outside the console, like in grafana.
It would be quite cumbersome, though scriptable, to have a data source in grafana per user/namespace, each with a label proxy of it's own.
Then each user would only have query access to their data source.