Multi-tenant superuser support

[lukipro]

Is your feature request related to a problem? Please describe.
Suppose I have Grafana Loki installed in multi tenant mode.
Grafana is configured to use generic OAuth.
In Grafana I have several dashboards connected to a single Loki datasource. This Loki datasource is actually behind a proxy which adds an appropriate 'X-Scope-OrgId' header to the request based on the contents of the forwarded OAuth token. So far so good - I can implement some logic in the proxy to map specific OAuth users to particular X-Scope-OrgId's.
Every user can query Loki for logs belonging to his role/X-Scope-OrgId.

However, there is no super user option in Loki in multi tenant mode.

Describe the solution you'd like
I would like to have an option to query all tenants in Loki when a user with admin role is authenticated by the OAuth provider. Can we have a dedicated header for this? Is this solution even allowed from the multi tenant point of view?

Describe alternatives you've considered
Running multiple Loki's in single-tenant mode. Each Loki accessible through its own Grafana datasource and attached to particular organization in Grafana.
This way a single Grafana admin could access all organization's Loki's.

Additional context
Add any other context or screenshots about the feature request here.

[slim-bean]

For posterity there were a few different discussions about this in a few different slack rooms but out of that came a couple ideas:

• The ability to pass in a list of tenant ID's to the query frontend which would issue these queries mostly like it does now with same parallelization but aggregating the result from all tenants into a single result.
• Run Loki in single tenant and have an upstream proxy inject a tenant label into writes and on reads add the tenant label with appropriate values to queries.

These were ideas intended to minimize changes to the underlying Loki and Cortex write/query code but I'm sure there are options there as well.

There are also some pros/cons to both these approaches however the second option could be implemented on Loki today.

Both of these options would have issues scaling to tens or hundreds of thousands of tenants in a single cluster.

We would like to avoid any kind of authentication (authn) or authorization (authz) within Loki itself, so solutions which pass in a single or multiple tenant ID's via headers are desirable.

[midnightconman]

Is this still an issue?

I was hoping to use loki in multi-tenant mode internally in my company to limit the amount of logs by namespace in our more than 10 kubernetes clusters... not having a mechanisms to allow multiple tenants in the same request makes this impossible without poor interface requirements, like requiring queries include a tenant label.

Does nobody use multi-tenant mode at scale for loki yet?

Why can't we just have a separator on the header (like comma -- if supplied)? That would give us 1mb of tenants we could add per request with (I assume) minimal code changes, right?

[cf-sewe]

@lukipro Could you potentially share a few more information about the mentioned proxy in front of loki? Did you find some howto guide somewhere which describes how to configure such proxy? Or could you share an example config?

Basically I would like to adopt this great idea to

• configure Grafana to forward OIDC token (Azure AD) to Loki
• implement proxy in front of Loki which evaluates the token and depending on group membership adds the OrgIds which the user has access to.

The X-Scope-OrgID hopefully supports lists, or how could a user which is permitted to Tenant A and B perform a query for logs in both Loki Tenants?

[slim-bean]

This is still an issue/concern and we plan on solving this. I'm closing this issue for now as we are focusing on pruning our open issues list to be a small set of actively actively worked/discussed issues. I can assure you this problem will eventually be solved however, it's among the more common requests!

[sharathfeb12]

+1

[sklarsa]

Have there been any status updates on this feature and where it falls on the roadmap?

[TheMatrix97]

Hey! Any updates on this issue? It's kind of a blocking issue for us.
Thanks!