feat(entitlement): add authorization feature

[sokoide]

What this PR does / why we need it:
It adds an authorization feature.

Loki now enabled native mTLS for both HTTP and gRPC with #4224 . However, the followings are still not supported.

1. There is no allow/deny per CNAME per action (read/write). All the valid client certs are accepted.
2. There is no per log line entitlement (allow/deny per logline for query and push)
3. Not only CNAMEs in client certs, but we sometimes want to put Loki behind a reverse proxy and use a specific header set by it (e.g. OIDC-USERNAME) as client user names

This feature enables all of the functions.

To make the authorization logic flexible, Loki talks to an external entitlement server (entserver-sample available) over gRPC. The entitlement server is supposed to run on the same host as Loki which means the gRPC call is made from Loki to localhost:21001 (configurable). Later versions may support mTLS between Loki and the entitlement server.

Which issue(s) this PR fixes:
Fixes #4224

Special notes for your reviewer:

Configuration:

If loki.yaml doesn't have authz_enabled, it works the same as before.

If it has auth_enabled: true (multi-tenancy), but authz_enabled: false, it works as the same as before which means multi-tenancy, but without authorization.

auth_enabled: true
authz_enabled: false

If it has auth_enabled: false (single-tenancy), but authz_enabled: true, it checks authorization for read (query* API) or write (push API) for fake organization with the other configs under entitlement tag.

auth_enabled: false
authz_enabled: true
entitlement:
  # entitlement server host:port
  grpc_server: "localhost:21001"
  # grpc_client_config not supported yet
  #grpc_client_config:
    #tls_enabled: true
    #tls_cert_path: /path/to/cert.pem
    #tls_key_path: /path/to/key.pem
    #tls_ca_path: /path/to/ca.pem

  # label name to check entitlement
  label_key: "job"
  # allow(true) or deny(false) access to the log if label_key doesn't exist in the log
  allow_access_if_label_key_doesnt_exist: false
  trusted_cnames:
    - host1.example.com
    - foo@EXAMPLE.COM
  # if the request is from trusted_cname and 'userid_header' is available, the value is used for clientUesrID
  # otherwise, cname is used for clientUserID
  # this is used when Loki is behind a reverse proxy
  userid_header: OIDC-USERNAME
  cache_retain_period: 1m # entitlement result cache TTL

If it has auth_enabled: true (multi-tenancy), but authz_enabled: true, authorization is checked for the org, user and action (read or write).

auth_enabled: true
authz_enabled: false
...

How to test the authorization feature:

1. Configure Loki with mTLS server and client certs (e.g. on loki1 host with CNAME=loki1)

auth_enabled: false
authz_enabled: true

entitlement:
  grpc_server: localhost:21001
  label_key: job
  trusted_cnames:
    - loki1
  cache_retain_period: 1m

server:
  ...
  http_tls_config:
    key_file: /path/to/key.pem
    cert_file: /path/to/cert.pem
    client_ca_file: /path/to/ca.pem
    client_auth_type: RequireAndVerifyClientCert
  grpc_tls_config:
    key_file: /path/to/key.pem
    cert_file: /path/to/cert.pem
    client_ca_file: /path/to/ca.pem
    client_auth_type: RequireAndVerifyClientCert

ingester_client:
  grpc_client_config:
    tls_enabled: true
    tls_key_path: /path/to/key.pem
    tls_cert_path: /path/to/cert.pem
    tls_ca_path: /path/to/ca.pem

2. Run entserer-sample on the same host
   ./entserver-sample -logLevel DEBUG -configFile /path/to/entserver-sample.yml # please configure the yaml

3. Make push or query requests
   Loki ingester (push) or querier (query) makes calls to the entserver and cached the result for cache_retain_period.
   The query detail is shown in stdout of entserver-sample with -logLevel DEBUG.
   The entserver-sample is a sample server for users to implement their entitlement server based on their requirements.
   entclient-sample makes a call to the entserver for testing purpose.

Checklist

• Documentation added
• Tests updated

[joankij]

+1

[sandy2008]

+1

[owen-d]

Hey @sokoide, thanks for the PR. While I can see this being helpful, I don't think we want to introduce any specific authorization into Loki itself. This allows us to keep the project simpler. I think it's more suitable to put an auth layer in front of Loki to enforce whichever policies make sense. This may not always be the case, but I don't feel comfortable adding this feature into the Loki source at this time. I'm especially sorry because of the effort you've put in here.

[sokoide]

Hello @owen-d, thank you very much for your review and explanation. I understood adding an authorization logic in Loki is not preferred.

We considered below to meet a security requirement (per logfile/label entitlement), but we couldn't overcome the following limitations and proposed the PR.
If you have a suggestion to solve it, we'd appreciate it!

1. Put an Nginx mTLS reverse proxy in front of Loki
   This can allow/deny per user (e.g. mTLS CNAME), however, it can't allow/deny per log file (per label) because labels are only visible inside Loki after a query. Labels are unknown when the reverse proxy processes the query request.

2. Have a tenant per entitlement group (multi-tenancy), set up an mTLS reverse proxy per tenant
   This works fine in a tenant, but there is no easy way to make a cross-tenant query. We'll need multiple queries by API then have to combine them. No good cross-tenant query with Grafana.

[owen-d]

Would building your own proxy component in front of Loki work? Doing so would allow you to apply any auth schemes you wish before sending to Loki or after performing a query to determine access.

[sokoide]

Hi @owen-d, I was only thinking about existing reverse proxies such as Nginx. Thank you very much for your advice. We'll consider moving the logic into a custom proxy.