Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Document multi-tenant queries. #5994

Merged
merged 7 commits into from
Jun 2, 2022
Merged

Document multi-tenant queries. #5994

Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
8cb3256
Document multi-tenant queries.
jeschkies Apr 22, 2022
0788599
Update docs/sources/operations/multi-tenancy.md
jeschkies May 2, 2022
ad71ec1
Merge remote-tracking branch 'grafana/main' into karsten/document-mul…
jeschkies May 3, 2022
66ef008
Merge remote-tracking branch 'refs/remotes/origin/karsten/document-mu…
jeschkies May 3, 2022
38d077e
Update docs/sources/operations/multi-tenancy.md
jeschkies May 3, 2022
46b4107
Update docs/sources/operations/multi-tenancy.md
jeschkies May 6, 2022
c19a8d2
Update docs/sources/operations/multi-tenancy.md
jeschkies May 6, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions docs/sources/operations/multi-tenancy.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,3 +17,19 @@ To run in multi-tenant mode, Loki should be started with `auth_enabled: true`.
Loki can be run in "single-tenant" mode where the `X-Scope-OrgID` header is not
required. In single-tenant mode, the tenant ID defaults to `fake`.

## Multi-tenant Queries
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder whether we should document how to setup a datasource in Grafana for multi-tenant queries. It is not trivial IMHO.


If run in multi-tenant mode, queries that gather results from multiple tenants can be enabled with
the `multi_tenant_queries_enabled: true` configuration option in the querier. Once enabled, multiple
tenant IDs can be defined in the HTTP header `X-Scope-OrgID` by concatenating them
with `|`. For instance a query for tenant A and B can set `X-Scope-OrgID: A|B`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That didnt work.
When use X-Scope-OrgID=onetenant|twotenant loki response 200 but empty data.

You must use X-Scope-OrgID x2 like that:
image

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Inserting. The piping worked for me. Let me investigate. Thanks for trying it out.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, I've tried it with the Docker compose setup in tools/dev/loki-boltdb-storage-s3. How did you try it out?

Copy link
Contributor

@xom4ek xom4ek May 4, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

analytics:
  reporting_enabled: false
auth_enabled: true
common:
  path_prefix: /var/loki
  replication_factor: 1
  storage:
    s3:
      access_key_id: access
      bucketnames: bucketname
      endpoint: endpoint
      http_config:
        idle_conn_timeout: 90s
        insecure_skip_verify: false
        response_header_timeout: 0s
      region: region
      secret_access_key: secret
compactor:
  shared_store: aws
  working_directory: /var/loki/compactor
frontend:
  log_queries_longer_than: 20s
memberlist:
  join_members:
  - loki-memberlist
querier:
  multi_tenant_queries_enabled: true
schema_config:
  configs:
  - from: "2021-09-07"
    index:
      period: 24h
      prefix: loki_index_
    object_store: aws
    schema: v11
    store: boltdb-shipper
storage_config:
  aws:
    access_key_id: access
    bucketnames: bucketname
    endpoint: endpoint
    http_config:
      idle_conn_timeout: 90s
      insecure_skip_verify: false
      response_header_timeout: 0s
    region: region
    secret_access_key: secret
  boltdb_shipper:
    active_index_directory: /var/loki/index
    cache_location: /var/loki/index_cache
    shared_store: s3
tracing:
  enabled: true

So i found that when use simple-scale helm chart.

If use X-Scope-OrgID='default|system'
image

If use X-Scope-OrgID='default
image

Copy link
Contributor

@xom4ek xom4ek May 4, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like on my instance multi-tenant queries didnt work.
I need send __tenant_id__ from log collector? Or loki create that labels selves?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What are the screenshots showing? What are tags here? X-Scope-OrgID='default|system' image should be correct on the datasource. However, that also means Promtail is sending logs either as default or system tenant.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let me explain.

When i use Loki datasource and set X-Scope-OrgID=default or X-Scope-OrgID=system - we get that graph:
image

But when i set X-Scope-OrgID=default|system i get "no data".

So my first screenshots just trace from loki where read service get 0 Series when tenants_ids=[default|system] and 3k when tenants_ids=[default]

About promtail - already have tenants in loki and X-Scope-OrgID without multitenant_query work fine - just in code and your example i see __tenant_id label and didnt see that in my loki installation

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That didnt work. When use X-Scope-OrgID=onetenant|twotenant loki response 200 but empty data.

You must use X-Scope-OrgID x2 like that: image

That didnt work correct - loki just take first of that Headers

Copy link
Contributor Author

@jeschkies jeschkies May 5, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your patience. I'm not really sure where you see tenants_ids=[default|system].

Anyhow, which version of Loki are you using? It must be a recent main build after at least commit c7cbeab.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes - i found problem.

We didnt update master branch after release.
So right now everything fine.
Thanks for help!


Only query endpoints support multi-tenant calls. Calls to `GET /loki/api/v1/tail`
and `POST /loki/api/v1/push` will return an HTTP 400 error if more than one tenant
is defined in the HTTP header.

Instant and range queries support label filtering on the tenant IDs. For example
`{app="foo", __tenant_id__=~"a.+"} | logfmt` will return results for all tenants
whose ID start with `a`. Tenant ID filtering in stages is not supported; `{app="foo"} | __tenant_id__="1" | logfmt` will not work.

In case the label `__tenant_id__` is already present in a log stream it is prepended with `original_`.