-
Notifications
You must be signed in to change notification settings - Fork 3.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Helm] Self-Monitoring fails when loki.auth_enabled is true and/or gateway.basicAuth.enabled is true #10873
Comments
Note: The current workaround I use to enable multitenancy without basic authentication on the gateway is to conditionally set the orgID header on nginx: gateway:
nginxConfig:
httpSnippet: |
{{ if .Values.loki.tenants }}
map $http_x_scope_orgid $new_x_scope_orgid {
default $http_x_scope_orgid; # use the existing header value by default
"" $remote_user; # if not set, use the $remote_user variable
}
proxy_set_header X-Scope-OrgID $new_x_scope_orgid;
{{ end }} |
Setting Chart already seems to know how to pass tenant and password to the chart. Implementation seems to be broken and not exposed via values file.
I manually patched
Chart already configures loki/production/helm/loki/templates/loki-canary/daemonset.yaml Lines 51 to 53 in 6b46e37
|
I have the same problem. |
Yeah same! Loki can't monitor itself with the default config. That gives me pause regarding the production-readiness of this chart, especially given how long this has been an issue. |
Describe the bug
Logs of the self monitoring deployment cannot be shipped to loki tenancy is activated (default) and/or if the gateway basic authentication is enabled.
Issue:
tenant_id
field which sets theX-Scope-OrgID
header source-code, promtail-doc.X-Scope-OrgID
header with the basic auth username which is resulting in an empty header value source-codeRelevant agent log:
Gateway logs with the default configuration:
Gateway logs when the
proxy_set_header
directive in the gateways nginx.conf is commented:To Reproduce
Steps to reproduce the behavior:
kubectl create ns loki && helm install --values loki.yml -n loki loki grafana/loki
Expected behavior
The default configuration works to send logs with the agent to loki (
loki.auth_enabled: true
). I.e. the agent configuration contains at leastbasic_auth.username
so that nginx set's the correct header. And if basic auth on the gateway is enabled also the password.Inspect the config:
Generated configuration with the default loki multi tenancy configuration (above):
Expected when gateway basic auth is not enabled:
(This could also be fixed by adjusting the default nginx config mentioned in the comment below)
Expected when gateway basic auth is enabled:
The password could either be set on the selfMonitoring tenant or in the loki tenants, but then it should be documented that the tenant the self monitoring uses must be configured with a password in the loki tenants (relevant source-code.
Environment:
Screenshots, Promtail config, or terminal output
See above
The text was updated successfully, but these errors were encountered: