Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

False warn for "per-tenant timeout not configured" #9801

Closed
BaskiGIT opened this issue Jun 26, 2023 · 1 comment · Fixed by #10083
Closed

False warn for "per-tenant timeout not configured" #9801

BaskiGIT opened this issue Jun 26, 2023 · 1 comment · Fixed by #10083
Labels
type/bug Somehing is not working as expected

Comments

@BaskiGIT
Copy link

BaskiGIT commented Jun 26, 2023

Describe the bug
With Loki 2.8.2 simple scalable mode ( read and write targets ) per-tenant timeout is configured using runtime configuration. But Loki shows a warn per-tenant timeout not configured, using default engine timeout (\"5m0s\"). This behavior will change in the next major to always use the default per-tenant timeout (\"5m\").

To Reproduce
Steps to reproduce the behavior:

  1. Create runtime configuration in runtime.yaml
overrides:
      fake:
        query_timeout: 10m
  1. Set runtime_config in loki config
runtime_config:
      file: /etc/loki_runtime/runtime.yaml
  1. Start Loki of version 2.8.2
  2. See the logs of read or write pods
  3. The first line of log reports a warn like below

level=warn ts=2023-06-26T14:24:12.394946274Z caller=loki.go:286 msg="per-tenant timeout not configured, using default engine timeout ("5m0s"). This behavior will change in the next major to always use the default per-tenant timeout ("5m")."

Expected behavior
Since query_timeout is already defined in runtime configuration this line of log is not expected to appear. Either Loki logging system is false or runtime configuration is not respected.

Environment:

  • gke
  • kustomize

Screenshots, Promtail config, or terminal output
image

@JStickler JStickler added the type/bug Somehing is not working as expected label Jun 28, 2023
@aminesnow
Copy link
Contributor

I've investigated this bug for the last couple of days. I can confirm that this is just an issue with the log wording.
The per-tenant Overrides are applied as expected, and the logic behind the warning only pertains to the global Limits Config and not the per-tenant one.
I'll open a PR soon to suggest a better wording for that warning message.

DylanGuedes pushed a commit that referenced this issue Aug 3, 2023
**What this PR does / why we need it**:
This PR fixes the warning messages that are logged in relation to the
timeouts migration.

The messages reference the per-tenant config, but the config in question
in the `AdjustForTimeoutsMigration` function is the global
`limits_config`. This is a source of confusion for users.


**Which issue(s) this PR fixes**:
Fixes #9801
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type/bug Somehing is not working as expected
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants