Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: guard against failed ingester requests for detected fields #14210

Closed
wants to merge 3 commits into from

Conversation

trevorwhitney
Copy link
Collaborator

@trevorwhitney trevorwhitney commented Sep 20, 2024

What this PR does / why we need it:

We have seen an increase in 500s from ingester for various queries. These 500s are causing entire detected field requests to fail. Since detected fields is just an approximation/sample anyway, this PR changes that behavior to guard against that in a few ways

  1. split requests in the querier into separate ingester and store requests, and process iterators from either if any are returned
  2. return an empty response rather than an error when no iterators are returned (we still log the error as a warn). this will give other splits in the QF a chance of returning data even if not all subqueries are successful.
  3. Remove the retry middleware for /detected_fields as it's just an approximation anyway, it doesn't need every subquery to return
  4. Add a fall through middleware so that failed request return an empty result rather than an error.

Checklist

  • Reviewed the CONTRIBUTING.md guide (required)
  • Documentation added
  • Tests updated
  • Title matches the required conventional commits format, see here
    • Note that Promtail is considered to be feature complete, and future development for logs collection will be in Grafana Alloy. As such, feat PRs are unlikely to be accepted unless a case can be made for the feature actually being a bug fix to existing behavior.
  • Changes that require user attention or interaction to upgrade are documented in docs/sources/setup/upgrade/_index.md
  • For Helm chart changes bump the Helm chart version in production/helm/loki/Chart.yaml and update production/helm/loki/CHANGELOG.md and production/helm/loki/README.md. Example PR
  • If the change is deprecating or removing a configuration option, update the deprecated-config.yaml and deleted-config.yaml files respectively in the tools/deprecated-config-checker directory. Example PR

Squashed commit of the following:

commit 2ef80ec
Author: Trevor Whitney <trevorjwhitney@gmail.com>
Date:   Fri Sep 20 13:24:35 2024 -0600

    chore: no retries for detected_fields

commit c7a471d
Author: Trevor Whitney <trevorjwhitney@gmail.com>
Date:   Fri Sep 20 13:05:33 2024 -0600

    feat: return empty result instead of error for failed detected fields queries

commit 7b3529a
Author: Trevor Whitney <trevorjwhitney@gmail.com>
Date:   Fri Sep 20 09:39:33 2024 -0600

    fix: guard against failed ingester requests for detected fields

    (cherry picked from commit fbf9018)
@trevorwhitney trevorwhitney marked this pull request as ready for review September 20, 2024 20:04
@trevorwhitney trevorwhitney requested a review from a team as a code owner September 20, 2024 20:04
@trevorwhitney
Copy link
Collaborator Author

closing in favor of #14212

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant