Fix logs cluster #1716
Merged
Fix logs cluster #1716
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
clayton-cornell
added
the
type/docs
| @@ -145,6 +145,20 @@ If {{< param "PRODUCT_NAME" >}} is _not_ running in clustered mode, then the blo | |||
| `loki.source.kubernetes` collects logs from every target it receives in its | |||
| arguments. | |||
|
|
|||
| Clustering only looks at the following labels for determining the shard key: | |||
There was a problem hiding this comment.
Suggested change
| Clustering only looks at the following labels for determining the shard key: | |
| Clustering looks at the following labels for determining the shard key: |
I think we can drop the only here.
ptodev
approved these changes
Oct 3, 2024
ptodev
pushed a commit
that referenced
this pull request
Oct 4, 2024
* fix clustering for logs * Simplify code. * Add changelog.
ptodev
pushed a commit
that referenced
this pull request
Oct 4, 2024
* fix clustering for logs * Simplify code. * Add changelog.
ptodev
added a commit
that referenced
this pull request
Oct 4, 2024
…SION file (#1828) * chore: bump yet-another-cloudwatch-exporter to v0.61.0 (#1690) * chore: bump yet-another-cloudwatch-exporter to v0.61.0 This bumps yet-another-cloudwatch-exporter to v0.61.0, which fixes [an issue with S3 metrics being reported as `NaN`](prometheus-community/yet-another-cloudwatch-exporter#728). Affected metrics have a value of `0` in the scraped prometheus metrics. With the update to v0.61.0, they are reported correctly. * fixup! chore: bump yet-another-cloudwatch-exporter to v0.61.0 JobLevelMetricFields has been removed with prometheus-community/yet-another-cloudwatch-exporter#1412 * fixup! chore: bump yet-another-cloudwatch-exporter to v0.61.0 * Update windows_exporter to v0.27.3 (#1785) * Update windows_exporter to v0.27.3 Signed-off-by: Jan-Otto Kröpke <mail@jkroepke.de> * Update windows_exporter to v0.27.3 Signed-off-by: Jan-Otto Kröpke <mail@jkroepke.de> --------- Signed-off-by: Jan-Otto Kröpke <mail@jkroepke.de> * build(deps): bump webpack from 5.82.1 to 5.94.0 in /internal/web/ui (#1594) Bumps [webpack](https://github.com/webpack/webpack) from 5.82.1 to 5.94.0. - [Release notes](https://github.com/webpack/webpack/releases) - [Commits](webpack/webpack@v5.82.1...v5.94.0) --- updated-dependencies: - dependency-name: webpack dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): bump rollup from 2.79.1 to 2.79.2 in /internal/web/ui (#1775) Bumps [rollup](https://github.com/rollup/rollup) from 2.79.1 to 2.79.2. - [Release notes](https://github.com/rollup/rollup/releases) - [Changelog](https://github.com/rollup/rollup/blob/master/CHANGELOG.md) - [Commits](rollup/rollup@v2.79.1...v2.79.2) --- updated-dependencies: - dependency-name: rollup dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * 1687 otelcol.exporter.awss3 config fixes (#1791) * Address loki.process config reloads (#1809) * Fix logs cluster (#1716) * fix clustering for logs * Simplify code. * Add changelog. * Reorder changelog * Update VERSION --------- Signed-off-by: Jan-Otto Kröpke <mail@jkroepke.de> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: morre <mmeyer@anaconda.com> Co-authored-by: Jan-Otto Kröpke <mail@jkroepke.de> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: cydergoth <cydergoth@gmail.com> Co-authored-by: Piotr <17101802+thampiotr@users.noreply.github.com> Co-authored-by: mattdurham <mattdurham@ppog.org>
pbailhache
pushed a commit
to pbailhache/alloy
that referenced
this pull request
Oct 17, 2024
* fix clustering for logs * Simplify code. * Add changelog.
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR Description
The
loki.source.kubernetescomponent uses the default clustering approach. Which adds things like ports, addresses and other things. When in reality we only care about 4 fields to uniquely identify a log location. This can cause duplication of log pulling. This limits the labels to the only ones that uniquely identify a log. This is copied by how the actual tailer deduplicates targets, unfortunately that comes after the cluster figures out how to allocate logs.Notes to the Reviewer
PR Checklist