Update kafka exporter dependency #6778
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ptodev
reviewed
Apr 3, 2024
There was a problem hiding this comment.
Thank you for working on this! I had a quick look and noted a few things in comments.
Btw, it would have been nice if we could document the metrics which the exporter produces. However, we don't seem to do this for any prometheus.exporter component, so it's ok not to do it in this PR.
docs/sources/flow/reference/components/prometheus.exporter.kafka.md
Outdated
Show resolved
Hide resolved
rfratto
added
variant/static
docs/sources/flow/reference/components/prometheus.exporter.kafka.md
Outdated
Show resolved
Hide resolved
docs/sources/flow/reference/components/prometheus.exporter.kafka.md
Outdated
Show resolved
Hide resolved
…ka.md Co-authored-by: Clayton Cornell <131809008+clayton-cornell@users.noreply.github.com>
…/agent into update-kafka-exporter-dependency
ptodev
approved these changes
Apr 11, 2024
| @@ -147,14 +188,21 @@ func New(logger log.Logger, c *Config) (integrations.Integration, error) { | |||
| return nil, fmt.Errorf("zookeeper lag is enabled but no zookeeper uri was provided") | |||
| } | |||
|
|
|||
| // 30 is the default value | |||
| if c.PruneIntervalSeconds != 30 { | |||
There was a problem hiding this comment.
Do we need to print the warning if PruneIntervalSeconds is set to 0?
There was a problem hiding this comment.
if PruneIntervalSeconds is different from 30 then it was explicitly set in the config by the user and that should trigger a warning (even if it is set to 0)
The exporter was reviewed by @gaantunes and has been merged to the official grafana fork already. If you have time you can also have a look at it if you want |
Co-authored-by: Paulin Todev <paulin.todev@gmail.com>
clayton-cornell
approved these changes
Apr 11, 2024
4 tasks
ptodev
added a commit
that referenced
this pull request
Apr 12, 2024
* update agent to local exporter * remove prune interval argument * add kafka exporter integration test * replace davidmparrott kafka exporter by wildum kafka exporter * update doc * update unit test * update converter * fix integration test * integration ci test * integration ci test * integration ci test * add longer sleep * add notes about metrics rename * fix copy paste mistake with metric name * avoid breaking changes by re-introducing pruneIntervalSeconds as a no-op * change kafka_exporter ref from wildum to grafana * Update docs/sources/flow/reference/components/prometheus.exporter.kafka.md Co-authored-by: Clayton Cornell <131809008+clayton-cornell@users.noreply.github.com> * fix unit tests * put the pruneIntervalSeconds arg in converter * Update CHANGELOG.md Co-authored-by: Paulin Todev <paulin.todev@gmail.com> * fix doc --------- Co-authored-by: Clayton Cornell <131809008+clayton-cornell@users.noreply.github.com> Co-authored-by: Paulin Todev <paulin.todev@gmail.com>
ptodev
added a commit
that referenced
this pull request
Apr 12, 2024
* Updating dependencies to fix CVEs: * CVE-2024-27304 * CVE-2024-27289 * CVE-2024-28180 * CVE-2024-24786 * Use stackdriver exporter fork with a fix for histogram sum + count (#6720) * Update kafka exporter dependency (#6778) Co-authored-by: kgeckhart <kgeckhart@users.noreply.github.com> Co-authored-by: William Dumont <william.dumont@grafana.com> Co-authored-by: Clayton Cornell <131809008+clayton-cornell@users.noreply.github.com>
github-actions
bot
added
the
frozen-due-to-age
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR Description
The agent is currently using this outdated fork of a popular kafka_exporter.
The fork that we use has not been updated with upstream for years and has a bug with the added lag metrics not being updated on metadata changes (this was reported via a customer escalation). These lag metrics are not in the upstream repo yet (see danielqsj/kafka_exporter#195 (comment)).
I forked the upstream repo, added the lag metrics, fixed them to be updated on metadata changes and merged my fork to a "new" grafana repo that the agent uses with this PR.
I added an integration test that spawns a kafka cluster with a producer and a consumer. The test runs an agent with the prometheus exporter and verifies that the expected metrics are collected correctly.
Fixes grafana/alloy#464
PR Checklist