Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Worker level task metrics #12446

Merged
merged 21 commits into from
Apr 26, 2022
Merged

Worker level task metrics #12446

merged 21 commits into from
Apr 26, 2022

Conversation

zachjsh
Copy link
Contributor

@zachjsh zachjsh commented Apr 18, 2022

Added a new monitor, WorkerTaskCountStatsMonitor, that allows each middle manage worker to report metrics for successful / failed tasks, and task slot usage. There is an exsiting TaskCountStatsMonitor, which allows for reporting of task metrics but is done so by the overlord and so task metric data for individual workers is lost, hence this monitor. This monitor is only supported on MiddleManager type NodeRole.

Also fixes an inconsistency in the name of the existing task metric for tracking taskslot usage.

This PR has:

  • been self-reviewed.
  • added documentation for new or modified features or behaviors.
  • added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
  • added or updated version, license, or notice information in licenses.yaml
  • added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
  • added integration tests.
  • been tested in a test Druid cluster.

@zachjsh zachjsh requested a review from jihoonson April 18, 2022 16:19
@@ -63,9 +63,15 @@
"task/pending/count" : { "dimensions" : ["dataSource"], "type" : "gauge" },
"task/waiting/count" : { "dimensions" : ["dataSource"], "type" : "gauge" },

"worker/task/failed/count" : { "dimensions" : ["category", "vesion"], "type" : "count" },
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

vesion -> version

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ahhh dang lol. Good catch

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed

@jon-wei jon-wei merged commit 564d6de into apache:master Apr 26, 2022
@zachjsh zachjsh deleted the worker-task-metrics branch April 26, 2022 18:02
@abhishekagarwal87 abhishekagarwal87 added this to the 24.0.0 milestone Aug 26, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants