-
Notifications
You must be signed in to change notification settings - Fork 2.3k
Add Metrics For The Merged Segment Warmer Flow #18929
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add Metrics For The Merged Segment Warmer Flow #18929
Conversation
|
❌ Gradle check result for 3dfd64f: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
3dfd64f to
166ec29
Compare
|
❌ Gradle check result for 166ec29: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
|
❌ Gradle check result for 12721aa: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
|
❌ Gradle check result for b5bc126: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
1c95c87 to
66fa8f1
Compare
|
❌ Gradle check result for 66fa8f1: null Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
|
❌ Gradle check result for 2a071c8: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
Signed-off-by: Aditya Khera <kheraadi@amazon.com>
Signed-off-by: Aditya Khera <kheraadi@amazon.com>
Signed-off-by: Aditya Khera <kheraadi@amazon.com>
Signed-off-by: Aditya Khera <kheraadi@amazon.com>
Signed-off-by: Aditya Khera <kheraadi@amazon.com>
Signed-off-by: Aditya Khera <kheraadi@amazon.com>
Signed-off-by: Aditya Khera <kheraadi@amazon.com>
Signed-off-by: Aditya Khera <kheraadi@amazon.com>
Signed-off-by: Aditya Khera <kheraadi@amazon.com>
Signed-off-by: Aditya Khera <kheraadi@amazon.com>
44437d4 to
a142adb
Compare
|
❌ Gradle check result for a142adb: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
Signed-off-by: Aditya Khera <kheraadi@amazon.com>
|
❌ Gradle check result for a68d57b: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
Signed-off-by: Aditya Khera <kheraadi@amazon.com>
|
❌ Gradle check result for d6be469: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
Signed-off-by: Aditya Khera <kheraadi@amazon.com>
|
❌ Gradle check result for a6c9a97: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
Signed-off-by: Aditya Khera <kheraadi@amazon.com>
|
❌ Gradle check result for ba2cec5: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
Signed-off-by: Aditya Khera <kheraadi@amazon.com>
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #18929 +/- ##
============================================
+ Coverage 73.11% 73.12% +0.01%
- Complexity 70661 70764 +103
============================================
Files 5724 5726 +2
Lines 323498 323750 +252
Branches 46852 46882 +30
============================================
+ Hits 236518 236748 +230
- Misses 67846 67941 +95
+ Partials 19134 19061 -73 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
) * Adding stats for merged segment warmer Signed-off-by: kh3ra <adityakh3ra@gmail.com> Signed-off-by: Aditya Khera <kheraadi@amazon.com> * Adding unit tests Signed-off-by: kh3ra <adityakh3ra@gmail.com> Signed-off-by: Aditya Khera <kheraadi@amazon.com> * SpotlessApply Signed-off-by: kh3ra <adityakh3ra@gmail.com> Signed-off-by: Aditya Khera <kheraadi@amazon.com> * Minor fix Signed-off-by: kh3ra <adityakh3ra@gmail.com> Signed-off-by: Aditya Khera <kheraadi@amazon.com> * minor fix Signed-off-by: kh3ra <adityakh3ra@gmail.com> Signed-off-by: Aditya Khera <kheraadi@amazon.com> * addressing review comments - tests to follow Signed-off-by: Aditya Khera <kheraadi@amazon.com> * Addressing review comments + added ITs Signed-off-by: Aditya Khera <kheraadi@amazon.com> * addressing review comments - tests to follow Signed-off-by: Aditya Khera <kheraadi@amazon.com> * Changelog + minor bug fix Signed-off-by: Aditya Khera <kheraadi@amazon.com> * Addressing review comments + added UTs Signed-off-by: Aditya Khera <kheraadi@amazon.com> * Fixing tests Signed-off-by: Aditya Khera <kheraadi@amazon.com> * Restored breaking publicAPIs Signed-off-by: Aditya Khera <kheraadi@amazon.com> * spotlessApply Signed-off-by: Aditya Khera <kheraadi@amazon.com> * Empty commit Signed-off-by: Aditya Khera <kheraadi@amazon.com> * Empty commit to trigger build Signed-off-by: Aditya Khera <kheraadi@amazon.com> * Test changes Signed-off-by: Aditya Khera <kheraadi@amazon.com> * Empty commit Signed-off-by: Aditya Khera <kheraadi@amazon.com> * Empty commit to trigger build Signed-off-by: Aditya Khera <kheraadi@amazon.com> * Fixing tests Signed-off-by: Aditya Khera <kheraadi@amazon.com> * fixes to merged segment warmer + tests Signed-off-by: Aditya Khera <kheraadi@amazon.com> * spotlessApply Signed-off-by: Aditya Khera <kheraadi@amazon.com> * Fixing ITs after rebase Signed-off-by: Aditya Khera <kheraadi@amazon.com> * Empty commit Signed-off-by: Aditya Khera <kheraadi@amazon.com> * Addressing review comments Signed-off-by: Aditya Khera <kheraadi@amazon.com> * rebased with main Signed-off-by: Aditya Khera <kheraadi@amazon.com> * Fixing tests Signed-off-by: Aditya Khera <kheraadi@amazon.com> * spotlessApply Signed-off-by: Aditya Khera <kheraadi@amazon.com> * YAML test fix Signed-off-by: Aditya Khera <kheraadi@amazon.com> * yaml test fix Signed-off-by: Aditya Khera <kheraadi@amazon.com> * test fixes Signed-off-by: Aditya Khera <kheraadi@amazon.com> --------- Signed-off-by: kh3ra <adityakh3ra@gmail.com> Signed-off-by: Aditya Khera <kheraadi@amazon.com> Co-authored-by: Aditya Khera <kheraadi@amazon.com>
) * Adding stats for merged segment warmer Signed-off-by: kh3ra <adityakh3ra@gmail.com> Signed-off-by: Aditya Khera <kheraadi@amazon.com> * Adding unit tests Signed-off-by: kh3ra <adityakh3ra@gmail.com> Signed-off-by: Aditya Khera <kheraadi@amazon.com> * SpotlessApply Signed-off-by: kh3ra <adityakh3ra@gmail.com> Signed-off-by: Aditya Khera <kheraadi@amazon.com> * Minor fix Signed-off-by: kh3ra <adityakh3ra@gmail.com> Signed-off-by: Aditya Khera <kheraadi@amazon.com> * minor fix Signed-off-by: kh3ra <adityakh3ra@gmail.com> Signed-off-by: Aditya Khera <kheraadi@amazon.com> * addressing review comments - tests to follow Signed-off-by: Aditya Khera <kheraadi@amazon.com> * Addressing review comments + added ITs Signed-off-by: Aditya Khera <kheraadi@amazon.com> * addressing review comments - tests to follow Signed-off-by: Aditya Khera <kheraadi@amazon.com> * Changelog + minor bug fix Signed-off-by: Aditya Khera <kheraadi@amazon.com> * Addressing review comments + added UTs Signed-off-by: Aditya Khera <kheraadi@amazon.com> * Fixing tests Signed-off-by: Aditya Khera <kheraadi@amazon.com> * Restored breaking publicAPIs Signed-off-by: Aditya Khera <kheraadi@amazon.com> * spotlessApply Signed-off-by: Aditya Khera <kheraadi@amazon.com> * Empty commit Signed-off-by: Aditya Khera <kheraadi@amazon.com> * Empty commit to trigger build Signed-off-by: Aditya Khera <kheraadi@amazon.com> * Test changes Signed-off-by: Aditya Khera <kheraadi@amazon.com> * Empty commit Signed-off-by: Aditya Khera <kheraadi@amazon.com> * Empty commit to trigger build Signed-off-by: Aditya Khera <kheraadi@amazon.com> * Fixing tests Signed-off-by: Aditya Khera <kheraadi@amazon.com> * fixes to merged segment warmer + tests Signed-off-by: Aditya Khera <kheraadi@amazon.com> * spotlessApply Signed-off-by: Aditya Khera <kheraadi@amazon.com> * Fixing ITs after rebase Signed-off-by: Aditya Khera <kheraadi@amazon.com> * Empty commit Signed-off-by: Aditya Khera <kheraadi@amazon.com> * Addressing review comments Signed-off-by: Aditya Khera <kheraadi@amazon.com> * rebased with main Signed-off-by: Aditya Khera <kheraadi@amazon.com> * Fixing tests Signed-off-by: Aditya Khera <kheraadi@amazon.com> * spotlessApply Signed-off-by: Aditya Khera <kheraadi@amazon.com> * YAML test fix Signed-off-by: Aditya Khera <kheraadi@amazon.com> * yaml test fix Signed-off-by: Aditya Khera <kheraadi@amazon.com> * test fixes Signed-off-by: Aditya Khera <kheraadi@amazon.com> --------- Signed-off-by: kh3ra <adityakh3ra@gmail.com> Signed-off-by: Aditya Khera <kheraadi@amazon.com> Co-authored-by: Aditya Khera <kheraadi@amazon.com>
Description
This PR introduces comprehensive metrics for monitoring merged segment warming operations, building upon the work from PR#18683, Issue#17528. These metrics provide insights into warming operations, data transfer, and performance characteristics.
Metrics Added
Cumulative Metrics
Added the following metrics under
merges.warmer:total_invocations_count: Tracks total number of warm operationstotal_time_millis: Total time spent in warming operationstotal_failure_count: Number of failed warming attemptstotal_bytes_sent: Total data volume uploadedtotal_bytes_received: Total data volume downloadedtotal_send_time: Cumulative time spent in uploadstotal_receive_time: Cumulative time spent in downloadsPoint-in-Time Metrics
ongoing_count: Current number of active warming operationsAPI Changes
/_nodes/stats/indices/merge/_stats/merge/_stats/merge?pretty&level=shards/_cat/shards?v&h=index,shard,prirep,merges.*&format=json/_cat/nodes?v&h=id,merges.*&format=json/_cat/indices?v&h=id,merges.warmer.*&format=jsonRelated Issues
Resolves #[Issue number to be closed when this PR is merged]
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.