Add cluster manager task throttling stats in nodes stats API #5790

dhwanilpatel · 2023-01-10T13:57:52Z

Signed-off-by: Dhwanil Patel dhwanip@amazon.com

Description

Add throttling stats in _nodes/stats API.

In Active master node's stats these stats will appear, for other nodes it will be 0. From active master node's stats we can get visibility on how many tasks are getting throttled.

Below is sample stats for two node cluster.

curl "localhost:9200/_nodes/stats/cluster_manager_throttling?pretty"
{
  "_nodes" : {
    "total" : 2,
    "successful" : 2,
    "failed" : 0
  },
  "cluster_name" : "opensearch",
  "nodes" : {
    "PZmL3O7dRBOg3d8NTRsSOQ" : {
      "timestamp" : 1673358595082,
      "name" : "a483e760c3eb.ant.amazon.com",
      "transport_address" : "127.0.0.1:9300",
      "host" : "127.0.0.1",
      "ip" : "127.0.0.1:9300",
      "roles" : [
        "cluster_manager",
        "data",
        "ingest",
        "remote_cluster_client"
      ],
      "attributes" : {
        "shard_indexing_pressure_enabled" : "true"
      },
      "cluster_manager_throttling" : {
        "cluster_manager_stats" : {
          "TotalThrottledTasks" : 18,
          "ThrottledTasksPerTaskType" : {
            "put-mapping" : 18
          }
        }
      }
    },
    "BPwWJ9vTSb2KJzPL9VZNcg" : {
      "timestamp" : 1673358595083,
      "name" : "a483e760c3eb.ant.amazon.com",
      "transport_address" : "127.0.0.1:9301",
      "host" : "127.0.0.1",
      "ip" : "127.0.0.1:9301",
      "roles" : [
        "cluster_manager",
        "data",
        "ingest",
        "remote_cluster_client"
      ],
      "attributes" : {
        "shard_indexing_pressure_enabled" : "true"
      },
      "cluster_manager_throttling" : {
        "cluster_manager_stats" : {
          "TotalThrottledTasks" : 0,
          "ThrottledTasksPerTaskType" : { }
        }
      }
    }
  }
}

Issues Resolved

#479

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Dhwanil Patel <dhwanip@amazon.com>

github-actions · 2023-01-10T14:03:33Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/9183/
CommitID: 9613607
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Signed-off-by: Dhwanil Patel <dhwanip@amazon.com>

github-actions · 2023-01-10T14:11:58Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/9184/
CommitID: de68b8a
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-01-10T14:20:02Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/9185/
CommitID: 195fbbf
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Signed-off-by: Dhwanil Patel <dhwanip@amazon.com>

server/src/main/java/org/opensearch/action/admin/cluster/node/stats/NodeStats.java

server/src/main/java/org/opensearch/cluster/service/ClusterManagerThrottlingStats.java

Bukhtawar · 2023-01-10T19:11:21Z

server/src/main/java/org/opensearch/cluster/service/ClusterManagerThrottlingStats.java

-    private Map<String, CounterMetric> throttledTasksCount = new ConcurrentHashMap<>();
+    private Map<String, CounterMetric> throttledTasksCount;
+
+    public ClusterManagerThrottlingStats() {


We might need additional metrics on throttle_time

Can you please elaborate more on this?
What we are expecting out of throttle_time metric?

time since we last throttled

Time since cluster manager performed has done throttling for any task? Or for different task type as well?

I think we can achieve it, as we can maintain time as well when we have performed last throttling on cluster manager node.

I am just wondering how this additional metric will help?

Signed-off-by: Dhwanil Patel <dhwanip@amazon.com>

github-actions · 2023-01-13T10:04:46Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/9373/
CommitID: 458ef18
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Signed-off-by: Dhwanil Patel <dhwanip@amazon.com>

github-actions · 2023-01-13T10:32:47Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/9374/
CommitID: f870b25
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Signed-off-by: Dhwanil Patel <dhwanip@amazon.com>

github-actions · 2023-01-13T11:11:52Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/9375/
CommitID: d80eca9
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

dhwanilpatel · 2023-01-13T11:35:22Z

Precommit is passing, gradle check is failing.

It is failing due to some connection issue with ubuntu. I will retry it later.

* What went wrong:
Execution failed for task ':test:fixtures:gcs-fixture:composeUp'.
> Exit-code 1 when calling /usr/bin/docker-compose, stdout: Step 1/11 : FROM ubuntu:18.04
  18.04: Pulling from library/ubuntu
  Digest: sha256:c1d0baf2425ecef88a2f0c3543ec43690dc16cc80d3c4e593bb95e4f45390e45
  Status: Downloaded newer image for ubuntu:18.04
   ---> e28a50f651f9
  Step 2/11 : RUN apt-get update -qqy
   ---> Running in 7c3adacdf556
  �[91mW: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/bionic/InRelease  Could not connect to archive.ubuntu.com:80 (185.125.190.36), connection timed out Could not connect to archive.ubuntu.com:80 (185.125.190.39), connection timed out
  W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/bionic-updates/InRelease  Unable to connect to archive.ubuntu.com:http:
  W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/bionic-backports/InRelease  Unable to connect to archive.ubuntu.com:http:
  W: Some index files failed to download. They have been ignored, or old ones used instead.

github-actions · 2023-01-13T17:37:49Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/9387/
CommitID: d80eca9
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Signed-off-by: Dhwanil Patel <dhwanip@amazon.com>

github-actions · 2023-01-16T11:06:26Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/9426/
CommitID: 39ed900
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-01-16T11:20:18Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/9427/
CommitID: 07c0d87
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Signed-off-by: Dhwanil Patel <dhwanip@amazon.com>

github-actions · 2023-01-16T11:59:46Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/9431/
CommitID: 34c6438
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Signed-off-by: Dhwanil Patel <dhwanip@amazon.com>

github-actions · 2023-01-16T15:07:06Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/9446/
CommitID: f8b0ce8
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

dhwanilpatel · 2023-01-16T15:48:26Z

NOTE on gradle check.

Backward compatible tests are failing. ./gradlew ':qa:mixed-cluster:v2.6.0#mixedClusterTest'.

Since we have added 2.6.0 version check for stats, it is expected to happen. It will fix once we merge backport 2.x PR. (#5871). Gradle check is passing in backport PR.

Similar behavior has been observed in other stats PR as well with version check. (#4932 (comment))

server/src/main/java/org/opensearch/action/admin/cluster/node/stats/NodeStats.java

opensearch-trigger-bot · 2023-01-19T21:34:43Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-5790-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 6da16e1e60760f630c8f7fe7ec1088feb13ff216
# Push it to GitHub
git push --set-upstream origin backport/backport-5790-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-5790-to-2.x.

Add throttling stats in nodes stats API

de68b8a

Signed-off-by: Dhwanil Patel <dhwanip@amazon.com>

dhwanilpatel force-pushed the throttling-stats branch from 9613607 to de68b8a Compare January 10, 2023 14:02

Merge remote-tracking branch 'upstream/main' into throttling-stats

195fbbf

Signed-off-by: Dhwanil Patel <dhwanip@amazon.com>

Fix test cases

a197db9

Signed-off-by: Dhwanil Patel <dhwanip@amazon.com>

dhwanilpatel marked this pull request as ready for review January 10, 2023 16:52

shwetathareja reviewed Jan 10, 2023

View reviewed changes

server/src/main/java/org/opensearch/action/admin/cluster/node/stats/NodeStats.java Show resolved Hide resolved

shwetathareja reviewed Jan 10, 2023

View reviewed changes

server/src/main/java/org/opensearch/cluster/service/ClusterManagerThrottlingStats.java Outdated Show resolved Hide resolved

Bukhtawar reviewed Jan 10, 2023

View reviewed changes

server/src/main/java/org/opensearch/cluster/service/ClusterManagerThrottlingStats.java Outdated Show resolved Hide resolved

Bukhtawar reviewed Jan 10, 2023

View reviewed changes

Added UT for stats and incorporated comments

458ef18

Signed-off-by: Dhwanil Patel <dhwanip@amazon.com>

Merge remote-tracking branch 'upstream/main' into throttling-stats

f870b25

Signed-off-by: Dhwanil Patel <dhwanip@amazon.com>

Renamed test case file

d80eca9

Signed-off-by: Dhwanil Patel <dhwanip@amazon.com>

Merge remote-tracking branch 'upstream/main' into throttling-stats

da4303b

Signed-off-by: Dhwanil Patel <dhwanip@amazon.com>

dhwanilpatel force-pushed the throttling-stats branch from 39ed900 to 07c0d87 Compare January 16, 2023 10:48

Incorporated comment and add equals method

34c6438

Signed-off-by: Dhwanil Patel <dhwanip@amazon.com>

dhwanilpatel force-pushed the throttling-stats branch from 07c0d87 to 34c6438 Compare January 16, 2023 11:30

Fix hashcode

f8b0ce8

Signed-off-by: Dhwanil Patel <dhwanip@amazon.com>

shwetathareja approved these changes Jan 17, 2023

View reviewed changes

Bukhtawar reviewed Jan 17, 2023

View reviewed changes

server/src/main/java/org/opensearch/action/admin/cluster/node/stats/NodeStats.java Show resolved Hide resolved

Bukhtawar approved these changes Jan 17, 2023

View reviewed changes

Bukhtawar changed the title ~~Add throttling stats in nodes stats API~~ Add cluster manager task throttling stats in nodes stats API Jan 17, 2023

Bukhtawar merged commit 6da16e1 into opensearch-project:main Jan 17, 2023

andrross added the backport 2.x Backport to 2.x branch label Jan 19, 2023

lukas-vlcek mentioned this pull request May 23, 2024

[DOC] Add missing cluster_manager_throttling nodes stats opensearch-project/documentation-website#7240

Closed

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add cluster manager task throttling stats in nodes stats API #5790

Add cluster manager task throttling stats in nodes stats API #5790

dhwanilpatel commented Jan 10, 2023

github-actions bot commented Jan 10, 2023

github-actions bot commented Jan 10, 2023

github-actions bot commented Jan 10, 2023

Bukhtawar Jan 10, 2023

dhwanilpatel Jan 13, 2023

Bukhtawar Jan 16, 2023

dhwanilpatel Jan 16, 2023

github-actions bot commented Jan 13, 2023

github-actions bot commented Jan 13, 2023

github-actions bot commented Jan 13, 2023

dhwanilpatel commented Jan 13, 2023

github-actions bot commented Jan 13, 2023

github-actions bot commented Jan 16, 2023

github-actions bot commented Jan 16, 2023

github-actions bot commented Jan 16, 2023

github-actions bot commented Jan 16, 2023

dhwanilpatel commented Jan 16, 2023

opensearch-trigger-bot bot commented Jan 19, 2023

Add cluster manager task throttling stats in nodes stats API #5790

Add cluster manager task throttling stats in nodes stats API #5790

Conversation

dhwanilpatel commented Jan 10, 2023

Description

Issues Resolved

Check List

github-actions bot commented Jan 10, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 10, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 10, 2023

Gradle Check (Jenkins) Run Completed with:

Bukhtawar Jan 10, 2023

Choose a reason for hiding this comment

dhwanilpatel Jan 13, 2023

Choose a reason for hiding this comment

Bukhtawar Jan 16, 2023

Choose a reason for hiding this comment

dhwanilpatel Jan 16, 2023

Choose a reason for hiding this comment

github-actions bot commented Jan 13, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 13, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 13, 2023

Gradle Check (Jenkins) Run Completed with:

dhwanilpatel commented Jan 13, 2023

github-actions bot commented Jan 13, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 16, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 16, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 16, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 16, 2023

Gradle Check (Jenkins) Run Completed with:

dhwanilpatel commented Jan 16, 2023

opensearch-trigger-bot bot commented Jan 19, 2023