Compute Segment Replication stats for SR backpressure. #6520

mch2 · 2023-03-01T05:49:59Z

Description

This PR introduces new mechanisms to keep track of the current replicas within a replication group. It is the first step in changes to add backpressure for lagging replicas when SR is enabled.

Adds new fields to CheckpointState inside a primary's ReplicationTracker to keep track of the following new metrics:
- checkpointTimers - a map of ReplicationCheckpoint to a timer. When the replica is caught up to at least that checkpoint, timers are stopped and removed. Timers are added when the primary publishes a new checkpoint. This allows us to compute the bytes behind and ongoing replication lag when requested.
- visibleReplicationCheckpoint - the currently searchable checkpoint on the replica.
- lastCompletedReplicationLag - lag for the most recently completed replication event.
ReplicationTracker also has a new method getSegmentReplicationStats to fetch these stats per in-sync replica in addition to the amount of bytes the replica is behind. In addition it returns the average replication lag for the whole group.
The new stats are also added to NodeStats API.

Example Response:

    "oltX47ZFRfSMY9k5cef3eA" : {
      "timestamp" : 1677792047146,
      "name" : "node-2",
      "roles" : [
        "cluster_manager",
        "data",
        "ingest",
        "remote_cluster_client"
      ],
      "attributes" : {
        "shard_indexing_pressure_enabled" : "true"
      },
      "segment_replication" : {
        "[so][5]" : {
          "rejected_requests" : 0,
          "replicas" : {
            "syu-_SmoRmK1HrzuaoIBHQ" : {
              "checkpoints_behind" : 0,
              "bytes_behind" : "0b",
              "current_replication_lag" : "0s",
              "last_completed_replication_lag" : "273ms"
            }
          }
        },
        "[so][2]" : {
          "rejected_requests" : 0,
          "replicas" : {
            "x3BEIA9wQKGDfht4vXHUMA" : {
              "checkpoints_behind" : 0,
              "bytes_behind" : "0b",
              "current_replication_lag" : "0s",
              "last_completed_replication_lag" : "146ms"
            }
          }
        },
        "[so][8]" : {
          "rejected_requests" : 0,
          "replicas" : {
            "pkb-3tEOTJiRZQjlTV3VuQ" : {
              "checkpoints_behind" : 1,
              "bytes_behind" : "14.3mb",
              "current_replication_lag" : "385ms",
              "last_completed_replication_lag" : "300ms"
            }
          }
        }
      }

Issues Resolved

closes #6388
related #4478

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

This PR introduces new mechanisms to keep track of the current replicas within a replication group intended to be used to apply backpressure. The new stats are also added to NodeStats. Signed-off-by: Marc Handalian <handalm@amazon.com>

Signed-off-by: Marc Handalian <handalm@amazon.com>

github-actions · 2023-03-02T20:10:27Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/11921/
CommitID: 51e9fb2
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

dreamer-89

@mch2 Posting comments from older review (PR got updated in between).

...er/src/internalClusterTest/java/org/opensearch/indices/replication/SegmentReplicationIT.java

server/src/main/java/org/opensearch/action/admin/cluster/node/stats/NodeStats.java

server/src/main/java/org/opensearch/action/admin/cluster/node/stats/NodesStatsRequest.java

server/src/main/java/org/opensearch/index/SegmentReplicationShardStats.java

server/src/main/java/org/opensearch/index/SegmentReplicationStats.java

server/src/main/java/org/opensearch/index/SegmentReplicationShardStats.java

github-actions · 2023-03-02T20:40:04Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/11922/
CommitID: 348b34d

codecov-commenter · 2023-03-02T20:42:39Z

Codecov Report

Merging #6520 (2c69235) into main (ee305d0) will decrease coverage by 0.02%.
The diff coverage is 43.78%.

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

@@             Coverage Diff              @@
##               main    #6520      +/-   ##
============================================
- Coverage     70.80%   70.78%   -0.02%     
- Complexity    59143    59157      +14     
============================================
  Files          4802     4807       +5     
  Lines        282965   283159     +194     
  Branches      40792    40819      +27     
============================================
+ Hits         200356   200440      +84     
- Misses        66187    66271      +84     
- Partials      16422    16448      +26

Impacted Files	Coverage Δ
...min/cluster/stats/TransportClusterStatsAction.java	`69.56% <ø> (ø)`
...nsearch/index/SegmentReplicationPerGroupStats.java	`0.00% <0.00%> (ø)`
.../org/opensearch/index/SegmentReplicationStats.java	`0.00% <0.00%> (ø)`
...eplication/checkpoint/PublishCheckpointAction.java	`23.80% <0.00%> (+0.37%)`	⬆️
...search/cluster/MockInternalClusterInfoService.java	`0.00% <0.00%> (ø)`
.../java/org/opensearch/test/InternalTestCluster.java	`58.25% <ø> (-0.09%)`	⬇️
...rch/action/admin/cluster/node/stats/NodeStats.java	`50.28% <22.22%> (-1.55%)`	⬇️
...ensearch/index/SegmentReplicationStatsTracker.java	`23.07% <23.07%> (ø)`
...opensearch/index/SegmentReplicationShardStats.java	`28.12% <28.12%> (ø)`
...src/main/java/org/opensearch/node/NodeService.java	`69.69% <50.00%> (-0.62%)`	⬇️
... and 486 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

Signed-off-by: Marc Handalian <handalm@amazon.com>

github-actions · 2023-03-03T04:02:57Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/11931/
CommitID: ab72c19

Also update to return an array of replicas vs a map of objects. Signed-off-by: Marc Handalian <handalm@amazon.com>

github-actions · 2023-03-03T05:32:06Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.index.IndexServiceTests.testAsyncTranslogTrimTaskOnClosedIndex

URL: https://build.ci.opensearch.org/job/gradle-check/11933/
CommitID: 3a135e9
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

github-actions · 2023-03-03T08:47:00Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/11934/
CommitID: 3997594
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-03-03T09:06:02Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/11935/
CommitID: c4d8ac1
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

… segments. This can happen when there is a new checkpoint published because of a SegmentInfos bump, but the replica does not need to fetch any files. Signed-off-by: Marc Handalian <handalm@amazon.com>

github-actions · 2023-03-03T18:06:57Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.indices.replication.SegmentReplicationIT.testCancellation
      1 org.opensearch.cluster.allocation.AwarenessAllocationIT.testThreeZoneOneReplicaWithForceZoneValueAndLoadAwareness

URL: https://build.ci.opensearch.org/job/gradle-check/11945/
CommitID: c0b6968
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

Signed-off-by: Marc Handalian <handalm@amazon.com>

github-actions · 2023-03-03T22:34:01Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/11956/
CommitID: 7fda8b7

server/src/main/java/org/opensearch/action/admin/cluster/node/stats/NodeStats.java

server/src/main/java/org/opensearch/index/SegmentReplicationShardStats.java

server/src/main/java/org/opensearch/action/admin/cluster/node/stats/NodeStats.java

server/src/main/java/org/opensearch/index/seqno/ReplicationTracker.java

server/src/main/java/org/opensearch/indices/replication/SegmentReplicationSourceHandler.java

server/src/main/java/org/opensearch/index/seqno/ReplicationTracker.java

Signed-off-by: Marc Handalian <handalm@amazon.com>

github-actions · 2023-03-06T00:14:57Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/11988/
CommitID: 2c69235

dreamer-89 · 2023-03-06T20:43:22Z

server/src/main/java/org/opensearch/index/SegmentReplicationStatsTracker.java

+ *
+ * @opensearch.internal
+ */
+public class SegmentReplicationStatsTracker {


nit: Can this class be removed with getStats() moved to SegmentReplicationPressureService as I don't see this class maintaining/tracking segrep stats. Please ignore if there is any pending future work which demands this class's existence.

Implemented with this commit - 473cc02. I've opened that up in draft and will open it for review once this lands.

mch2 · 2023-03-07T06:44:18Z

Would like a second reviewer on this and 473cc02. @Bukhtawar, @sachinpkale, @ashking94 available for review?

Note - 473cc02 implements actual enforcement of pressure from these computed metrics. I'll also be putting up a separate PR to actually check for and fail stale replicas from both the primary & replica sides.

mch2 · 2023-03-12T20:07:42Z

@dreamer-89 I think we should consolidate these stats into the segment_replication API rather than in NodeStats. Will discard this and open a separate PR.

This comment was marked as outdated.

Sign in to view

mch2 force-pushed the bp branch from 1d04bb7 to d172b4a Compare March 1, 2023 06:19

This comment was marked as outdated.

Sign in to view

mch2 force-pushed the bp branch from 1fafab4 to b54c96d Compare March 1, 2023 07:07

This comment was marked as outdated.

Sign in to view

mch2 force-pushed the bp branch from b54c96d to b72855a Compare March 1, 2023 07:38

This comment was marked as outdated.

Sign in to view

mch2 mentioned this pull request Mar 1, 2023

[Segment Replication & Remote Translog] Back-pressure and Recovery for lagging replica copies #4478

Closed

mch2 force-pushed the bp branch from b72855a to 96bbc77 Compare March 1, 2023 08:16

This comment was marked as outdated.

Sign in to view

mch2 force-pushed the bp branch from c7e0133 to 75c4874 Compare March 2, 2023 00:16

This comment was marked as outdated.

Sign in to view

mch2 force-pushed the bp branch from 75c4874 to 5199d5b Compare March 2, 2023 01:18

This comment was marked as outdated.

Sign in to view

mch2 force-pushed the bp branch from 5199d5b to 3a510b2 Compare March 2, 2023 01:57

This comment was marked as outdated.

Sign in to view

mch2 force-pushed the bp branch 2 times, most recently from 4d062a2 to 9e6d73c Compare March 2, 2023 04:03

This comment was marked as outdated.

Sign in to view

mch2 force-pushed the bp branch 2 times, most recently from 5720723 to 4372177 Compare March 2, 2023 18:36

mch2 force-pushed the bp branch from 4372177 to dabec65 Compare March 2, 2023 18:45

Fix NodeStats bwc tests..

348b34d

Signed-off-by: Marc Handalian <handalm@amazon.com>

mch2 force-pushed the bp branch from 51e9fb2 to 348b34d Compare March 2, 2023 20:07

dreamer-89 reviewed Mar 2, 2023

View reviewed changes

PR feedback

ab72c19

Signed-off-by: Marc Handalian <handalm@amazon.com>

Update API response to include node Id instead of allocation ID.

3a135e9

Also update to return an array of replicas vs a map of objects. Signed-off-by: Marc Handalian <handalm@amazon.com>

mch2 force-pushed the bp branch from 3997594 to c4d8ac1 Compare March 3, 2023 08:46

Fix bug where timer would not be stopped if replica did not fetch any…

c0b6968

… segments. This can happen when there is a new checkpoint published because of a SegmentInfos bump, but the replica does not need to fetch any files. Signed-off-by: Marc Handalian <handalm@amazon.com>

mch2 force-pushed the bp branch from c4d8ac1 to c0b6968 Compare March 3, 2023 17:37

Revert rename of segment_replication path.

7fda8b7

Signed-off-by: Marc Handalian <handalm@amazon.com>

dreamer-89 reviewed Mar 5, 2023

View reviewed changes

PR feedback.

2c69235

Signed-off-by: Marc Handalian <handalm@amazon.com>

This comment was marked as outdated.

Sign in to view

dreamer-89 reviewed Mar 6, 2023

View reviewed changes

dreamer-89 approved these changes Mar 6, 2023

View reviewed changes

mch2 mentioned this pull request Mar 7, 2023

Implement Segment replication Backpressure #6563

Merged

6 tasks

mch2 mentioned this pull request Mar 9, 2023

[Segment Replication] Send Remote shard failure for lagging replicas #6606

Closed

mch2 closed this Mar 12, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Compute Segment Replication stats for SR backpressure. #6520

Compute Segment Replication stats for SR backpressure. #6520

mch2 commented Mar 1, 2023 •

edited

Loading

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

github-actions bot commented Mar 2, 2023

dreamer-89 left a comment

github-actions bot commented Mar 2, 2023

codecov-commenter commented Mar 2, 2023 •

edited

Loading

github-actions bot commented Mar 3, 2023

github-actions bot commented Mar 3, 2023

github-actions bot commented Mar 3, 2023

github-actions bot commented Mar 3, 2023

github-actions bot commented Mar 3, 2023

github-actions bot commented Mar 3, 2023

This comment was marked as outdated.

This comment was marked as outdated.

github-actions bot commented Mar 6, 2023

dreamer-89 Mar 6, 2023

mch2 Mar 7, 2023

mch2 commented Mar 7, 2023

mch2 commented Mar 12, 2023

Compute Segment Replication stats for SR backpressure. #6520

Compute Segment Replication stats for SR backpressure. #6520

Conversation

mch2 commented Mar 1, 2023 • edited Loading

Description

Issues Resolved

Check List

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

github-actions bot commented Mar 2, 2023

Gradle Check (Jenkins) Run Completed with:

dreamer-89 left a comment

Choose a reason for hiding this comment

github-actions bot commented Mar 2, 2023

Gradle Check (Jenkins) Run Completed with:

codecov-commenter commented Mar 2, 2023 • edited Loading

Codecov Report

github-actions bot commented Mar 3, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Mar 3, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Mar 3, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Mar 3, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Mar 3, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Mar 3, 2023

Gradle Check (Jenkins) Run Completed with:

This comment was marked as outdated.

This comment was marked as outdated.

github-actions bot commented Mar 6, 2023

Gradle Check (Jenkins) Run Completed with:

dreamer-89 Mar 6, 2023

Choose a reason for hiding this comment

mch2 Mar 7, 2023

Choose a reason for hiding this comment

mch2 commented Mar 7, 2023

mch2 commented Mar 12, 2023

mch2 commented Mar 1, 2023 •

edited

Loading

codecov-commenter commented Mar 2, 2023 •

edited

Loading