Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Backport 2.x] Segment Replication - Update /_cat/segment_replication API with backpressure metrics. (#6674) #6727

Closed
wants to merge 1 commit into from

Conversation

mch2
Copy link
Member

@mch2 mch2 commented Mar 16, 2023

Manual backport of #6674

…ressure metrics. (opensearch-project#6674)

* Segment Replication - Update Segment Replication API with backpressure metrics.

This change updates the existing /_cat/segment_replication API to include backpressure metrics.
It does this by returning stats from primary shards for its tracked replication group and merging it with metrics returned from replicas.
Primary captured  metrics will now appear by default, with replica per sync events showing when detailed=true is set.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* PR Feedback.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Fixed current_lag header alias.

Signed-off-by: Marc Handalian <handalm@amazon.com>

---------

Signed-off-by: Marc Handalian <handalm@amazon.com>
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@dreamer-89
Copy link
Member

Gradle Check (Jenkins) Run Completed with:

> Task :server:internalClusterTest

REPRODUCE WITH: ./gradlew ':server:internalClusterTest' --tests "org.opensearch.cluster.shards.ClusterShardLimitIT.testCreateIndexWithMaxClusterShardSetting" -Dtests.seed=89B60E8FE0F69067 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=da -Dtests.timezone=Asia/Kashgar -Druntime.java=17

org.opensearch.cluster.shards.ClusterShardLimitIT > testCreateIndexWithMaxClusterShardSetting FAILED
    java.lang.AssertionError: MaxShardPerCluster 3 should be greater than or equal to MaxShardPerNode 1000
        at __randomizedtesting.SeedInfo.seed([89B60E8FE0F69067:C350FF314DA856DF]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.opensearch.cluster.shards.ClusterShardLimitIT.setMaxShardLimit(ClusterShardLimitIT.java:750)
        at org.opensearch.cluster.shards.ClusterShardLimitIT.testCreateIndexWithMaxClusterShardSetting(ClusterShardLimitIT.java:259)

@dreamer-89
Copy link
Member

Gradle Check (Jenkins) Run Completed with:

REPRODUCE WITH: ./gradlew ':server:test' --tests "org.opensearch.index.translog.RemoteFSTranslogTests.testConcurrentWriteViewsAndSnapshot" -Dtests.seed=C95CCC85159CEF44 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=pl -Dtests.timezone=Africa/Asmera -Druntime.java=17

org.opensearch.index.translog.RemoteFSTranslogTests > testConcurrentWriteViewsAndSnapshot FAILED
    com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=241, name=writer_1, state=RUNNABLE, group=TGRP-RemoteFSTranslogTests]
        at __randomizedtesting.SeedInfo.seed([C95CCC85159CEF44:ED8535A8CE0B1CC7]:0)

        Caused by:
        java.lang.AssertionError: Expected non-empty readers
            at __randomizedtesting.SeedInfo.seed([C95CCC85159CEF44]:0)
            at org.opensearch.index.translog.RemoteFsTranslog.deleteStaleRemotePrimaryTermsAndMetadataFiles(RemoteFsTranslog.java:398)
            at org.opensearch.index.translog.RemoteFsTranslog.trimUnreferencedReaders(RemoteFsTranslog.java:368)
            at org.opensearch.index.translog.RemoteFSTranslogTests$2.doRun(RemoteFSTranslogTests.java:863)

@Rishikesh1159
Copy link
Member

Looks like test causing gradle check to fail shards.ClusterShardLimitIT > testCreateIndexWithMaxClusterShardSetting is not related to this PR. I can reproduce the test failing on opensearch 2.x branch also with seeds: -Dtests.seed=C95CCC85159CEF44, -Dtests.seed=89B60E8FE0F69067

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@Rishikesh1159
Copy link
Member

Need to rebase

@Rishikesh1159
Copy link
Member

Closing this PR. Another PR:#6741 manually backported this changes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants