Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Segment Replication] Update bwc test to rely on segments for verification #8267

Merged
merged 2 commits into from
Jun 28, 2023

Conversation

dreamer-89
Copy link
Member

@dreamer-89 dreamer-89 commented Jun 26, 2023

Description

This change improves the testIndexingWithSegRep bwc test by relying on _cat/segments API to verify shard store and doc count and is preferred because of below reasons:

  1. index/_stats API does not return data for all shards specially when it is newly created (this is probably not all shard copy have documents and thus no stats), thus test fails as mentioned here [Backport] [Segment Replication] Added mixed and rolling upgrade bwc test (#7537) #7626 (comment)
  2. The only reason for _stats was to build nodeId to shardId map so that targeted _search can be made using "_shards:" + shardNumber + "|_only_nodes:" + nodeId preference. This is unnecessary two step process.
  3. _cat/segments resonates better with segment replication as we are verifying segment names and doc count.

Check List

  • New functionality includes testing.
    • All tests pass
  • New functionality has been documented.
    • New functionality has javadoc added
  • Commits are signed per the DCO using --signoff
  • Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

  • RESULT: UNSTABLE ❕
  • TEST FAILURES:
      1 org.opensearch.cluster.allocation.ClusterRerouteIT.testDelayWithALargeAmountOfShards
      1 org.opensearch.action.admin.indices.create.CreateIndexIT.testCreateAndDeleteIndexConcurrently
      1 org.opensearch.action.admin.indices.create.CreateIndexIT.classMethod

@codecov
Copy link

codecov bot commented Jun 26, 2023

Codecov Report

Merging #8267 (e57f170) into main (4c02dd1) will increase coverage by 0.02%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##               main    #8267      +/-   ##
============================================
+ Coverage     70.90%   70.92%   +0.02%     
- Complexity    56710    56764      +54     
============================================
  Files          4733     4733              
  Lines        268333   268333              
  Branches      39322    39322              
============================================
+ Hits         190254   190325      +71     
+ Misses        61996    61986      -10     
+ Partials      16083    16022      -61     

see 451 files with indirect coverage changes

…erify doc count

Signed-off-by: Suraj Singh <surajrider@gmail.com>

Self review

Signed-off-by: Suraj Singh <surajrider@gmail.com>

remove unused imports

Signed-off-by: Suraj Singh <surajrider@gmail.com>

Handle 0 doc count segments

Signed-off-by: Suraj Singh <surajrider@gmail.com>
Signed-off-by: Suraj Singh <surajrider@gmail.com>
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

  • RESULT: UNSTABLE ❕
  • TEST FAILURES:
      1 org.opensearch.snapshots.RestoreSnapshotIT.testRestoreInSameRemoteStoreEnabledIndex
      1 org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testNodeDropWithOngoingReplication

@dreamer-89
Copy link
Member Author

@kotwanikunal : Request for review

@dreamer-89 dreamer-89 merged commit 270abeb into opensearch-project:main Jun 28, 2023
dreamer-89 added a commit to dreamer-89/OpenSearch that referenced this pull request Jun 28, 2023
…ation (opensearch-project#8267)

* [Segment Replication] Use _cat/segments vs index stats + _search to verify doc count

Signed-off-by: Suraj Singh <surajrider@gmail.com>

Self review

Signed-off-by: Suraj Singh <surajrider@gmail.com>

remove unused imports

Signed-off-by: Suraj Singh <surajrider@gmail.com>

Handle 0 doc count segments

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Use 1 minute timeout for assertBusy validations and comments

Signed-off-by: Suraj Singh <surajrider@gmail.com>

---------

Signed-off-by: Suraj Singh <surajrider@gmail.com>

Add missing import statement

Signed-off-by: Suraj Singh <surajrider@gmail.com>
dreamer-89 added a commit to dreamer-89/OpenSearch that referenced this pull request Jul 11, 2023
…ation (opensearch-project#8267)

* [Segment Replication] Use _cat/segments vs index stats + _search to verify doc count

Signed-off-by: Suraj Singh <surajrider@gmail.com>

Self review

Signed-off-by: Suraj Singh <surajrider@gmail.com>

remove unused imports

Signed-off-by: Suraj Singh <surajrider@gmail.com>

Handle 0 doc count segments

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Use 1 minute timeout for assertBusy validations and comments

Signed-off-by: Suraj Singh <surajrider@gmail.com>

---------

Signed-off-by: Suraj Singh <surajrider@gmail.com>

Add missing import statement

Signed-off-by: Suraj Singh <surajrider@gmail.com>
dreamer-89 added a commit to dreamer-89/OpenSearch that referenced this pull request Jul 12, 2023
…ation (opensearch-project#8267)

* [Segment Replication] Use _cat/segments vs index stats + _search to verify doc count

Signed-off-by: Suraj Singh <surajrider@gmail.com>

Self review

Signed-off-by: Suraj Singh <surajrider@gmail.com>

remove unused imports

Signed-off-by: Suraj Singh <surajrider@gmail.com>

Handle 0 doc count segments

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Use 1 minute timeout for assertBusy validations and comments

Signed-off-by: Suraj Singh <surajrider@gmail.com>

---------

Signed-off-by: Suraj Singh <surajrider@gmail.com>

Add missing import statement

Signed-off-by: Suraj Singh <surajrider@gmail.com>
dreamer-89 added a commit that referenced this pull request Jul 13, 2023
…test (#7537) (#7626)

* [Segment Replication] Added mixed and rolling upgrade bwc test (#7537)

* [Segment Replication] Added mixed cluster bwc test

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Remove unnecessary gradle task for segrep

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Spotless fix

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Spotless fix

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* [Segment Replication] Rolling upgrade test

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* PR feedback and cleanup

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Verify replica doc count only when it is assigned

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Remove wait for yellow cluster

Signed-off-by: Suraj Singh <surajrider@gmail.com>

---------

Signed-off-by: Suraj Singh <surajrider@gmail.com>
(cherry picked from commit 0788feb)

Add version check to skip test for 1.x bwc branches

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* [Segment Replication] Wait for shard relocation before building node to shard allocation map (#7876)

* [Segment Replication] Wait for shard relocation before building node to shard allocation map

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Remove unused imports

Signed-off-by: Suraj Singh <surajrider@gmail.com>

---------

Signed-off-by: Suraj Singh <surajrider@gmail.com>

Fix imports

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* [Segment Replication] Update segrep bwc tests to verify replica checkpoints and skip tests for 1.x bwc versions (#8203)

* [Segment Replication] Verify segment replication stats in bwc test

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Log cleanup

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Spotless check

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Add version check to skip test for 1.x bwc branches

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Add version check to skip test for 1.x bwc branches for mixed clusters

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Add version string in build to identify bwc version

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Use correct bwc version string

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Address review comments from #7626

Signed-off-by: Suraj Singh <surajrider@gmail.com>

---------

Signed-off-by: Suraj Singh <surajrider@gmail.com>

[Segment Replication] Use _cat/segments vs index stats + _search to verify doc count

Signed-off-by: Suraj Singh <surajrider@gmail.com>

Self review

Signed-off-by: Suraj Singh <surajrider@gmail.com>

remove unused imports

Signed-off-by: Suraj Singh <surajrider@gmail.com>

Handle 0 doc count segments

Signed-off-by: Suraj Singh <surajrider@gmail.com>

Add missing import statement

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* [Segment Replication] Update bwc test to rely on segments for verification (#8267)

* [Segment Replication] Use _cat/segments vs index stats + _search to verify doc count

Signed-off-by: Suraj Singh <surajrider@gmail.com>

Self review

Signed-off-by: Suraj Singh <surajrider@gmail.com>

remove unused imports

Signed-off-by: Suraj Singh <surajrider@gmail.com>

Handle 0 doc count segments

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Use 1 minute timeout for assertBusy validations and comments

Signed-off-by: Suraj Singh <surajrider@gmail.com>

---------

Signed-off-by: Suraj Singh <surajrider@gmail.com>

Add missing import statement

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Fix missing import

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Logger change

Signed-off-by: Suraj Singh <surajrider@gmail.com>

---------

Signed-off-by: Suraj Singh <surajrider@gmail.com>
baba-devv pushed a commit to baba-devv/OpenSearch that referenced this pull request Jul 29, 2023
…ation (opensearch-project#8267)

* [Segment Replication] Use _cat/segments vs index stats + _search to verify doc count

Signed-off-by: Suraj Singh <surajrider@gmail.com>

Self review

Signed-off-by: Suraj Singh <surajrider@gmail.com>

remove unused imports

Signed-off-by: Suraj Singh <surajrider@gmail.com>

Handle 0 doc count segments

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Use 1 minute timeout for assertBusy validations and comments

Signed-off-by: Suraj Singh <surajrider@gmail.com>

---------

Signed-off-by: Suraj Singh <surajrider@gmail.com>
shiv0408 pushed a commit to Gaurav614/OpenSearch that referenced this pull request Apr 25, 2024
…ation (opensearch-project#8267)

* [Segment Replication] Use _cat/segments vs index stats + _search to verify doc count

Signed-off-by: Suraj Singh <surajrider@gmail.com>

Self review

Signed-off-by: Suraj Singh <surajrider@gmail.com>

remove unused imports

Signed-off-by: Suraj Singh <surajrider@gmail.com>

Handle 0 doc count segments

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Use 1 minute timeout for assertBusy validations and comments

Signed-off-by: Suraj Singh <surajrider@gmail.com>

---------

Signed-off-by: Suraj Singh <surajrider@gmail.com>
Signed-off-by: Shivansh Arora <hishiv@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants