-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Segment Replication] Update bwc test to rely on segments for verification #8267
Merged
dreamer-89
merged 2 commits into
opensearch-project:main
from
dreamer-89:update-bwc-test
Jun 28, 2023
Merged
[Segment Replication] Update bwc test to rely on segments for verification #8267
dreamer-89
merged 2 commits into
opensearch-project:main
from
dreamer-89:update-bwc-test
Jun 28, 2023
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dreamer-89
requested review from
reta,
anasalkouz,
andrross,
Bukhtawar,
CEHENKLE,
dblock,
gbbafna,
setiah,
kartg,
kotwanikunal,
mch2,
nknize,
owaiskazi19,
Rishikesh1159,
ryanbogan,
saratvemulapalli,
shwetathareja,
tlfeng,
VachaShah,
dbwiddis and
sachinpkale
as code owners
June 26, 2023 20:47
Gradle Check (Jenkins) Run Completed with:
|
Gradle Check (Jenkins) Run Completed with:
|
Codecov Report
@@ Coverage Diff @@
## main #8267 +/- ##
============================================
+ Coverage 70.90% 70.92% +0.02%
- Complexity 56710 56764 +54
============================================
Files 4733 4733
Lines 268333 268333
Branches 39322 39322
============================================
+ Hits 190254 190325 +71
+ Misses 61996 61986 -10
+ Partials 16083 16022 -61 |
qa/rolling-upgrade/src/test/java/org/opensearch/upgrades/IndexingIT.java
Show resolved
Hide resolved
qa/rolling-upgrade/src/test/java/org/opensearch/upgrades/IndexingIT.java
Show resolved
Hide resolved
Poojita-Raj
approved these changes
Jun 27, 2023
dreamer-89
force-pushed
the
update-bwc-test
branch
from
June 28, 2023 04:20
4de7ebc
to
88af9e9
Compare
…erify doc count Signed-off-by: Suraj Singh <surajrider@gmail.com> Self review Signed-off-by: Suraj Singh <surajrider@gmail.com> remove unused imports Signed-off-by: Suraj Singh <surajrider@gmail.com> Handle 0 doc count segments Signed-off-by: Suraj Singh <surajrider@gmail.com>
Signed-off-by: Suraj Singh <surajrider@gmail.com>
dreamer-89
force-pushed
the
update-bwc-test
branch
from
June 28, 2023 04:20
88af9e9
to
e57f170
Compare
Gradle Check (Jenkins) Run Completed with:
|
Gradle Check (Jenkins) Run Completed with:
|
@kotwanikunal : Request for review |
kotwanikunal
approved these changes
Jun 28, 2023
Rishikesh1159
approved these changes
Jun 28, 2023
dreamer-89
added a commit
to dreamer-89/OpenSearch
that referenced
this pull request
Jun 28, 2023
…ation (opensearch-project#8267) * [Segment Replication] Use _cat/segments vs index stats + _search to verify doc count Signed-off-by: Suraj Singh <surajrider@gmail.com> Self review Signed-off-by: Suraj Singh <surajrider@gmail.com> remove unused imports Signed-off-by: Suraj Singh <surajrider@gmail.com> Handle 0 doc count segments Signed-off-by: Suraj Singh <surajrider@gmail.com> * Use 1 minute timeout for assertBusy validations and comments Signed-off-by: Suraj Singh <surajrider@gmail.com> --------- Signed-off-by: Suraj Singh <surajrider@gmail.com> Add missing import statement Signed-off-by: Suraj Singh <surajrider@gmail.com>
dreamer-89
added a commit
to dreamer-89/OpenSearch
that referenced
this pull request
Jul 11, 2023
…ation (opensearch-project#8267) * [Segment Replication] Use _cat/segments vs index stats + _search to verify doc count Signed-off-by: Suraj Singh <surajrider@gmail.com> Self review Signed-off-by: Suraj Singh <surajrider@gmail.com> remove unused imports Signed-off-by: Suraj Singh <surajrider@gmail.com> Handle 0 doc count segments Signed-off-by: Suraj Singh <surajrider@gmail.com> * Use 1 minute timeout for assertBusy validations and comments Signed-off-by: Suraj Singh <surajrider@gmail.com> --------- Signed-off-by: Suraj Singh <surajrider@gmail.com> Add missing import statement Signed-off-by: Suraj Singh <surajrider@gmail.com>
dreamer-89
added a commit
to dreamer-89/OpenSearch
that referenced
this pull request
Jul 12, 2023
…ation (opensearch-project#8267) * [Segment Replication] Use _cat/segments vs index stats + _search to verify doc count Signed-off-by: Suraj Singh <surajrider@gmail.com> Self review Signed-off-by: Suraj Singh <surajrider@gmail.com> remove unused imports Signed-off-by: Suraj Singh <surajrider@gmail.com> Handle 0 doc count segments Signed-off-by: Suraj Singh <surajrider@gmail.com> * Use 1 minute timeout for assertBusy validations and comments Signed-off-by: Suraj Singh <surajrider@gmail.com> --------- Signed-off-by: Suraj Singh <surajrider@gmail.com> Add missing import statement Signed-off-by: Suraj Singh <surajrider@gmail.com>
dreamer-89
added a commit
that referenced
this pull request
Jul 13, 2023
…test (#7537) (#7626) * [Segment Replication] Added mixed and rolling upgrade bwc test (#7537) * [Segment Replication] Added mixed cluster bwc test Signed-off-by: Suraj Singh <surajrider@gmail.com> * Remove unnecessary gradle task for segrep Signed-off-by: Suraj Singh <surajrider@gmail.com> * Spotless fix Signed-off-by: Suraj Singh <surajrider@gmail.com> * Spotless fix Signed-off-by: Suraj Singh <surajrider@gmail.com> * [Segment Replication] Rolling upgrade test Signed-off-by: Suraj Singh <surajrider@gmail.com> * PR feedback and cleanup Signed-off-by: Suraj Singh <surajrider@gmail.com> * Verify replica doc count only when it is assigned Signed-off-by: Suraj Singh <surajrider@gmail.com> * Remove wait for yellow cluster Signed-off-by: Suraj Singh <surajrider@gmail.com> --------- Signed-off-by: Suraj Singh <surajrider@gmail.com> (cherry picked from commit 0788feb) Add version check to skip test for 1.x bwc branches Signed-off-by: Suraj Singh <surajrider@gmail.com> * [Segment Replication] Wait for shard relocation before building node to shard allocation map (#7876) * [Segment Replication] Wait for shard relocation before building node to shard allocation map Signed-off-by: Suraj Singh <surajrider@gmail.com> * Remove unused imports Signed-off-by: Suraj Singh <surajrider@gmail.com> --------- Signed-off-by: Suraj Singh <surajrider@gmail.com> Fix imports Signed-off-by: Suraj Singh <surajrider@gmail.com> * [Segment Replication] Update segrep bwc tests to verify replica checkpoints and skip tests for 1.x bwc versions (#8203) * [Segment Replication] Verify segment replication stats in bwc test Signed-off-by: Suraj Singh <surajrider@gmail.com> * Log cleanup Signed-off-by: Suraj Singh <surajrider@gmail.com> * Spotless check Signed-off-by: Suraj Singh <surajrider@gmail.com> * Add version check to skip test for 1.x bwc branches Signed-off-by: Suraj Singh <surajrider@gmail.com> * Add version check to skip test for 1.x bwc branches for mixed clusters Signed-off-by: Suraj Singh <surajrider@gmail.com> * Add version string in build to identify bwc version Signed-off-by: Suraj Singh <surajrider@gmail.com> * Use correct bwc version string Signed-off-by: Suraj Singh <surajrider@gmail.com> * Address review comments from #7626 Signed-off-by: Suraj Singh <surajrider@gmail.com> --------- Signed-off-by: Suraj Singh <surajrider@gmail.com> [Segment Replication] Use _cat/segments vs index stats + _search to verify doc count Signed-off-by: Suraj Singh <surajrider@gmail.com> Self review Signed-off-by: Suraj Singh <surajrider@gmail.com> remove unused imports Signed-off-by: Suraj Singh <surajrider@gmail.com> Handle 0 doc count segments Signed-off-by: Suraj Singh <surajrider@gmail.com> Add missing import statement Signed-off-by: Suraj Singh <surajrider@gmail.com> * [Segment Replication] Update bwc test to rely on segments for verification (#8267) * [Segment Replication] Use _cat/segments vs index stats + _search to verify doc count Signed-off-by: Suraj Singh <surajrider@gmail.com> Self review Signed-off-by: Suraj Singh <surajrider@gmail.com> remove unused imports Signed-off-by: Suraj Singh <surajrider@gmail.com> Handle 0 doc count segments Signed-off-by: Suraj Singh <surajrider@gmail.com> * Use 1 minute timeout for assertBusy validations and comments Signed-off-by: Suraj Singh <surajrider@gmail.com> --------- Signed-off-by: Suraj Singh <surajrider@gmail.com> Add missing import statement Signed-off-by: Suraj Singh <surajrider@gmail.com> * Fix missing import Signed-off-by: Suraj Singh <surajrider@gmail.com> * Logger change Signed-off-by: Suraj Singh <surajrider@gmail.com> --------- Signed-off-by: Suraj Singh <surajrider@gmail.com>
baba-devv
pushed a commit
to baba-devv/OpenSearch
that referenced
this pull request
Jul 29, 2023
…ation (opensearch-project#8267) * [Segment Replication] Use _cat/segments vs index stats + _search to verify doc count Signed-off-by: Suraj Singh <surajrider@gmail.com> Self review Signed-off-by: Suraj Singh <surajrider@gmail.com> remove unused imports Signed-off-by: Suraj Singh <surajrider@gmail.com> Handle 0 doc count segments Signed-off-by: Suraj Singh <surajrider@gmail.com> * Use 1 minute timeout for assertBusy validations and comments Signed-off-by: Suraj Singh <surajrider@gmail.com> --------- Signed-off-by: Suraj Singh <surajrider@gmail.com>
6 tasks
shiv0408
pushed a commit
to Gaurav614/OpenSearch
that referenced
this pull request
Apr 25, 2024
…ation (opensearch-project#8267) * [Segment Replication] Use _cat/segments vs index stats + _search to verify doc count Signed-off-by: Suraj Singh <surajrider@gmail.com> Self review Signed-off-by: Suraj Singh <surajrider@gmail.com> remove unused imports Signed-off-by: Suraj Singh <surajrider@gmail.com> Handle 0 doc count segments Signed-off-by: Suraj Singh <surajrider@gmail.com> * Use 1 minute timeout for assertBusy validations and comments Signed-off-by: Suraj Singh <surajrider@gmail.com> --------- Signed-off-by: Suraj Singh <surajrider@gmail.com> Signed-off-by: Shivansh Arora <hishiv@amazon.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This change improves the
testIndexingWithSegRep
bwc test by relying on _cat/segments API to verify shard store and doc count and is preferred because of below reasons:index/_stats
API does not return data for all shards specially when it is newly created (this is probably not all shard copy have documents and thus no stats), thus test fails as mentioned here [Backport] [Segment Replication] Added mixed and rolling upgrade bwc test (#7537) #7626 (comment)_stats
was to build nodeId to shardId map so that targeted_search
can be made using"_shards:" + shardNumber + "|_only_nodes:" + nodeId
preference. This is unnecessary two step process.Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.