Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mute flaky testThreeZoneOneReplicaWithForceZoneValueAndLoadAwareness #11767

Merged
merged 1 commit into from
Jan 8, 2024

Conversation

imRishN
Copy link
Member

@imRishN imRishN commented Jan 5, 2024

Description

This change mutes the flaky test - testThreeZoneOneReplicaWithForceZoneValueAndLoadAwareness.

The test fails frequently because OpenSearch in a way follows greedy approach while allocating shards and doesn't compute the optimal allocation for all the shards that needs to be allocated. This approach based on certain filters and rules tries to control nodes where shards are assigned.

The unassigned shards causing test failure is due to the same above reason where a node where the shard was supposed to be assigned created a conflict with the awareness allocation decider. Hence, it is stuck in a state, waiting for space to allocate the unassigned shard because it cannot assign it to the only node with space. This also seemed more likely to happen in this particular test case because it is creating a 15 nodes cluster and over 120 shards which increases the probability of landing up in such a case. A smaller cluster with lesser shards would be less likely to land up in such a case.

Disabling this test till we fix up the allocation algorithm which can be taken up as part of #10554

Related Issues

Resolves #[Issue number to be closed when this PR is merged]
#7401 #3603

Check List

  • [ ] New functionality includes testing.
    • [ ] All tests pass
  • [] New functionality has been documented.
    • [ ] New functionality has javadoc added
  • [ ] Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
  • Commits are signed per the DCO using --signoff
  • [] Commit changes are listed out in CHANGELOG.md file (See: Changelog)
  • [ ] Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Copy link
Contributor

github-actions bot commented Jan 5, 2024

❌ Gradle check result for 69d30c1: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Copy link
Contributor

github-actions bot commented Jan 5, 2024

Compatibility status:

Checks if related components are compatible with change 5b53974

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/sql.git]

Signed-off-by: Rishab Nahata <rnnahata@amazon.com>
Copy link
Contributor

github-actions bot commented Jan 7, 2024

❕ Gradle check result for 5b53974: UNSTABLE

  • TEST FAILURES:
      1 org.opensearch.remotestore.RemoteIndexPrimaryRelocationIT.testPrimaryRelocationWhileIndexing

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

Copy link

codecov bot commented Jan 7, 2024

Codecov Report

Attention: 165 lines in your changes are missing coverage. Please review.

Comparison is base (36bd674) 71.27% compared to head (5b53974) 71.23%.
Report is 10 commits behind head on main.

Files Patch % Lines
...org/opensearch/index/mapper/NumberFieldMapper.java 43.62% 61 Missing and 23 partials ⚠️
.../document/SortedUnsignedLongDocValuesSetQuery.java 15.09% 43 Missing and 2 partials ⚠️
.../main/java/org/apache/lucene/util/LongHashSet.java 39.62% 21 Missing and 11 partials ⚠️
...ch/common/xcontent/JsonToStringXContentParser.java 81.25% 1 Missing and 2 partials ⚠️
.../org/opensearch/http/netty4/Netty4HttpChannel.java 50.00% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##               main   #11767      +/-   ##
============================================
- Coverage     71.27%   71.23%   -0.05%     
- Complexity    59176    59191      +15     
============================================
  Files          4909     4911       +2     
  Lines        278429   278666     +237     
  Branches      40460    40528      +68     
============================================
+ Hits         198443   198500      +57     
- Misses        63528    63647     +119     
- Partials      16458    16519      +61     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@sachinpkale sachinpkale changed the title Fix flaky testThreeZoneOneReplicaWithForceZoneValueAndLoadAwareness Mute flaky testThreeZoneOneReplicaWithForceZoneValueAndLoadAwareness Jan 8, 2024
@imRishN imRishN marked this pull request as ready for review January 8, 2024 06:52
@sachinpkale sachinpkale merged commit 89e4727 into opensearch-project:main Jan 8, 2024
94 of 130 checks passed
@sachinpkale sachinpkale added the backport 2.x Backport to 2.x branch label Jan 8, 2024
opensearch-trigger-bot bot pushed a commit that referenced this pull request Jan 8, 2024
…11767)

Signed-off-by: Rishab Nahata <rnnahata@amazon.com>
(cherry picked from commit 89e4727)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
sachinpkale pushed a commit that referenced this pull request Jan 8, 2024
…11767) (#11787)

(cherry picked from commit 89e4727)

Signed-off-by: Rishab Nahata <rnnahata@amazon.com>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
rayshrey pushed a commit to rayshrey/OpenSearch that referenced this pull request Mar 18, 2024
shiv0408 pushed a commit to Gaurav614/OpenSearch that referenced this pull request Apr 25, 2024
…pensearch-project#11767)

Signed-off-by: Rishab Nahata <rnnahata@amazon.com>
Signed-off-by: Shivansh Arora <hishiv@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport 2.x Backport to 2.x branch skip-changelog
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants