Fix flaky tests 'testDeleteAllAndListAllPits' and 'testDeleteWhileSearch'

[bharath-techie]

Description

testDeleteAllAndListAllPits :
List all PITs and delete all PITs both were triggered asynchronously and test was flaky in times where delete all finishes before get all. So fixed the test to order these two calls ( get will always finish first and then delete will be triggered )

testDeleteWhileSearch:
Added check for an error message which is also one of the valid scenarios

Tested using :

./gradlew ':client:rest-high-level:asyncIntegTest' --tests "org.opensearch.client.PitIT.testDeleteAllAndListAllPits" -Dtests.iters=1000

 ./gradlew ':server:internalClusterTest' --tests "org.opensearch.search.pit.DeletePitMultiNodeIT.testDeleteWhileSearch" -Dtests.iters=1000

Related Issues

• [BUG] PitIT.testDeleteAllAndListAllPits test failure #5030
• [BUG] Test DeletePitMultiNodeIT.testDeleteWhileSearch is flaky #9048

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented. [Not applicable]
  • New functionality has javadoc added
• Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog) [Not applicable]
• Public documentation issue/PR created [Not applicable]

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Compatibility status:

Checks if related components are compatible with change 611e482

Incompatible components

Incompatible components: [https://github.com/opensearch-project/performance-analyzer.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/neural-search.git]

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.index.shard.RemoteIndexShardTests.testSegRepSucceedsOnPreviousCopiedFiles

• URL: https://build.ci.opensearch.org/job/gradle-check/29078/
• CommitID: 6d0f429
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[codecov]

Codecov Report

Merging #10946 (611e482) into main (be65f54) will increase coverage by 0.04%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##               main   #10946      +/-   ##
============================================
+ Coverage     71.21%   71.26%   +0.04%     
- Complexity    58689    58714      +25     
============================================
  Files          4870     4870              
  Lines        276539   276539              
  Branches      40190    40190              
============================================
+ Hits         196945   197078     +133     
+ Misses        63192    62987     -205     
- Partials      16402    16474      +72     

see 471 files with indirect coverage changes

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/29079/
• CommitID: 9211542
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/29082/
• CommitID: 2b5c5d6
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/29123/
• CommitID: 3731fe9
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/29135/
• CommitID: f22f628
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.index.reindex.DeleteByQueryBasicTests.testDeleteByQueryWithMultipleIndices

• URL: https://build.ci.opensearch.org/job/gradle-check/29161/
• CommitID: 611e482
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[peternied]

Failing checks are inspected and point to the corresponding known issue

@bharath-techie It looks like there were many failing checks - can you please inspect and create/update issues before this is merged?

[bharath-techie]

Gradle check failures known issues seen in this PR checks :
#9499
#10755
#10945
#10902
#10154

Created issues :
#10972 - RemoteStoreIT flaky test

[bharath-techie]

Can someone help merging this PR ?

[bharath-techie]

@kotwanikunal @peternied can you help review / merge the code if the fix looks good ?

[peternied]

Gradle check failures known issues seen in this PR checks :
...

@bharath-techie Thank you very much!

[peternied]

Thanks for addressing this flaky test @bharath-techie