[Flaky Test] Fix Flaky Test SearchTimeoutIT.testSimpleTimeout

[kkewwei]

Description

When numDocs=1000:

1. The testSimpleTimeout will cost several minutes, when scoring each doc, it will cost 500ms, it's a long time to iterating all the doc in queryphase.
   

   OpenSearch/server/src/internalClusterTest/java/org/opensearch/search/SearchTimeoutIT.java

   Line 138 in 5aa6509

   Thread.sleep(500);

2. ReaderContext is created before executing queryphase and released after the fetchphase.
3. the life time of ReaderContext is 1min(determined by search.keep_alive_interval)
4. If queryPhase costs too much time, the ReaderContext may be released before fetchphase, so the fetch/Id will be failed, which hit the case.
   

Related Issues

Resolves #16056 #9401

Check List

• Functionality includes testing.
• API changes companion pull request created, if applicable.
• Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

✅ Gradle check result for 51c35eb: SUCCESS

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 72.09%. Comparing base (5ba909a) to head (51c35eb).
Report is 4 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #16828      +/-   ##
============================================
- Coverage     72.11%   72.09%   -0.02%     
+ Complexity    65237    65176      -61     
============================================
  Files          5318     5318              
  Lines        304003   304011       +8     
  Branches      43992    43993       +1     
============================================
- Hits         219228   219190      -38     
+ Misses        66874    66834      -40     
- Partials      17901    17987      +86     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[github-actions]

❌ Gradle check result for d1fc8be: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[kkewwei]

❌ Gradle check result for d1fc8be: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

SpecificClusterManagerNodesIT.testElectOnlyBetweenClusterManagerNodes #15944