[Backport 2.x] Fix weighted shard routing state across search requests

[opensearch-trigger-bot]

Backport cd860ec from #6004.

[codecov-commenter]

Codecov Report

Merging #6060 (cd860ec) into 2.x (579f1a6) will increase coverage by 0.31%.
The diff coverage is 50.13%.

❗ Current head cd860ec differs from pull request most recent head ccf77fa. Consider uploading reports for the commit ccf77fa to get more accurate results

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

@@             Coverage Diff              @@
##                2.x    #6060      +/-   ##
============================================
+ Coverage     70.37%   70.68%   +0.31%     
+ Complexity    58890    58714     -176     
============================================
  Files          4769     4775       +6     
  Lines        282621   280993    -1628     
  Branches      41169    40590     -579     
============================================
- Hits         198899   198626     -273     
+ Misses        67063    66031    -1032     
+ Partials      16659    16336     -323     

Impacted Files	Coverage Δ
...ark/time/NanoTimeVsCurrentTimeMillisBenchmark.java	0.00% <0.00%> (ø)
...nsearch/gradle/precommit/LicenseHeadersTask.groovy	0.00% <ø> (ø)
.../java/org/opensearch/gradle/JdkDownloadPlugin.java	36.26% <0.00%> (-1.10%)	⬇️
...rg/opensearch/gradle/OpenSearchTestBasePlugin.java	6.81% <0.00%> (-0.25%)	⬇️
...ternal/InternalDistributionArchiveCheckPlugin.java	0.00% <0.00%> (ø)
...ternal/InternalDistributionArchiveSetupPlugin.java	0.00% <ø> (ø)
...ensearch/gradle/precommit/FilePermissionsTask.java	78.78% <ø> (ø)
...search/gradle/precommit/ForbiddenPatternsTask.java	88.40% <ø> (ø)
...g/opensearch/gradle/precommit/LoggerUsageTask.java	0.00% <ø> (ø)
...ensearch/gradle/precommit/ThirdPartyAuditTask.java	0.00% <0.00%> (ø)
... and 1049 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/10131/
• CommitID: ccf77fa
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/10271/
• CommitID: ccf77fa
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/10548/
• CommitID: ccf77fa

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.cluster.routing.allocation.decider.DiskThresholdDeciderIT.testIndexCreateBlockWithAReadOnlyBlock
      1 org.opensearch.cluster.routing.allocation.decider.DiskThresholdDeciderIT.testIndexCreateBlockWhenAllNodesExceededHighWatermark

• URL: https://build.ci.opensearch.org/job/gradle-check/10551/
• CommitID: 86e3821
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.