Fix AwarenessAttributeDecommissionIT flaky tests

[imRishN]

Description

Few tests in AwarenessAttributeDecommissionIT fails transiently as while making the client request if a viaNode is not specified it uses a RandomizingClient which can let the request land up on a decommissioned node and then we see exception that cluster manager is not discovered which is not the case. This PR aims to always pick a active node to route the request.

Issues Resolved

#5220
#5221
#5189

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/6886/
• CommitID: 7d90a56

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/6887/
• CommitID: a1b83dc
  Please examine the workflow log, locate, and copy-paste the failure below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/6888/
• CommitID: 462b82d
  Please examine the workflow log, locate, and copy-paste the failure below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/6890/
• CommitID: bb8cfbd

[codecov-commenter]

Codecov Report

Merging #5252 (56e7c65) into main (831b3a0) will decrease coverage by 0.06%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##               main    #5252      +/-   ##
============================================
- Coverage     70.97%   70.90%   -0.07%     
+ Complexity    58186    58126      -60     
============================================
  Files          4708     4708              
  Lines        277558   277558              
  Branches      40190    40190              
============================================
- Hits         197006   196813     -193     
- Misses        64444    64621     +177     
- Partials      16108    16124      +16     

Impacted Files	Coverage Δ
...search/indices/recovery/RecoveryTargetHandler.java	0.00% <0.00%> (-100.00%)	⬇️
...a/org/opensearch/client/cluster/ProxyModeInfo.java	0.00% <0.00%> (-55.00%)	⬇️
...ch/transport/ReceiveTimeoutTransportException.java	50.00% <0.00%> (-50.00%)	⬇️
...rch/client/transport/NoNodeAvailableException.java	28.57% <0.00%> (-42.86%)	⬇️
...h/action/ingest/SimulateDocumentVerboseResult.java	60.71% <0.00%> (-39.29%)	⬇️
...java/org/opensearch/threadpool/ThreadPoolInfo.java	56.25% <0.00%> (-37.50%)	⬇️
.../opensearch/client/indices/CloseIndexResponse.java	56.25% <0.00%> (-35.00%)	⬇️
...r/src/main/java/org/opensearch/http/HttpUtils.java	16.66% <0.00%> (-33.34%)	⬇️
...functionscore/ExponentialDecayFunctionBuilder.java	40.00% <0.00%> (-28.00%)	⬇️
...a/org/opensearch/index/mapper/MapperException.java	75.00% <0.00%> (-25.00%)	⬇️
... and 477 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/6895/
• CommitID: 4b7e20f
  Please examine the workflow log, locate, and copy-paste the failure below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/6899/
• CommitID: f08e365

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/6902/
• CommitID: 7ddf3e3

[dblock]

Makes sense, merge?

[imRishN]

Makes sense, merge?

Yes, we can merge this now. I'll push another empty commit to verify it passes again and not flaky anymore

[dblock]

Looks like some transient connectivity failures ... let's keep retrying for now.

@imRishN instead of empty commits you can git commit --amend, then git push origin integ-test-fix -f.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/6906/
• CommitID: 56e7c65

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/6907/
• CommitID: 86af94c

[imRishN]

@dblock Thanks for merging it. Can you please also help me add 2.x backport tag so that this commit doesn't get missed in future releases? Thanks!