[Backport 2.x] Enhance debug, trace & info logs for remote store flows (#10182)

[ashking94]

Backports e617da3 from #10182

[github-actions]

Compatibility status:

Checks if related components are compatible with change bde1dff

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/neural-search.git]

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/26329/
• CommitID: da2d8bd
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/26334/
• CommitID: cd33926
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/26337/
• CommitID: f2392b1
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: null ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/26367/
• CommitID: 8432507
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/26368/
• CommitID: bde1dff
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/26371/
• CommitID: bde1dff
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testDeleteOperations

• URL: https://build.ci.opensearch.org/job/gradle-check/26372/
• CommitID: bde1dff
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[codecov]

Codecov Report

Merging #10237 (bde1dff) into 2.x (ecb3ed5) will increase coverage by 0.00%.
Report is 3 commits behind head on 2.x.
The diff coverage is 86.27%.

@@            Coverage Diff             @@
##                2.x   #10237    +/-   ##
==========================================
  Coverage     70.74%   70.75%            
+ Complexity    58276    58261    -15     
==========================================
  Files          4814     4814            
  Lines        275430   275523    +93     
  Branches      40493    40501     +8     
==========================================
+ Hits         194853   194940    +87     
+ Misses        63973    63878    -95     
- Partials      16604    16705   +101     

Files	Coverage Δ
...src/main/java/org/opensearch/core/index/Index.java	100.00% <ø> (ø)
...c/main/java/org/opensearch/index/IndexService.java	75.59% <100.00%> (ø)
...rch/index/remote/RemoteSegmentTransferTracker.java	81.10% <100.00%> (+0.17%)	⬆️
...index/shard/CloseableRetryableRefreshListener.java	81.03% <100.00%> (+0.67%)	⬆️
...java/org/opensearch/index/shard/StoreRecovery.java	56.95% <ø> (+0.67%)	⬆️
...earch/index/store/RemoteSegmentStoreDirectory.java	89.67% <100.00%> (-0.40%)	⬇️
...ndex/store/RemoteSegmentStoreDirectoryFactory.java	92.30% <100.00%> (ø)
...rc/main/java/org/opensearch/index/store/Store.java	81.71% <100.00%> (+0.33%)	⬆️
...search/index/translog/InternalTranslogManager.java	66.66% <100.00%> (-4.67%)	⬇️
...rg/opensearch/index/translog/RemoteFsTranslog.java	70.53% <100.00%> (-0.41%)	⬇️
... and 12 more

... and 458 files with indirect coverage changes