[Backport 2.x] [Remote State] fix lock release before deletion is completed

[opensearch-trigger-bot]

Backport c400d84 from #10611.

[github-actions]

Compatibility status:

Checks if related components are compatible with change 53a5ad5

Incompatible components

Incompatible components: [https://github.com/opensearch-project/security-analytics.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/neural-search.git]

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/28578/
• CommitID: 53a5ad5
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[linuxpi]

Failed Test -

org.opensearch.index.shard.IndexShardTests.testRefreshLevelRestoreShardFromRemoteStore

Ran locally with 10 iterations, none failed

./gradlew ':server:test' --tests "org.opensearch.index.shard.IndexShardTests.testRefreshLevelRestoreShardFromRemoteStore" -Dtests.iters=10

created flaky test bug #10785

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/28661/
• CommitID: 53a5ad5
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[linuxpi]

failed tests

org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock #10006
org.opensearch.search.aggregations.metrics.CardinalityWithRequestBreakerIT.testRequestBreaker {p0={"search.concurrent_segment_search.enabled":"true"}} #10154
org.opensearch.search.suggest.CompletionSuggestSearchIT.testSkipDuplicates {p0={"search.concurrent_segment_search.enabled":"true"}} #10798

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/28665/
• CommitID: 53a5ad5

[codecov]

Codecov Report

Merging #10778 (53a5ad5) into 2.x (8896f58) will decrease coverage by 0.03%.
Report is 4 commits behind head on 2.x.
The diff coverage is 78.94%.

@@             Coverage Diff              @@
##                2.x   #10778      +/-   ##
============================================
- Coverage     70.94%   70.91%   -0.03%     
+ Complexity    58705    58660      -45     
============================================
  Files          4843     4843              
  Lines        277261   277281      +20     
  Branches      40673    40675       +2     
============================================
- Hits         196690   196647      -43     
- Misses        63858    63870      +12     
- Partials      16713    16764      +51     

Files	Coverage Δ
.../indices/replication/common/ReplicationTarget.java	72.50% <100.00%> (-5.14%)	⬇️
...arch/gateway/remote/RemoteClusterStateService.java	69.74% <33.33%> (+0.19%)	⬆️
...nsearch/index/store/RemoteStoreFileDownloader.java	92.15% <84.00%> (-0.87%)	⬇️
...ices/replication/RemoteStoreReplicationSource.java	85.24% <75.00%> (-5.83%)	⬇️

... and 500 files with indirect coverage changes