Prevent deletion of snapshot backing index

[Vishalks]

Description

The changes prevent deletion of snapshots that are backing searchable snapshot indexes

Issues Resolved

#4967

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/6346/
• CommitID: e9f450a
  Please examine the workflow log, locate, and copy-paste the failure below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/6347/
• CommitID: bf32979
  Please examine the workflow log, locate, and copy-paste the failure below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/6352/
• CommitID: 3b43e06
  Please examine the workflow log, locate, and copy-paste the failure below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/6394/
• CommitID: eabebd8
  Please examine the workflow log, locate, and copy-paste the failure below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/6395/
• CommitID: cca5ccc
  Please examine the workflow log, locate, and copy-paste the failure below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/6397/
• CommitID: 0228972
  Please examine the workflow log, locate, and copy-paste the failure below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[andrross]

Can you add some tests of the new behavior?

[Vishalks]

Can you add some tests of the new behavior?

Already working on it while I get any review comments on the PR.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/6713/
• CommitID: dbbe321
  Please examine the workflow log, locate, and copy-paste the failure below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/6871/
• CommitID: e09a8b9
  Please examine the workflow log, locate, and copy-paste the failure below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/7092/
• CommitID: fb6d33d
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/7093/
• CommitID: 081a207

[codecov-commenter]

Codecov Report

Merging #5069 (575ff53) into main (933e8c3) will decrease coverage by 0.00%.
The diff coverage is 86.36%.

@@             Coverage Diff              @@
##               main    #5069      +/-   ##
============================================
- Coverage     71.07%   71.07%   -0.01%     
- Complexity    58135    58141       +6     
============================================
  Files          4704     4705       +1     
  Lines        277266   277288      +22     
  Branches      40147    40151       +4     
============================================
  Hits         197080   197080              
- Misses        64001    64021      +20     
- Partials      16185    16187       +2     

Impacted Files	Coverage Δ
...arch/snapshots/SnapshotInUseDeletionException.java	40.00% <40.00%> (ø)
.../main/java/org/opensearch/OpenSearchException.java	92.35% <100.00%> (-0.20%)	⬇️
...n/java/org/opensearch/snapshots/SnapshotUtils.java	87.30% <100.00%> (+3.96%)	⬆️
...ava/org/opensearch/snapshots/SnapshotsService.java	47.82% <100.00%> (-2.18%)	⬇️
...rch/test/junit/listeners/ReproduceInfoPrinter.java	9.52% <0.00%> (-66.67%)	⬇️
...cluster/coordination/PendingClusterStateStats.java	20.00% <0.00%> (-48.00%)	⬇️
...opensearch/persistent/PersistentTasksExecutor.java	22.22% <0.00%> (-44.45%)	⬇️
...adcast/BroadcastShardOperationFailedException.java	55.55% <0.00%> (-44.45%)	⬇️
.../admin/cluster/reroute/ClusterRerouteResponse.java	60.00% <0.00%> (-40.00%)	⬇️
...luster/routing/allocation/RoutingExplanations.java	62.06% <0.00%> (-37.94%)	⬇️
... and 493 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/7094/
• CommitID: 6b601ee

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.index.ShardIndexingPressureConcurrentExecutionTests.testReplicaThreadedUpdateToShardLimitsAndRejections

• URL: https://build.ci.opensearch.org/job/gradle-check/7095/
• CommitID: 8434fbd
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[kotwanikunal]

LGTM.

[dblock]

Do we have similar behavior elsewhere where we return a 403 when some resource is holding the one being deleted?

By returning FORBIDDEN (401) we are breaking the contract in which FORBIDDEN means "server doesn't want to tell you why this is not allowed". We are explicitly telling the client that this is not allowed. Should this be a 400? Or a 405 (operation not allowed?), or some kind of other not permitted error?

[dblock]

I'll hit approve. Leaving it to @andrross to final-comment, if you think we're making a mistake, then let's discuss. I am not sure anymore.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.smoketest.SmokeTestMultiNodeClientYamlTestSuiteIT.test {yaml=pit/10_basic/Delete all}

• URL: https://build.ci.opensearch.org/job/gradle-check/7185/
• CommitID: 6481b11
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/7249/
• CommitID: 8341095
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/7250/
• CommitID: eb73220
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      3 org.opensearch.index.ShardIndexingPressureConcurrentExecutionTests.testReplicaThreadedUpdateToShardLimitsAndRejections

• URL: https://build.ci.opensearch.org/job/gradle-check/7253/
• CommitID: e5ec3aa
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/7259/
• CommitID: 575ff53

[opensearch-trigger-bot]

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-5069-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 08ac17fd7b9a604ecffee0a8e0d666f4dd372dba
# Push it to GitHub
git push --set-upstream origin backport/backport-5069-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-5069-to-2.x.

[dblock]

Is this a breaking change for 2.x? Was the user experience broken when these snapshots were deleted in 2.4.0?

@Vishalks this will need a manual backport

[andrross]

Was the user experience broken when these snapshots were deleted in 2.4.0?

@dblock Yes, it was broken. Searchable snapshots are still behind a feature flag so we have some latitude for changing behavior here.

[Vishalks]

Is this a breaking change for 2.x? Was the user experience broken when these snapshots were deleted in 2.4.0?

@Vishalks this will need a manual backport

Starting the backport