"Wait for snapshot" action in Delete phase of ILM doesn't check if index was backed up

[yuliacech]

Hello team,

while adding a field for "wait for snapshot policy" to Delete phase in Index Lifecycle Management UI, I noticed that this action does not in fact ensure that a snapshot of the index exists before deleting the index. This can lead to irreversible data loss of documents in the managed index.

How to recreate this behaviour:

1. Create a repository for snapshots
2. Create a snapshot policy that backs up an index
3. Create a different index (this is the managed index) with an alias for rollover
4. Create an index lifecycle policy that deletes the managed index with "wait for snapshot" option
5. After conditions for Delete phase are met and snapshot policy is executed, the managed index is deleted. This index can't be restored as the snapshot contains a different index.

Expand for console commands

PUT /_snapshot/my_repo
{
  "type": "fs",
  "settings": {
    "location": "./my_repo_test",
    "compress": true
  }
}

(index to be backed up my_snapshot_index, snapshots created every minute and deleted after 10 min)

PUT _slm/policy/my_snapshot_policy
{
  "name": "<snapshot-{now}>",
  "schedule": "0 * * * * ?",
  "repository": "my_repo",
  "config": {
    "indices": [
      "my_snapshot_index"
    ]
  },
  "retention": {
    "expire_after": "10m",
    "min_count": 1,
    "max_count": 3
  }
}

PUT /my_test_index-1
PUT /my_test_index-1/_alias/my_alias

(rollover after 3 docs, delete after my_snapshot_policy created a snapshot).

PUT _ilm/policy/my_policy
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_size": "50gb",
            "max_age": "30d",
            "max_docs": 3
          },
          "set_priority": {
            "priority": 100
          }
        }
      },
      "delete": {
        "min_age": "0d",
        "actions": {
          "wait_for_snapshot": {
            "policy": "my_snapshot_policy"
          },
          "delete": {}
        }
      }
    }
  }
}

[elasticmachine]

Pinging @elastic/es-core-features (:Core/Features/ILM+SLM)

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[gmarouli]

After a small investigation we saw that there a couple of things that could be done to fix this:

1. Before we check if the SLM policy has finished executing, we check if this policy applies this index, if not, we will treat as an miconfiguration error and inform the user that the policy does not apply to this index and there is no guarantee there is a "fresh" snapshot of this index. We will not delete until the policy updated to apply to this index or it's skipped (I need to play around with this to provide the exact steps).
2. If the policy is correct we want to make sure that there has been a recent successful run, either by checking that there is a snapshot that includes this index or by checking the cluster state versions. This requires a bit more thought but it should be possible.

Looking at the two ideas above, the first one is "quite" simple and it will solve this issue for the majority of our users. We will move forward with this one first and we can see the impact.