ILM will fail to completely stop if any indices are configured with an ILM policy that does not exist

[gwbrown]

Elasticsearch version (bin/elasticsearch --version): 6.7.1 and earlier

Description of the problem including expected versus actual behavior:
If any index uses a lifecycle policy which has a index.lifecycle.name set which refers to an ILM policy that does not exist, when ILM is stopped, the value returned by GET _ilm/status will remain as STOPPING and never transition to STOPPED.

It should rapidly transition to STOPPED as soon as no indices are in the middle of a Shrink action.

Notably, ILM activity does mostly stop: While the ILM status STOPPING, only Shrink actions will be run, so this is primarily an issue with the ILM Status API.

Steps to reproduce: Create an index with index.lifecycle.name set to an ILM policy that does not exist and call the ILM Stop API.

PUT testindex
{
  "settings": {
    "index.number_of_shards": 3,
    "index.lifecycle.name": "this_policy_doesnt_exist"
  }
}

POST _ilm/stop

GET _ilm/status

Provide logs (if relevant):
Log messages similar to:

policy [this_policy_doesnt_exist] for index [testindex] does not exist, recording this in step_info for this index

Will appear in the logs. This is expected when an index is set to use a policy that does not exist as well, but it may help to identify the issue.

Workaround: If the policies that don't exist are removed from all indices referencing them using the Remove Policy API, then the ILM status will rapidly transition to STOPPED as expected.

[elasticmachine]

Pinging @elastic/es-core-features