Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add an ILM action to ensure a snapshot contains the index prior to being deleted #45067

Closed
dakrone opened this issue Jul 31, 2019 · 2 comments · Fixed by #50454
Closed

Add an ILM action to ensure a snapshot contains the index prior to being deleted #45067

dakrone opened this issue Jul 31, 2019 · 2 comments · Fixed by #50454
Assignees
Labels
:Data Management/ILM+SLM Index and Snapshot lifecycle management >feature

Comments

@dakrone
Copy link
Member

dakrone commented Jul 31, 2019

It's a common pattern with data to go through the lifecycle of an index, and then take a snapshot immediately prior to the index being deleted. It would be great if we could add functionality similar with this by combining parts of the SLM and ILM features in ES.

For example, if an index had a policy like:

{
  "policy": {
    "phases": {
      "delete": {
        "min_age": "15d",
        "actions": {
          "snapshot": {
            "policy": "my-slm-policy"
          },
          "delete": {}
        }
      }
    }
  }
}

ILM would wait until the index entered the delete phase, it would then wait until a snapshot from the my-slm-policy SLM policy had been taken (it would not manually kick off a snapshot, since that could potentially create scores of unwanted snapshots), and then it would delete the index.

/cc @yaronp68 as you asked me to open an issue for this.

@dakrone dakrone added >feature :Data Management/ILM+SLM Index and Snapshot lifecycle management labels Jul 31, 2019
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-core-features

@JohanPeeters
Copy link

I absolutely agree that there is a need to ensure that an index has been archived before it is deleted. I can see this could be achieved by adding a 'snapshot' action to the delete phase to induce a wait on an SLM policy, as suggested. However, currently, snapshot lifecycle policies are very backup-oriented and do not support archiving terribly well imo - archiving usually needs to be triggered by the age of an index. As far as I can tell, there is no elegant way of ensuring that an SLM policy selects only indices of a certain age for its snapshot. If such capability was added, this proposal would suit my purposes perfectly.

@probakowski probakowski self-assigned this Dec 17, 2019
probakowski added a commit to probakowski/elasticsearch that referenced this issue Dec 20, 2019
This change add new ILM action to wait for SLM policy execution to ensure that index has snapshot before deletion.

Closes elastic#45067.
probakowski added a commit that referenced this issue Jan 9, 2020
This change add new ILM action to wait for SLM policy execution to ensure that index has snapshot before deletion.

Closes #45067
probakowski added a commit to probakowski/elasticsearch that referenced this issue Jan 13, 2020
This change add new ILM action to wait for SLM policy execution to ensure that index has snapshot before deletion.

Closes elastic#45067
probakowski added a commit that referenced this issue Jan 14, 2020
* ILM action to wait for SLM policy execution (#50454)

This change add new ILM action to wait for SLM policy execution to ensure that index has snapshot before deletion.

Closes #45067

* Fix flaky TimeSeriesLifecycleActionsIT#testWaitForSnapshot test

This change adds some randomness and cleanup step to TimeSeriesLifecycleActionsIT#testWaitForSnapshot and testWaitForSnapshotSlmExecutedBefore tests in attempt to make them stable.

Reletes to #50781

* Formatting changes

* Longer timeout

* Fix Map.of in Java8

* Unused import removed
SivagurunathanV pushed a commit to SivagurunathanV/elasticsearch that referenced this issue Jan 23, 2020
This change add new ILM action to wait for SLM policy execution to ensure that index has snapshot before deletion.

Closes elastic#45067
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Data Management/ILM+SLM Index and Snapshot lifecycle management >feature
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants