-
Notifications
You must be signed in to change notification settings - Fork 8.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[DOCS] Add ILM tutorial #59502
[DOCS] Add ILM tutorial #59502
Conversation
Pinging @elastic/kibana-docs (Team:Docs) |
This is still a work in progress, I know I need to check on at least the following things:
|
=== Tutorial: Use ILM to manage Filebeat time-based indices | ||
|
||
With index lifecycle management (ILM), you can create policies to define actions | ||
that will be performed automatically against indices as they age. In this tutorial, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
that are automatically performed
node.attr.data: "warm" | ||
-------------------------------------------------------------------------------- | ||
|
||
* A server with Filebeat installed and configured to send logs to the `elasticsearch` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Add ending period
[float] | ||
==== View the filebeat ILM policy | ||
|
||
Filebeat includes a default ILM policy which enables rollover. Index lifecycle |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
which > that
The default policy is enough to prevent the creation of many tiny daily indices. | ||
You can modify the policy to meet more complex requirements. | ||
|
||
* Activate the warm phase. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Change bullet points to numbered steps.
- Activate the warm phase.
- Set either of the following:
bullet list - In Select a node attribute ...
* Activate the warm phase. | ||
|
||
+ | ||
You have a choice to make, you can’t meet the requirements exactly with rollover |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Make this step 2 instead of a substep:
Set either of the following:
more quickly than intended if it reaches the *Maximum index size* condition before | ||
*Maximum age*. | ||
|
||
** Provide a value for *Timing for warm phase*. Setting this to *15* will keep the |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd put this first because it is the default.
indices on hot nodes for a range of 15-45 days, depending on when the initial | ||
rollover occurred. | ||
|
||
* *Select a node attribute to control shard allocation*. For this example, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In Select a node attribute to control shard allocation, set data:warm(2) to migrate shards to warm data nodes.
[role="screenshot"] | ||
image::images/tutorial-ilm-modify-default-warm-phase-rollover.png["Modify to add warm phase"] | ||
|
||
* Activate the delete phase. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Make this one step.
Activate the delete phase and set the Timing to 60 days.
To create a custom policy in {kib}, go to *Management > Index Lifecycle Policies > Create Policy*. | ||
|
||
|
||
* *Warm phase* |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Combine.
- Activate the warm phase and configure it as follows.
* After 60 days, move the index into the cold phase and onto less expensive hardware. | ||
* *Delete phase* | ||
+ | ||
Activate the delete phase and set the timing. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
- Activate the delete phase and set the timing.
@gchaps @DanRoscigno I made the changes recommended above and pushed a new commit. I'm still waiting to see how my custom policy behaves when it rolls over in 3 days. |
@gchaps @DanRoscigno I verified that ILM behaves the way I expected with this new policy. I think this is ready for another review! |
to label data nodes as hot or warm. This step is required to migrate shards between | ||
nodes using shard allocation awareness. | ||
+ | ||
For example, you can set this in `elasticsearch.yml` for each data node: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For example, you can set this in `elasticsearch.yml` for each data node: | |
For example, you can set this in your `elasticsearch.yml` for each data node: |
can create a custom policy. For this option, you will use Filebeat daily indices | ||
without rollover. | ||
|
||
To create a custom policy in {kib}, go to *Management > Index Lifecycle Policies > |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should this be step 1?
|
||
* Move the index into the warm phase, shrink the index down to a single shard, | ||
and force merge to a single segment. | ||
. Configure the index to use the new policy in *{kib} > Management > Index Lifecycle |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is hard to parse. Can you make it a series of substeps?
3. Configure the index to use the new policy.
a. In Index Lifecycle Policies, select your policy.
b....
c....
[[example-using-index-lifecycle-policy]] | ||
=== Example of using an index lifecycle policy | ||
=== Tutorial: Use ILM to manage Filebeat time-based indices |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We maintain a list of attributes here: https://github.com/elastic/docs/blob/master/shared/attributes.asciidoc
Try to use those when possible. They ensure we can make docs-wide term changes easily.
=== Tutorial: Use ILM to manage Filebeat time-based indices | |
=== Tutorial: Use {ilm-init} to manage {filebeat} time-based indices |
=== Example of using an index lifecycle policy | ||
=== Tutorial: Use ILM to manage Filebeat time-based indices | ||
|
||
With index lifecycle management (ILM), you can create policies to define actions |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
With index lifecycle management (ILM), you can create policies to define actions | |
With {ilm} ({ilm-init}), you can create policies to define actions |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm replacing an existing page, so I was leaving the old anchor in case it was referenced elsewhere. Do you still suggest to change it?
With index lifecycle management (ILM), you can create policies to define actions | ||
that are automatically performed against indices as they age. In this tutorial, | ||
you will use {kib}’s *Index Lifecycle Policies* to modify and create ILM policies. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
With index lifecycle management (ILM), you can create policies to define actions that are automatically performed against indices as they age.
This implies that ILM only works based on age, but we can also take actions based on index size or number of docs. More importantly, this doesn't really address the why of ILM. It helps you automatically manage your indices for performance and retention. You may want to consider reusing a sentence from https://www.elastic.co/guide/en/elasticsearch/reference/master/overview-index-lifecycle-management.html.
In this tutorial, you will use Kibana’s Index Lifecycle Policies to modify and create ILM policies.
Nit: I'd reword to avoid future tense if possible. Something like:
This tutorial shows you how to use use {kib}'s *Index Lifecycle Policies* to
create and change {ilm-init} policies.
The bolded Index Lifecycle Policies seemed unnecessary to me, but maybe that's just a Kibana docs style.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yep, it's a Kibana docs style.
[float] | ||
==== Scenario |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should use [discrete]
instead of [float]
when possible. I'd also add an explicit anchor here.
[float] | |
==== Scenario | |
[discrete] | |
[[example-using-index-lifecycle-policy-scenario]] | |
==== Scenario |
You’ve been tasked with sending syslog files to an {es} cluster. This | ||
project has the following data retention guidelines for log data: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You’ve been tasked with sending syslog files to an {es} cluster. This | |
project has the following data retention guidelines for log data: | |
You’re tasked with sending syslog files to an {es} cluster. This | |
log data has the following data retention guidelines: |
You’ve been tasked with sending syslog files to an {es} cluster. This | ||
project has the following data retention guidelines for log data: | ||
|
||
* Keep logs on hot data nodes for 30 days |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is the first mention of hot-warm architecture.
If readers should be familiar with those concepts, maybe add a link to a conceptual doc in the introduction?
project has the following data retention guidelines for log data: | ||
|
||
* Keep logs on hot data nodes for 30 days | ||
* Rollover to a new index if the size reaches 50GB |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
"Roll over" should be two words when used as a verb. One word as a noun.
* Rollover to a new index if the size reaches 50GB | |
* Roll over any index containing log data to a new index when it reaches 50GB |
** Set replicas to 1 | ||
** Force merge indices to a single segment |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
** Set replicas to 1
** Force merge indices to a single segment
I think these two items need more context/explanation. I imagine many users may not be familiar with the concept of replicas, force merges, or segments. At the least, provide a link.
* Add tutorial for ILM with filebeat * Change screenshots and add additional steps * Update screenshots, add numbered steps, and other minor edits * Incorporate feedback: update links, formatting, and minor edits * Move tip inline with list * Apply suggestions from code review Co-Authored-By: James Rodewig <james.rodewig@elastic.co> * Move TIP inline . . . again * Put TIP inline Co-authored-by: James Rodewig <james.rodewig@elastic.co>
* Add tutorial for ILM with filebeat * Change screenshots and add additional steps * Update screenshots, add numbered steps, and other minor edits * Incorporate feedback: update links, formatting, and minor edits * Move tip inline with list * Apply suggestions from code review Co-Authored-By: James Rodewig <james.rodewig@elastic.co> * Move TIP inline . . . again * Put TIP inline Co-authored-by: James Rodewig <james.rodewig@elastic.co>
* Add tutorial for ILM with filebeat * Change screenshots and add additional steps * Update screenshots, add numbered steps, and other minor edits * Incorporate feedback: update links, formatting, and minor edits * Move tip inline with list * Apply suggestions from code review Co-Authored-By: James Rodewig <james.rodewig@elastic.co> * Move TIP inline . . . again * Put TIP inline Co-authored-by: James Rodewig <james.rodewig@elastic.co>
* Add tutorial for ILM with filebeat * Change screenshots and add additional steps * Update screenshots, add numbered steps, and other minor edits * Incorporate feedback: update links, formatting, and minor edits * Move tip inline with list * Apply suggestions from code review Co-Authored-By: James Rodewig <james.rodewig@elastic.co> * Move TIP inline . . . again * Put TIP inline Co-authored-by: James Rodewig <james.rodewig@elastic.co>
* Add tutorial for ILM with filebeat * Change screenshots and add additional steps * Update screenshots, add numbered steps, and other minor edits * Incorporate feedback: update links, formatting, and minor edits * Move tip inline with list * Apply suggestions from code review Co-Authored-By: James Rodewig <james.rodewig@elastic.co> * Move TIP inline . . . again * Put TIP inline Co-authored-by: James Rodewig <james.rodewig@elastic.co>
* Add tutorial for ILM with filebeat * Change screenshots and add additional steps * Update screenshots, add numbered steps, and other minor edits * Incorporate feedback: update links, formatting, and minor edits * Move tip inline with list * Apply suggestions from code review Co-Authored-By: James Rodewig <james.rodewig@elastic.co> * Move TIP inline . . . again * Put TIP inline Co-authored-by: James Rodewig <james.rodewig@elastic.co>
* Add tutorial for ILM with filebeat * Change screenshots and add additional steps * Update screenshots, add numbered steps, and other minor edits * Incorporate feedback: update links, formatting, and minor edits * Move tip inline with list * Apply suggestions from code review Co-Authored-By: James Rodewig <james.rodewig@elastic.co> * Move TIP inline . . . again * Put TIP inline Co-authored-by: James Rodewig <james.rodewig@elastic.co>
* Add tutorial for ILM with filebeat * Change screenshots and add additional steps * Update screenshots, add numbered steps, and other minor edits * Incorporate feedback: update links, formatting, and minor edits * Move tip inline with list * Apply suggestions from code review Co-Authored-By: James Rodewig <james.rodewig@elastic.co> * Move TIP inline . . . again * Put TIP inline Co-authored-by: James Rodewig <james.rodewig@elastic.co>
* Add tutorial for ILM with filebeat * Change screenshots and add additional steps * Update screenshots, add numbered steps, and other minor edits * Incorporate feedback: update links, formatting, and minor edits * Move tip inline with list * Apply suggestions from code review Co-Authored-By: James Rodewig <james.rodewig@elastic.co> * Move TIP inline . . . again * Put TIP inline Co-authored-by: James Rodewig <james.rodewig@elastic.co> Co-authored-by: James Rodewig <james.rodewig@elastic.co>
* Add tutorial for ILM with filebeat * Change screenshots and add additional steps * Update screenshots, add numbered steps, and other minor edits * Incorporate feedback: update links, formatting, and minor edits * Move tip inline with list * Apply suggestions from code review Co-Authored-By: James Rodewig <james.rodewig@elastic.co> * Move TIP inline . . . again * Put TIP inline Co-authored-by: James Rodewig <james.rodewig@elastic.co> Co-authored-by: James Rodewig <james.rodewig@elastic.co>
* Add tutorial for ILM with filebeat * Change screenshots and add additional steps * Update screenshots, add numbered steps, and other minor edits * Incorporate feedback: update links, formatting, and minor edits * Move tip inline with list * Apply suggestions from code review Co-Authored-By: James Rodewig <james.rodewig@elastic.co> * Move TIP inline . . . again * Put TIP inline Co-authored-by: James Rodewig <james.rodewig@elastic.co> Co-authored-by: James Rodewig <james.rodewig@elastic.co>
* master: (132 commits) document code splitting for client code (elastic#62593) Escape single quotes surrounded by double quotes (elastic#63229) [Endpoint] Update cli mapping to match endpoint package (elastic#63372) update in-app links to metricbeat configuration docs (elastic#63295) investigation notes field (documentation / metadata) (elastic#63386) [Maps] fix bug where toggling Scaling type does not re-fetch data (elastic#63326) [Alerting] set correct parameter for unauthented email action (elastic#63086) [Telemetry] force staging urls in tests (elastic#63356) Migrate legacy maps service to NP & update refs (elastic#60942) Fix task manager query to return tasks to retry (elastic#63360) [Endpoint] Policy list support for URL pagination state (elastic#63291) [Canvas] Migrate saved object mappings and migrations to Kibana Platform (elastic#58891) [DOCS] Add ILM tutorial (elastic#59502) [Maps] Add SOURCE_TYPES enumeration (elastic#62975) [Maps] update geospatial filters to use geo_shape query for geo_point fields (elastic#62966) Move away from npStart for embeddables in canvas (elastic#62680) Use MapInput type from Maps plugin (elastic#61539) Update to pagination for workpad and templates (elastic#62050) [SIEM] Fix AlertsTable id (elastic#63368) Consistent terminology around cypress test data (elastic#63279) ...
* master: document code splitting for client code (elastic#62593) Escape single quotes surrounded by double quotes (elastic#63229) [Endpoint] Update cli mapping to match endpoint package (elastic#63372) update in-app links to metricbeat configuration docs (elastic#63295) investigation notes field (documentation / metadata) (elastic#63386) [Maps] fix bug where toggling Scaling type does not re-fetch data (elastic#63326) [Alerting] set correct parameter for unauthented email action (elastic#63086) [Telemetry] force staging urls in tests (elastic#63356) Migrate legacy maps service to NP & update refs (elastic#60942) Fix task manager query to return tasks to retry (elastic#63360) [Endpoint] Policy list support for URL pagination state (elastic#63291) [Canvas] Migrate saved object mappings and migrations to Kibana Platform (elastic#58891) [DOCS] Add ILM tutorial (elastic#59502) [Maps] Add SOURCE_TYPES enumeration (elastic#62975) [Maps] update geospatial filters to use geo_shape query for geo_point fields (elastic#62966) Move away from npStart for embeddables in canvas (elastic#62680)
* [DOCS] Add ILM tutorial (#59502) * Add tutorial for ILM with filebeat * Change screenshots and add additional steps * Update screenshots, add numbered steps, and other minor edits * Incorporate feedback: update links, formatting, and minor edits * Move tip inline with list * Apply suggestions from code review Co-Authored-By: James Rodewig <james.rodewig@elastic.co> * Move TIP inline . . . again * Put TIP inline Co-authored-by: James Rodewig <james.rodewig@elastic.co> * Update ILM overview link for 7.0 Co-authored-by: James Rodewig <james.rodewig@elastic.co>
* [DOCS] Add ILM tutorial (#59502) * Add tutorial for ILM with filebeat * Change screenshots and add additional steps * Update screenshots, add numbered steps, and other minor edits * Incorporate feedback: update links, formatting, and minor edits * Move tip inline with list * Apply suggestions from code review Co-Authored-By: James Rodewig <james.rodewig@elastic.co> * Move TIP inline . . . again * Put TIP inline Co-authored-by: James Rodewig <james.rodewig@elastic.co> * Update ILM overview link for 7.1 Co-authored-by: James Rodewig <james.rodewig@elastic.co>
* [DOCS] Add ILM tutorial (#59502) * Add tutorial for ILM with filebeat * Change screenshots and add additional steps * Update screenshots, add numbered steps, and other minor edits * Incorporate feedback: update links, formatting, and minor edits * Move tip inline with list * Apply suggestions from code review Co-Authored-By: James Rodewig <james.rodewig@elastic.co> * Move TIP inline . . . again * Put TIP inline Co-authored-by: James Rodewig <james.rodewig@elastic.co> * Update ILM overview link for 7.2 Co-authored-by: James Rodewig <james.rodewig@elastic.co>
* [DOCS] Add ILM tutorial (#59502) * Add tutorial for ILM with filebeat * Change screenshots and add additional steps * Update screenshots, add numbered steps, and other minor edits * Incorporate feedback: update links, formatting, and minor edits * Move tip inline with list * Apply suggestions from code review Co-Authored-By: James Rodewig <james.rodewig@elastic.co> * Move TIP inline . . . again * Put TIP inline Co-authored-by: James Rodewig <james.rodewig@elastic.co> * Update ILM overview link for 7.3 Co-authored-by: James Rodewig <james.rodewig@elastic.co>
* [DOCS] Add ILM tutorial (#59502) * Add tutorial for ILM with filebeat * Change screenshots and add additional steps * Update screenshots, add numbered steps, and other minor edits * Incorporate feedback: update links, formatting, and minor edits * Move tip inline with list * Apply suggestions from code review Co-Authored-By: James Rodewig <james.rodewig@elastic.co> * Move TIP inline . . . again * Put TIP inline Co-authored-by: James Rodewig <james.rodewig@elastic.co> * Update ILM link for 7.5 Co-authored-by: James Rodewig <james.rodewig@elastic.co>
* [DOCS] Add ILM tutorial (#59502) * Add tutorial for ILM with filebeat * Change screenshots and add additional steps * Update screenshots, add numbered steps, and other minor edits * Incorporate feedback: update links, formatting, and minor edits * Move tip inline with list * Apply suggestions from code review Co-Authored-By: James Rodewig <james.rodewig@elastic.co> * Move TIP inline . . . again * Put TIP inline Co-authored-by: James Rodewig <james.rodewig@elastic.co> * Update ILM overview link for 7.5 Co-authored-by: James Rodewig <james.rodewig@elastic.co> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
* Add tutorial for ILM with filebeat * Change screenshots and add additional steps * Update screenshots, add numbered steps, and other minor edits * Incorporate feedback: update links, formatting, and minor edits * Move tip inline with list * Apply suggestions from code review Co-Authored-By: James Rodewig <james.rodewig@elastic.co> * Move TIP inline . . . again * Put TIP inline Co-authored-by: James Rodewig <james.rodewig@elastic.co>
Summary
Added a walkthrough using ILM with filebeat indices
Closes #29363
[skip-ci]
Preview the tutorial