Use ILM to manage internal indices

[gwbrown]

Now that we have a system for curating indices built-in to Elasticsearch in Index Lifecycle Management, we should use this for internal indices which either already roll over through another mechanism or would benefit from automatically rolling over.

Some of this work has already been completed, see #37443 for an example.

Internal indices which we may want to manage this way include:

• Watcher
  • .watch-history-*
• Monitoring
  • .monitoring-alerts-*
  • .monitoring-beats-*
  • .monitoring-es-*
  • .monitoring-kibana-*
  • .monitoring-logstash-*
• ML (maybe?)
  • .ml-anomalies-*
  • .ml-state-*

The above list is preliminary and may be missing indices or contain indices we may choose to not manage with ILM.

[elasticmachine]

Pinging @elastic/es-core-features

[elasticmachine]

Pinging @elastic/ml-core

[ppf2]

.reporting-* from Kibana reporting is another potential candidate.

[benwtrent]

@gwbrown we have talked about doing this for .ml-anomalies* and .ml-state* indices.

Related issues:

• .ml-anomalies* [ML] Add an ML results index rollover endpoint #29946
• .ml-state* [ML] Support rollover of state index #29938

[hendrikmuhs]

related: #38805

Because ILM is a module that can be turned off, any dependency introduced must also add checks to prevent combinations like "ILM:off" + "Dependent Module:on"

[Leaf-Lin]

Adding apm-* indices as another option?

[Aqualie]

Is this still being worked on? I'm currently working through cleaning up my own indices as I have ILM setup for all my indexes and cut down my total count to just 41. Looking all the numerous system indices created it's currently at 81... 2x the amount of my own indices... It'd be great if we could use the same ILM logic and perhaps just set a total shard size of 50GB rather than these daily indices...

.monitoring-logstash-7-2020.04.30
.apm-agent-configuration
.kibana_1
.kibana_2
.kibana_3
.kibana_4
.kibana_5
.kibana_task_manager_1
.kibana_task_manager_2
.logstash
.management-beats
.ml-annotations-6
.ml-anomalies-custom-linux_anomalous_network_activity_ecs
.ml-anomalies-custom-linux_anomalous_network_port_activity_ecs
.ml-anomalies-custom-linux_anomalous_network_service
.ml-anomalies-custom-linux_anomalous_network_url_activity_ecs
.ml-anomalies-custom-linux_anomalous_process_all_hosts_ecs
.ml-anomalies-custom-linux_anomalous_user_name_ecs
.ml-anomalies-custom-packetbeat_dns_tunneling
.ml-anomalies-custom-packetbeat_rare_dns_question
.ml-anomalies-custom-packetbeat_rare_server_domain
.ml-anomalies-custom-packetbeat_rare_urls
.ml-anomalies-custom-packetbeat_rare_user_agent
.ml-anomalies-custom-rare_process_by_host_linux_ecs
.ml-anomalies-custom-rare_process_by_host_windows_ecs
.ml-anomalies-custom-suspicious_login_activity_ecs
.ml-anomalies-custom-windows_anomalous_network_activity_ecs
.ml-anomalies-custom-windows_anomalous_path_activity_ecs
.ml-anomalies-custom-windows_anomalous_process_all_hosts_ecs
.ml-anomalies-custom-windows_anomalous_process_creation
.ml-anomalies-custom-windows_anomalous_script
.ml-anomalies-custom-windows_anomalous_service
.ml-anomalies-custom-windows_anomalous_user_name_ecs
.ml-anomalies-custom-windows_rare_user_runas_event
.ml-anomalies-custom-windows_rare_user_type10_remote_login
.ml-anomalies-shared
.ml-config
.ml-meta
.ml-notifications
.ml-notifications-000001
.ml-state
.monitoring-alerts-7
.monitoring-beats-7-2020.04.28
.monitoring-beats-7-2020.04.29
.monitoring-beats-7-2020.04.30
.monitoring-beats-7-2020.05.01
.monitoring-beats-7-2020.05.02
.monitoring-beats-7-2020.05.03
.monitoring-beats-7-2020.05.04
.monitoring-beats-7-2020.05.05
.monitoring-es-7-2020.04.29
.monitoring-es-7-2020.04.30
.monitoring-es-7-2020.05.01
.monitoring-es-7-2020.05.02
.monitoring-es-7-2020.05.03
.monitoring-es-7-2020.05.04
.monitoring-es-7-2020.05.05
.monitoring-kibana-7-2020.04.29
.monitoring-kibana-7-2020.04.30
.monitoring-kibana-7-2020.05.01
.monitoring-kibana-7-2020.05.02
.monitoring-kibana-7-2020.05.03
.monitoring-kibana-7-2020.05.04
.monitoring-kibana-7-2020.05.05
.monitoring-logstash-7-2020.04.29
.monitoring-logstash-7-2020.05.01
.monitoring-logstash-7-2020.05.02
.monitoring-logstash-7-2020.05.03
.monitoring-logstash-7-2020.05.04
.monitoring-logstash-7-2020.05.05
.security-7
.tasks
.triggered_watches
.watcher-history-10-2020.04.29
.watcher-history-10-2020.04.30
.watcher-history-10-2020.05.01
.watcher-history-10-2020.05.02
.watcher-history-10-2020.05.03
.watcher-history-10-2020.05.04
.watcher-history-10-2020.05.05
.watches

[dakrone]

@Aqualie yes this is still being worked on, currently we're working on the ILM bootstrap issue that requires setting up an alias, that's part of what #53100 and #53101 are part of, once those are released it will be easier to transition these indices to use ILM.

[skmizuho]

Following...
Until this is done, I was wondering what is the best practice on autodeleting the .monitoring indices after, say 30 days?

[jakelandis]

Until this is done, I was wondering what is the best practice on autodeleting the .monitoring indices after, say 30 days?

The recommended approach is to continue to use the cleaner service : https://www.elastic.co/guide/en/elasticsearch/reference/current/local-exporter.html#local-exporter-cleaner. Note - this requires enabling the local exporter (if you don't have it enabled already, it is the default) and if you don't want to actually self monitor you can disable the collection of monitoring data (per the reference guide). This is admittedly a bit award but we are working towards removing the cleaner service in favor of ILM.

Also note - it is possible to use ILM today to delete these indices by modifying the .monitoring index template, but we do not recommend that since the system can land new versions of the template and overwrite your changes.

[hellmanj]

@jakelandis Your recommended solution doesn't work in ESS

Your changes cannot be applied
Elasticsearch - 'xpack.monitoring.collection.enabled': is not allowed

What should I do? BTW #53100 and #53101 are closed, can we now expect this to be fixed ASAP?

[rseldner]

@jakelandis Your recommended solution doesn't work in ESS

Your changes cannot be applied
Elasticsearch - 'xpack.monitoring.collection.enabled': is not allowed

You can set it dynamically on ESS.

PUT /_cluster/settings
{
 "persistent" : {
   "xpack.monitoring.history.duration" : "30d"
 }
}

[jakelandis]

Stack monitoring, when using MetricBeat or Fleet on 8.0+ will use data streams and ILM. Legacy monitoring and Metricbeat 6/7.x will continue to use daily indices whose retention can be configured with xpack.monitoring.history.duration. related #82498

I am closing this issue as to my knowledge all 8.x internal time series indices are configured to use ILM.