Description
This issue serves as an overarching issue for what we call System Indices and
details our plan to implement this feature.
As Elasticsearch and the rest of the stack has evolved, there is an increasing
reliance on indices as an implementation detail of features. Certain features
are critical, like Security, and this implementation detail should not be
necessary for a user to know or think about. Unfortunately, this implementation
detail has been exposed to users in various ways.
In general, most of the indices that would be considered system indices have
names that start with a . and are referred to as dot indices. This is modeled
after dot files to suggest that the index is an implementation detail that
should not be interacted with. Searches over all indices, such as those done by
Kibana by default, include dot indices, while to a user all indices means all
indices they created. Snapshots include dot indices by default, but restoring
dot indices does not work since the intended index already exists and
should be wholly replaced. And upgrades of dot indices are managed with
complicated aliases and versioned names, where each dot index has its own
versioning and upgrade logic that must be run externally.
Dot indices will be superseded by system indices that are registered through
a module or plugin. A special plugin type will be added that defines the system
indices that the plugin provides along with the mappings and APIs necessary to
access these indices. The existing search API (and other data APIs) will not be
able to search a system index. APIs used for monitoring and troubleshooting will
continue to operate as they do today and include system indices in their output.
There are some dot indices that do not necessarily fit the mold of a normal
system index; instead they store data that the system produces with the intent
that users can also query against this data. For these indices we will move to
adding a hidden index that will not be resolved by default for wildcards.
These indices can be specifically requested or an IndicesOption can be specified
so that these indices are not ignored during wildcard resolution. An index may
only be marked as hidden at creation time and will be done so through the use
of an index setting.
Hidden indices will not may have names that are prefixed by a dot. Hidden indices will not
inherit from a global index template.
In order to facilitate the migration from accessing dot indices directly the following
phased approach will be taken.
The initial work is the introduction of the infrastructure to define hidden indices and
system index plugins with dedicated APIs.
Tasks
- Deprecate the creation with dot indices in 7.x unless the call is made from a system index plugin or the index is hidden. Deprecate creation of dot-prefixed index names except for hidden and system indices #49959
- Introduce hidden indices. Implement hidden indices #50452
- Introduce hidden aliases. Implement hidden aliases #52547
- Create .kibana system index plugins with APIs necessary for migration Introduce system index APIs for Kibana #52385
- Handle hidden indices in the Index Management UI (Update Index Management to support hidden indices kibana#56532)
- Logstash plugin for pipelines Add logstash system index APIs #53350
The above steps will get us to a point where other teams can begin migrating from direct access to their system index and make use of the new APIs. The majority of the internal (to Elasticsearch) work will still need to be implemented.
- System index reads in their own threadpool
- System index writes in their own threadpool
- Prioritized recovery of system indices
- Security changes for logstash system index APIs
- System index interaction deprecation
- System indices should always be allowed to be created even with
action.auto_create_indexset. - Verify whether we need more than require_alias for hidden indices with a write alias
- Setting in component templates that allows overriding
action.auto_create_index. This will enable hidden indices for things like watcher and monitoring to always be created. - API for deletion of System Indices
- Allow system indices to opt-in to being created upon startup with specified mappings, settings, aliases
- Snapshot and restore of system indices
- Allow updating mappings/settings on existing system indices
- Disallow direct access of system indices
- Enable automated upgrade of indices
- Add allocation knowledge for cases like hot/warm/cold and availability zones
- Smarter replication for system indices (continue to use auto expand 0-1 or maybe a new scheme?)
- Flood stage watermark bypass?
System Index Patterns
IngestGeoIpPlugin
.geoip_databases
KibanaPlugin
.kibana_*(alias:.kibana).reporting-*.apm-agent-configuration.apm-custom-link
AsyncResultsIndexPlugin
.async-search
EnrichPlugin
.enrich-*
Fleet
.fleet-actions~(-results*)(alias:.fleet-actions).fleet-agents*(alias:.fleet-agents).fleet-enrollment-api-keys*(alias:.fleet-enrollment-api-keys).fleet-policies-[0-9]+*(alias:.fleet-policies).fleet-policies-leader*(alias:.fleet-policies-leader).fleet-servers*(alias:.fleet-servers).fleet-artifacts*(alias:.fleet-artifacts).fleet-actions-results
Logstash
.logstash
MachineLearning
.ml-meta*.ml-config*.ml-inference-*
[associated indices].ml-anomalies-*.ml-state*.ml-stats-*.ml-notifications*.ml-annotations*
SearchableSnapshots
.snapshot-blob-cache
Security
.security-[0-9]+(alias:.security).security-tokens-[0-9]+(alias:.security-tokens)
Transforms
.transform-internal-*
[associated indices].transform-notifications-*
Watcher
.watches*.triggered_watches*
APIs allowed to access system indices
The APIs that we currently plan to allow system index access for are:
- GET _cluster/health
- GET {index}/_recovery
- GET _cluster/allocation/explain
- GET _cluster/state
- POST _cluster/reroute
- GET {index}/_stats
- GET {index}/_segments
- GET {index}/_shard_stores
- GET _cat/[aliases,indices,health,recovery,shards,segments]