-
Notifications
You must be signed in to change notification settings - Fork 24.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Create system indices on startup if configured #61656
Comments
Pinging @elastic/es-core-infra (:Core/Infra/Core) |
What would be an example of a system index for which we don't know the configuration ahead of time? |
Kibana is an example since plugins can change some of their mappings. |
I spent a while looking for prior art in the codebase, and the closest thing seems to be |
There is also a |
It's a shame we don't have a single implementation for creating templates - I have some experimental code to declare which templates a system index relies upon, wait until they exist, and then create the system index. However that doesn't work for e.g. |
I think we should move away from templates for system indices that we are going to create; the values should be handled in code like async search does. I have been thinking about this some more today and one thought that I had relates to the auto creation of indices; we could defer creating until time of write. We would just hook into the creation of indices so that write requests will create the system index with all of the proper mappings/settings. This has a downside that get and search requests would still need to handle IndexNotFoundException. @gwbrown @williamrandolph thoughts? |
I'll add a bit more for why it might be best to avoid creating system indices that may go unused by users and that is the fact that each shard has overhead and our docs state 20 shards per GB of heap. If we think about small nodes and clusters, by pre-creating unused indices we are limiting already scarce resources for these users. |
That was indeed my concern too about pre-creating indices. So perhaps we can define some common patterns for dealing with possibly-not-existing-yet indices, and creating them on-demand. Then our plugins and 3rd-party plugin authors could just hook into the common framework. |
Just summarising some earlier email discussions: Since Kibana's indices have dynamic mappings these can't be created on startup. However, auto-created indices still sometimes occur due to user error which then creates a hard to recover from situation. Kibana would like for the Elasticsearch Kibana plugin to disable auto-create for all it's system indices elastic/kibana#81790 |
Part of elastic#61656. Add the necessary support for automatically creating and updating system indices. This works by making it possible to create a system index descriptor with all the information needed to manage the mappings, settings and aliases. Follow-up work will opt existing indices into this framework.
Part of #61656. Add the necessary support for automatically creating and updating system indices. This works by making it possible to create a system index descriptor with all the information needed to manage the mappings, settings and aliases. Follow-up work will opt existing indices into this framework.
Part of elastic#61656. Add the necessary support for automatically creating and updating system indices. This works by making it possible to create a system index descriptor with all the information needed to manage the mappings, settings and aliases. Follow-up work will opt existing indices into this framework.
Backport of #65604. Part of #61656. Add the necessary support for automatically creating and updating system indices. This works by making it possible to create a system index descriptor with all the information needed to manage the mappings, settings and aliases. Follow-up work will opt existing indices into this framework.
Part of elastic#61656. Auto-create the .logstash index using the system index infrastructure.
Backport of #65959. Part of #61656. Change the `.tasks` system index descriptor so that the index can be automatically managed by Elasticsearch e.g. created on-demand, mapping kept up-to-date, etc. Also add an integration test to exercise the `SystemIndexManager` end-to-end, and cherry-pick #66605 to add more system index tests.
Part of elastic#61656. Managing `.tasks` via the system index infrastructure is causing a BWC issue, so this commit reverts the index back to being created via a template until we can figure out the problem.
Part of elastic#61656. Partial revet of elastic#66640. Managing `.tasks` via the system index infrastructure is causing a BWC issue, so this commit reverts the index back to being created via a template until we can figure out the problem.
Part of #61656. Change the Security plugin so that its system indices are managed automatically by the system indices infrastructure. Also add an `origin` field to `CreateIndexRequest` and `UpdateSettingsRequest`.
Backport of elastic#67114. Part of elastic#61656. Change the Security plugin so that its system indices are managed automatically by the system indices infrastructure. Also add an `origin` field to `CreateIndexRequest` and `UpdateSettingsRequest`.
Part of #61656. Migrate the `.watches` and `.triggered_watches` system indices to use the auto-create infrastructure. The watcher history indices are left alone. As part of this work, a `SystemIndexDescriptor` now inspects its mappings to determine whether it has any dynamic mappings. This influences how strict Elasticsearch is with enforcing the descriptor's mappings, since ES cannot know in advanced what all the mappings will be. This PR also fixes the `SystemIndexManager` so that (1) it doesn't fall over when attempting to inspect the state of an index that hasn't been created yet, and (2) does update mappings if there's no version field in the mapping metadata.
Part of elastic#61656. Migrate the `.watches` and `.triggered_watches` system indices to use the auto-create infrastructure. The watcher history indices are left alone. As part of this work, a `SystemIndexDescriptor` now inspects its mappings to determine whether it has any dynamic mappings. This influences how strict Elasticsearch is with enforcing the descriptor's mappings, since ES cannot know in advanced what all the mappings will be. This PR also fixes the `SystemIndexManager` so that (1) it doesn't fall over when attempting to inspect the state of an index that hasn't been created yet, and (2) does update mappings if there's no version field in the mapping metadata.
Backport of #67588. Part of #61656. Migrate the `.watches` and `.triggered_watches` system indices to use the auto-create infrastructure. The watcher history indices are left alone. As part of this work, a `SystemIndexDescriptor` now inspects its mappings to determine whether it has any dynamic mappings. This influences how strict Elasticsearch is with enforcing the descriptor's mappings, since ES cannot know in advanced what all the mappings will be. This PR also fixes the `SystemIndexManager` so that (1) it doesn't fall over when attempting to inspect the state of an index that hasn't been created yet, and (2) does update mappings if there's no version field in the mapping metadata.
We have now migrated all system indices that Elasticsearch can manage itself to the system indices framework. As such, this issue can now be closed. |
System indices are necessary for Elasticsearch and the stack to function and should be available as soon as possible. Upon introduction of a new system index or startup of a cluster, a mechanism should be provided so that system indices can be automatically created if configured to do so.
Some system indices that belong to the stack, such as the kibana indices will not have their configuration stored in the system index plugin so this is an example of system indices that we would not want to be created automatically.
For system indices that we know the configuration for ahead of time, then we should enable the automated creation of these to prevent multiple components from needing to maintain their own methods of creating these indices and then later updating them when the time comes. Configuration in this context means the mappings, settings, and any aliases necessary for the system indices.
The text was updated successfully, but these errors were encountered: