Deprecate creation of dot-prefixed index names except for hidden and system indices #49959
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Pinging @elastic/es-core-infra (:Core/Infra/Core) |
This commit deprecates the creation of dot-prefixed index names (e.g. `.watches`) unless they are registered by a plugin that extends `SystemIndexPlugin`. This is the first step towards more thorough proections for system indices. This commit also modifies several plugins which use dot-prefixed indices to register indices they own as system indices.
gwbrown
force-pushed
the
sys-ind/basic-plugin
branch
from
444310c to
c731646
Compare
|
I've confirmed that there's a need for supporting patterns of system indices (see the Enrich processor), rather than just having them be static names, so this is going to be updated to support that very soon. |
jaymode
reviewed
Dec 17, 2019
server/src/main/java/org/elasticsearch/cluster/metadata/MetaDataCreateIndexService.java
Outdated
Show resolved
Hide resolved
server/src/main/java/org/elasticsearch/snapshots/RestoreService.java
Outdated
Show resolved
Hide resolved
...r/src/main/java/org/elasticsearch/action/admin/indices/rollover/TransportRolloverAction.java
Outdated
Show resolved
Hide resolved
server/src/main/java/org/elasticsearch/indices/SystemIndexDescriptor.java
Outdated
Show resolved
Hide resolved
gwbrown
changed the title
gwbrown
added a commit
to gwbrown/elasticsearch
that referenced
this pull request
Jan 28, 2020
…system indices (elastic#49959) This commit deprecates the creation of dot-prefixed index names (e.g. .watches) unless they are either 1) a hidden index, or 2) registered by a plugin that extends SystemIndexPlugin. This is the first step towards more thorough protections for system indices. This commit also modifies several plugins which use dot-prefixed indices to register indices they own as system indices, and adds a plugin to register .tasks as a system index.
gwbrown
added a commit
that referenced
this pull request
Jan 28, 2020
…system indices (#49959) This commit deprecates the creation of dot-prefixed index names (e.g. .watches) unless they are either 1) a hidden index, or 2) registered by a plugin that extends SystemIndexPlugin. This is the first step towards more thorough protections for system indices. This commit also modifies several plugins which use dot-prefixed indices to register indices they own as system indices, and adds a plugin to register .tasks as a system index.
38 tasks
kaituo
added a commit
to kaituo/anomaly-detection
that referenced
this pull request
May 18, 2020
This PR made the following changes: 1. Update to JDK 14 2. Hack to overstep potential ES 7.7 bug when installing plugins for test cluster 3. Upgrade gradle, nebula and jacoco versions 4. Change how we register rest endpoint handler as required by ES 7.7 5. Impelement SystemIndexPlugin and register the descriptor according to elastic/elasticsearch#49959 6. added new version of job scheduler zip Testing done: 1. gralde build passes 2. Manually tested all public APIs
kaituo
added a commit
to opendistro-for-elasticsearch/anomaly-detection
that referenced
this pull request
May 19, 2020
This PR made the following changes: 1. Update to JDK 14 2. Hack to overstep potential ES 7.7 bug when installing plugins for test cluster 3. Upgrade gradle, nebula and jacoco versions 4. Change how we register rest endpoint handler as required by ES 7.7 5. Impelement SystemIndexPlugin and register the descriptor according to elastic/elasticsearch#49959 6. added new version of job scheduler zip Testing done: 1. gralde build passes 2. Manually tested all public APIs
kaituo
pushed a commit
to kaituo/anomaly-detection
that referenced
this pull request
Mar 6, 2021
Previously, we registered AD indices as system indices as AD indices' names start with a dot. ES deprecates the creation of dot-prefixed index names except for hidden and system indices (elastic/elasticsearch#49959). Starting 7.10, ES adds a dedicated thread pool for system index write operations. System index writes/reads have higher priority than user index writes/reads. For example, system index writes can be forced regardless of whether the current index pressure is high or not (https://github.com/elastic/elasticsearch/blob/242083a36e02496aae9214dc41b89372022e7076/server/src/main/java/org/elasticsearch/index/IndexingPressure.java#L62-L73). AD indices are not more important than other user indices. We don't want AD index reads/writes to impact user indices' reads/writes. This PR removes AD indices out of the system index list and marks them hidden indices instead. This change does not impact created AD indices. They are still system indices. Newly created AD indices will be hidden instead of system indices. This change won't impact search/index API. To list hidden indices, one can use localhost:9200/_cat/indices?v&expand_wildcards=all Testing done: 1. Ran backward-compatibility tests. After updating AD, old detectors run fine, and we can create/run new detectors. All public APIs still work.
kaituo
pushed a commit
to kaituo/anomaly-detection
that referenced
this pull request
Mar 6, 2021
Previously, we registered AD indices as system indices as AD indices' names start with a dot. ES deprecates the creation of dot-prefixed index names except for hidden and system indices (elastic/elasticsearch#49959). Starting 7.10, ES adds a dedicated thread pool for system index write operations. System index writes/reads have higher priority than user index writes/reads. For example, system index writes can be forced regardless of whether the current index pressure is high or not (https://github.com/elastic/elasticsearch/blob/242083a36e02496aae9214dc41b89372022e7076/server/src/main/java/org/elasticsearch/index/IndexingPressure.java#L62-L73). AD indices are not more important than other user indices. We don't want AD index reads/writes to impact user indices' reads/writes. This PR removes AD indices out of the system index list and marks them hidden indices instead. This change does not impact created AD indices. They are still system indices. Newly created AD indices will be hidden instead of system indices. This change won't impact search/index API. To list hidden indices, one can use localhost:9200/_cat/indices?v&expand_wildcards=all Testing done: 1. Ran backward-compatibility tests. After updating AD, old detectors run fine, and we can create/run new detectors. All public APIs still work.
kaituo
added a commit
to opendistro-for-elasticsearch/anomaly-detection
that referenced
this pull request
Mar 8, 2021
Previously, we registered AD indices as system indices as AD indices' names start with a dot. ES deprecates the creation of dot-prefixed index names except for hidden and system indices (elastic/elasticsearch#49959). Starting 7.10, ES adds a dedicated thread pool for system index write operations. System index writes/reads have higher priority than user index writes/reads. For example, system index writes can be forced regardless of whether the current index pressure is high or not (https://github.com/elastic/elasticsearch/blob/242083a36e02496aae9214dc41b89372022e7076/server/src/main/java/org/elasticsearch/index/IndexingPressure.java#L62-L73). AD indices are not more important than other user indices. We don't want AD index reads/writes to impact user indices' reads/writes. This PR removes AD indices out of the system index list and marks them hidden indices instead. This change does not impact created AD indices. They are no longer system indices. Using indices metadata section in localhost:9200/_cluster/state?pretty output can confirm that (look for "system" key under each index ). Newly created AD indices will be hidden instead of system indices. This change won't impact search/index API. To list hidden indices, one can use localhost:9200/_cat/indices?v&expand_wildcards=all Testing done: * Ran backward-compatibility tests. After updating AD, old detectors run fine, and we can create/run new detectors. All public APIs still work.
hamersu9t
added a commit
to hamersu9t/anomaly-detection
that referenced
this pull request
Aug 10, 2024
This PR made the following changes: 1. Update to JDK 14 2. Hack to overstep potential ES 7.7 bug when installing plugins for test cluster 3. Upgrade gradle, nebula and jacoco versions 4. Change how we register rest endpoint handler as required by ES 7.7 5. Impelement SystemIndexPlugin and register the descriptor according to elastic/elasticsearch#49959 6. added new version of job scheduler zip Testing done: 1. gralde build passes 2. Manually tested all public APIs
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This commit deprecates the creation of dot-prefixed index names (e.g.
.watches) unless they are either 1) a hidden index, or 2) registered bya plugin that extends
SystemIndexPlugin. This is the first steptowards more thorough protections for system indices.
This commit also modifies several plugins which use dot-prefixed indices
to register indices they own as system indices, and adds a plugin to
register
.tasksas a system index.