forked from elastic/elasticsearch
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add
ignore_missing_component_templates config option (elastic#92436)
This change introduces the configuration option `ignore_missing_component_templates` as discussed in elastic#92426 The implementation [option 6](elastic#92426 (comment)) was picked with a slight adjustment meaning no patterns are allowed. ## Implementation During the creation of an index template, the list of component templates is checked if all component templates exist. This check is extended to skip any component templates which are listed under `ignore_missing_component_templates`. An index template that skips the check for the component template `logs-foo@custom` looks as following: ``` PUT _index_template/logs-foo { "index_patterns": ["logs-foo-*"], "data_stream": { }, "composed_of": ["logs-foo@package", "logs-foo@custom"], "ignore_missing_component_templates": ["logs-foo@custom"], "priority": 500 } ``` The component template `logs-foo@package` has to exist before creation. It can be created with: ``` PUT _component_template/logs-foo@custom { "template": { "mappings": { "properties": { "host.ip": { "type": "ip" } } } } } ``` ## Testing For manual testing, different scenarios can be tested. To simplify testing, the commands from `.http` file are added. Before each test run, a clean cluster is expected. ### New behaviour, missing component template With the new config option, it must be possible to create an index template with a missing component templates without getting an error: ``` ### Add logs-foo@package component template PUT http://localhost:9200/ _component_template/logs-foo@package Authorization: Basic elastic password Content-Type: application/json { "template": { "mappings": { "properties": { "host.name": { "type": "keyword" } } } } } ### Add logs-foo index template PUT http://localhost:9200/ _index_template/logs-foo Authorization: Basic elastic password Content-Type: application/json { "index_patterns": ["logs-foo-*"], "data_stream": { }, "composed_of": ["logs-foo@package", "logs-foo@custom"], "ignore_missing_component_templates": ["logs-foo@custom"], "priority": 500 } ### Create data stream PUT http://localhost:9200/ _data_stream/logs-foo-bar Authorization: Basic elastic password Content-Type: application/json ### Check if mappings exist GET http://localhost:9200/ logs-foo-bar Authorization: Basic elastic password Content-Type: application/json ``` It is checked if all templates could be created and data stream mappings are correct. ### Old behaviour, with all component templates In the following, a component template is made optional but it already exists. It is checked, that it will show up in the mappings: ``` ### Add logs-foo@package component template PUT http://localhost:9200/ _component_template/logs-foo@package Authorization: Basic elastic password Content-Type: application/json { "template": { "mappings": { "properties": { "host.name": { "type": "keyword" } } } } } ### Add logs-foo@custom component template PUT http://localhost:9200/ _component_template/logs-foo@custom Authorization: Basic elastic password Content-Type: application/json { "template": { "mappings": { "properties": { "host.ip": { "type": "ip" } } } } } ### Add logs-foo index template PUT http://localhost:9200/ _index_template/logs-foo Authorization: Basic elastic password Content-Type: application/json { "index_patterns": ["logs-foo-*"], "data_stream": { }, "composed_of": ["logs-foo@package", "logs-foo@custom"], "ignore_missing_component_templates": ["logs-foo@custom"], "priority": 500 } ### Create data stream PUT http://localhost:9200/ _data_stream/logs-foo-bar Authorization: Basic elastic password Content-Type: application/json ### Check if mappings exist GET http://localhost:9200/ logs-foo-bar Authorization: Basic elastic password Content-Type: application/json ``` ### Check old behaviour Ensure, that the old behaviour still exists when a component template is used that is not part of `ignore_missing_component_templates`: ``` ### Add logs-foo index template PUT http://localhost:9200/ _index_template/logs-foo Authorization: Basic elastic password Content-Type: application/json { "index_patterns": ["logs-foo-*"], "data_stream": { }, "composed_of": ["logs-foo@package", "logs-foo@custom"], "ignore_missing_component_templates": ["logs-foo@custom"], "priority": 500 } ``` Co-authored-by: Lee Hinman <dakrone@users.noreply.github.com>
- Loading branch information
1 parent
845178b
commit 85a3187
Showing
9 changed files
with
423 additions
and
19 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| pr: 92436 | ||
| summary: Add `ignore_missing_component_templates` config option | ||
| area: Indices APIs | ||
| type: enhancement | ||
| issues: | ||
| - 92426 |
95 changes: 95 additions & 0 deletions
95
docs/reference/indices/ignore-missing-component-templates.asciidoc
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,95 @@ | ||
| [[ignore_missing_component_templates]] | ||
| == Config ignore_missing_component_templates | ||
|
|
||
| The configuration option `ignore_missing_component_templates` can be used when an index template references a component template that might not exist. Every time a data stream is created based on the index template, the existence of the component template will be checked. If it exists, it will used to form the index's composite settings. If it does not exist, it is ignored. | ||
|
|
||
| === Usage example | ||
|
|
||
| In the following, one component template and an index template are created. The index template references two component templates, but only the `@package` one exists. | ||
|
|
||
|
|
||
| Create the component template `logs-foo_component1`. This has to be created before the index template as it is not optional: | ||
|
|
||
| [source,console] | ||
| ---- | ||
| PUT _component_template/logs-foo_component1 | ||
| { | ||
| "template": { | ||
| "mappings": { | ||
| "properties": { | ||
| "host.name": { | ||
| "type": "keyword" | ||
| } | ||
| } | ||
| } | ||
| } | ||
| } | ||
| ---- | ||
|
|
||
| Next, the index template will be created and it references two component templates: | ||
|
|
||
| [source,JSON] | ||
| ---- | ||
| "composed_of": ["logs-foo_component1", "logs-foo_component2"] | ||
| ---- | ||
|
|
||
| Before, only the `logs-foo_component1` compontent template was created, meaning the `logs-foo_component2` is missing. Because of this the following entry was added to the config: | ||
|
|
||
| [source,JSON] | ||
| ---- | ||
| "ignore_missing_component_templates": ["logs-foo_component2"], | ||
| ---- | ||
|
|
||
| During creation of the template, it will not validate that `logs-foo_component2` exists: | ||
|
|
||
|
|
||
| [source,console] | ||
| ---- | ||
| PUT _index_template/logs-foo | ||
| { | ||
| "index_patterns": ["logs-foo-*"], | ||
| "data_stream": { }, | ||
| "composed_of": ["logs-foo_component1", "logs-foo_component2"], | ||
| "ignore_missing_component_templates": ["logs-foo_component2"], | ||
| "priority": 500 | ||
| } | ||
| ---- | ||
| // TEST[continued] | ||
|
|
||
| The index template `logs-foo` was successfully created. A data stream can be created based on this template: | ||
|
|
||
| [source,console] | ||
| ---- | ||
| PUT _data_stream/logs-foo-bar | ||
| ---- | ||
| // TEST[continued] | ||
|
|
||
| Looking at the mappings of the data stream, it will contain the `host.name` field. | ||
|
|
||
| At a later stage, the missing component template might be added: | ||
|
|
||
| [source,console] | ||
| ---- | ||
| PUT _component_template/logs-foo_component2 | ||
| { | ||
| "template": { | ||
| "mappings": { | ||
| "properties": { | ||
| "host.ip": { | ||
| "type": "ip" | ||
| } | ||
| } | ||
| } | ||
| } | ||
| } | ||
| ---- | ||
| // TEST[continued] | ||
|
|
||
| This will not have an immediate effect on the data stream. The mapping `host.ip` will only show up in the data stream mappings when the data stream is rolled over automatically next time or a manual rollover is triggered: | ||
|
|
||
| [source,console] | ||
| ---- | ||
| POST logs-foo-bar/_rollover | ||
| ---- | ||
| // TEST[continued] | ||
| // TEST[teardown:data_stream_cleanup] |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.