Add `ignore_missing_component_templates` config option #92436

ruflin · 2022-12-19T08:23:02Z

This change introduces the configuration option ignore_missing_component_templates as discussed in #92426 The implementation option 6 was picked with a slight adjustment meaning no patterns are allowed.

Implementation

During the creation of an index template, the list of component templates is checked if all component templates exist. This check is extended to skip any component templates which are listed under ignore_missing_component_templates. An index template that skips the check for the component template logs-foo@custom looks as following:

PUT _index_template/logs-foo
{
  "index_patterns": ["logs-foo-*"],
  "data_stream": { },
  "composed_of": ["logs-foo@package", "logs-foo@custom"],
  "ignore_missing_component_templates": ["logs-foo@custom"],
  "priority": 500
}

The component template logs-foo@package has to exist before creation. It can be created with:

PUT _component_template/logs-foo@custom
{
  "template": {
    "mappings": {
      "properties": {
        "host.ip": {
          "type": "ip"
        }
      }
    }
  }
}

Testing

For manual testing, different scenarios can be tested. To simplify testing, the commands from .http file are added. Before each test run, a clean cluster is expected.

New behaviour, missing component template

With the new config option, it must be possible to create an index template with a missing component templates without getting an error:

### Add logs-foo@package component template

PUT http://localhost:9200/
    _component_template/logs-foo@package
Authorization: Basic elastic password
Content-Type: application/json

{
  "template": {
    "mappings": {
      "properties": {
        "host.name": {
          "type": "keyword"
        }
      }
    }
  }
}

### Add logs-foo index template

PUT http://localhost:9200/
    _index_template/logs-foo
Authorization: Basic elastic password
Content-Type: application/json

{
  "index_patterns": ["logs-foo-*"],
  "data_stream": { },
  "composed_of": ["logs-foo@package", "logs-foo@custom"],
  "ignore_missing_component_templates": ["logs-foo@custom"],
  "priority": 500
}

### Create data stream

PUT http://localhost:9200/
    _data_stream/logs-foo-bar
Authorization: Basic elastic password
Content-Type: application/json

### Check if mappings exist

GET http://localhost:9200/
    logs-foo-bar
Authorization: Basic elastic password
Content-Type: application/json

It is checked if all templates could be created and data stream mappings are correct.

Old behaviour, with all component templates

In the following, a component template is made optional but it already exists. It is checked, that it will show up in the mappings:

### Add logs-foo@package component template

PUT http://localhost:9200/
    _component_template/logs-foo@package
Authorization: Basic elastic password
Content-Type: application/json

{
  "template": {
    "mappings": {
      "properties": {
        "host.name": {
          "type": "keyword"
        }
      }
    }
  }
}

### Add logs-foo@custom component template

PUT http://localhost:9200/
    _component_template/logs-foo@custom
Authorization: Basic elastic password
Content-Type: application/json

{
  "template": {
    "mappings": {
      "properties": {
        "host.ip": {
          "type": "ip"
        }
      }
    }
  }
}

### Add logs-foo index template

PUT http://localhost:9200/
    _index_template/logs-foo
Authorization: Basic elastic password
Content-Type: application/json

{
  "index_patterns": ["logs-foo-*"],
  "data_stream": { },
  "composed_of": ["logs-foo@package", "logs-foo@custom"],
  "ignore_missing_component_templates": ["logs-foo@custom"],
  "priority": 500
}

### Create data stream

PUT http://localhost:9200/
    _data_stream/logs-foo-bar
Authorization: Basic elastic password
Content-Type: application/json

### Check if mappings exist

GET http://localhost:9200/
    logs-foo-bar
Authorization: Basic elastic password
Content-Type: application/json

Check old behaviour

Ensure, that the old behaviour still exists when a component template is used that is not part of ignore_missing_component_templates:

### Add logs-foo index template

PUT http://localhost:9200/
    _index_template/logs-foo
Authorization: Basic elastic password
Content-Type: application/json

{
  "index_patterns": ["logs-foo-*"],
  "data_stream": { },
  "composed_of": ["logs-foo@package", "logs-foo@custom"],
  "ignore_missing_component_templates": ["logs-foo@custom"],
  "priority": 500
}

As described in elastic#92426 a config option to not ignore missing component templates is needed. This introduces the config option `ignore_missing_component_templates`. If set to true, missing component templates are ignored. If set to false or not set at all, the existing behaviour applies. This is currently a draft PR as it only contains the functionality. It still needs tests and docs. Goal is to get an initial set of opinions on it.

server/src/main/java/org/elasticsearch/cluster/metadata/ComposableIndexTemplate.java

server/src/main/java/org/elasticsearch/cluster/metadata/MetadataIndexTemplateService.java

docs/reference/indices/ignore-missing-component-templates.asciidoc

abdonpijpelink

Left some comments and suggestions that should help the code snippets CI tests pass.

docs/reference/indices/ignore-missing-component-templates.asciidoc

ruflin · 2023-01-16T15:33:03Z

@dakrone Who would be a good person to have a first read of this PR. I would like to know if the approach I'm following is the expected path as this is has been my first ES PR in quite some time.

ruflin · 2023-01-17T07:16:46Z

What exactly is the bwc ci build? How can I run this one locally to figure out what is not working?

.../src/yamlRestTest/resources/rest-api-spec/test/indices.put_index_template/15_composition.yml

ruflin · 2023-01-18T11:48:55Z

Update: Found the issue. The bot put two areas into the list but seems only 1 is allowed.

The docs-check currently fails with:

12:46:52 Execution failed for task ':validateChangelogs'.
12:46:52 > Verification failed. See the report at: file:/dev/shm/elastic+elasticsearch+pull-request+docs-check/build/reports/validateYaml.txt
12:46:52   Error validating YAML: 1 files contained 1 violations

But it is not clear to me why the changelog is invalid?

ruflin · 2023-01-18T13:19:03Z

@dakrone Thanks for the review. I did a first round of changes and by now also CI is happy. Could you take another look?

dakrone

Okay, I left a few more minor comments, but this is pretty close.

docs/reference/indices/ignore-missing-component-templates.asciidoc

.../src/yamlRestTest/resources/rest-api-spec/test/indices.put_index_template/15_composition.yml

dakrone · 2023-01-27T17:18:14Z

server/src/main/java/org/elasticsearch/cluster/metadata/MetadataIndexTemplateService.java

+                            + name
+                            + "] specifies a missing component templates ["
+                            + missingComponentTemplate
+                            + "] that does not exist and is not part of 'ignore_missing_component_templates'"


If it is used with a non empty list but template is not found, we leave a not in the logs.

I'm not sure what you mean by this, if it has a non-empty list and the template is not found, we should throw an exception like this currently does (rather than logging anything).

server/src/test/java/org/elasticsearch/cluster/metadata/ComposableIndexTemplateTests.java

server/src/test/java/org/elasticsearch/cluster/metadata/MetadataIndexTemplateServiceTests.java

…dices.put_index_template/15_composition.yml Co-authored-by: Lee Hinman <dakrone@users.noreply.github.com>

…ableIndexTemplateTests.java Co-authored-by: Lee Hinman <dakrone@users.noreply.github.com>

docs/reference/indices/ignore-missing-component-templates.asciidoc

.../src/yamlRestTest/resources/rest-api-spec/test/indices.put_index_template/15_composition.yml

server/src/test/java/org/elasticsearch/cluster/metadata/MetadataIndexTemplateServiceTests.java

ruflin · 2023-01-30T09:22:02Z

docs/changelog/92436.yaml

@@ -0,0 +1,7 @@
+pr: 92436
+summary: Add `ignore_missing_component_templates` config option
+# The bot was creating `area: "Data streams, Indices APIs"` but this seems to be invalid


@dakrone I'll remove this line but maybe worth someone checks on why this is happening. It seems the bot can't handle 2 labels well (or the other way around).

That's because PRs only support a single green label, so the entry does not get duplicated in the changelog. This change would be indices APIs more than data streams (you don't have to change the labels, since you fixed this.)

ruflin · 2023-01-30T12:15:41Z

server/src/main/java/org/elasticsearch/cluster/metadata/MetadataIndexTemplateService.java

@@ -505,17 +505,51 @@ public static void validateV2TemplateRequest(Metadata metadata, String name, Com
        }

        final Map<String, ComponentTemplate> componentTemplates = metadata.componentTemplates();
+        final List<String> ignoreMissingComponentTemplates = (
+            template.getIgnoreMissingComponentTemplates() == null? List.of():template.getIgnoreMissingComponentTemplates()


I was going forth and back on this one. Either I check for null in getIgnoreMissingComponentTemplates and always return an empty list or do it here. It seems for non requirement params null is in many places and excepted value. Also it put a bit a challenge in many other places that expect the entry to not show up by default if it is entry. So I do the check here.

dakrone · 2023-01-30T22:56:06Z

@elasticmachine run elasticsearch-ci/part-1

dakrone

LGTM, thanks for iterating on this @ruflin, I'll merge it in.

ruflin · 2023-01-31T18:28:11Z

@dakrone Thanks for the reviews and feedback! Looking forward to my future contributions ;-)

This change introduces the configuration option `ignore_missing_component_templates` as discussed in elastic#92426 The implementation [option 6](elastic#92426 (comment)) was picked with a slight adjustment meaning no patterns are allowed. ## Implementation During the creation of an index template, the list of component templates is checked if all component templates exist. This check is extended to skip any component templates which are listed under `ignore_missing_component_templates`. An index template that skips the check for the component template `logs-foo@custom` looks as following: ``` PUT _index_template/logs-foo { "index_patterns": ["logs-foo-*"], "data_stream": { }, "composed_of": ["logs-foo@package", "logs-foo@custom"], "ignore_missing_component_templates": ["logs-foo@custom"], "priority": 500 } ``` The component template `logs-foo@package` has to exist before creation. It can be created with: ``` PUT _component_template/logs-foo@custom { "template": { "mappings": { "properties": { "host.ip": { "type": "ip" } } } } } ``` ## Testing For manual testing, different scenarios can be tested. To simplify testing, the commands from `.http` file are added. Before each test run, a clean cluster is expected. ### New behaviour, missing component template With the new config option, it must be possible to create an index template with a missing component templates without getting an error: ``` ### Add logs-foo@package component template PUT http://localhost:9200/ _component_template/logs-foo@package Authorization: Basic elastic password Content-Type: application/json { "template": { "mappings": { "properties": { "host.name": { "type": "keyword" } } } } } ### Add logs-foo index template PUT http://localhost:9200/ _index_template/logs-foo Authorization: Basic elastic password Content-Type: application/json { "index_patterns": ["logs-foo-*"], "data_stream": { }, "composed_of": ["logs-foo@package", "logs-foo@custom"], "ignore_missing_component_templates": ["logs-foo@custom"], "priority": 500 } ### Create data stream PUT http://localhost:9200/ _data_stream/logs-foo-bar Authorization: Basic elastic password Content-Type: application/json ### Check if mappings exist GET http://localhost:9200/ logs-foo-bar Authorization: Basic elastic password Content-Type: application/json ``` It is checked if all templates could be created and data stream mappings are correct. ### Old behaviour, with all component templates In the following, a component template is made optional but it already exists. It is checked, that it will show up in the mappings: ``` ### Add logs-foo@package component template PUT http://localhost:9200/ _component_template/logs-foo@package Authorization: Basic elastic password Content-Type: application/json { "template": { "mappings": { "properties": { "host.name": { "type": "keyword" } } } } } ### Add logs-foo@custom component template PUT http://localhost:9200/ _component_template/logs-foo@custom Authorization: Basic elastic password Content-Type: application/json { "template": { "mappings": { "properties": { "host.ip": { "type": "ip" } } } } } ### Add logs-foo index template PUT http://localhost:9200/ _index_template/logs-foo Authorization: Basic elastic password Content-Type: application/json { "index_patterns": ["logs-foo-*"], "data_stream": { }, "composed_of": ["logs-foo@package", "logs-foo@custom"], "ignore_missing_component_templates": ["logs-foo@custom"], "priority": 500 } ### Create data stream PUT http://localhost:9200/ _data_stream/logs-foo-bar Authorization: Basic elastic password Content-Type: application/json ### Check if mappings exist GET http://localhost:9200/ logs-foo-bar Authorization: Basic elastic password Content-Type: application/json ``` ### Check old behaviour Ensure, that the old behaviour still exists when a component template is used that is not part of `ignore_missing_component_templates`: ``` ### Add logs-foo index template PUT http://localhost:9200/ _index_template/logs-foo Authorization: Basic elastic password Content-Type: application/json { "index_patterns": ["logs-foo-*"], "data_stream": { }, "composed_of": ["logs-foo@package", "logs-foo@custom"], "ignore_missing_component_templates": ["logs-foo@custom"], "priority": 500 } ``` Co-authored-by: Lee Hinman <dakrone@users.noreply.github.com>

ruflin self-assigned this Dec 19, 2022

elasticsearchmachine added v8.7.0 external-contributor Pull request authored by a developer outside the Elasticsearch team labels Dec 19, 2022

ruflin commented Dec 19, 2022

View reviewed changes

server/src/main/java/org/elasticsearch/cluster/metadata/ComposableIndexTemplate.java Outdated Show resolved Hide resolved

server/src/main/java/org/elasticsearch/cluster/metadata/ComposableIndexTemplate.java Outdated Show resolved Hide resolved

ruflin mentioned this pull request Dec 19, 2022

[Composable templates] Improve support for customizing settings and mappings for groups of data streams #92426

Closed

ruflin changed the title ~~Add ignore_missing_component_tempaltes config option~~ Add ignore_missing_component_templates config option Dec 19, 2022

ruflin added 7 commits January 9, 2023 11:40

Merge branch 'main' into ignore-missing-component-template

5aaa91b

adjust from bool to ignore_missing_component_templates as an array

3befa9b

update unit tests

21ca694

add one more constructor to keep old behaviour

d4a6c31

Merge branch 'main' into ignore-missing-component-template

5b8a46f

fix some unit tests

ab46b1b

add basic tests

64e141f

ruflin commented Jan 16, 2023

View reviewed changes

server/src/main/java/org/elasticsearch/cluster/metadata/MetadataIndexTemplateService.java Outdated Show resolved Hide resolved

ruflin added 4 commits January 16, 2023 11:33

remove not needed test

1a18d50

Merge branch 'main' into ignore-missing-component-template

0512edf

fixing test cases

5dd2ee7

add documentation

c28b24b

ruflin commented Jan 16, 2023

View reviewed changes

docs/reference/indices/ignore-missing-component-templates.asciidoc Outdated Show resolved Hide resolved

add more docs

9c68539

abdonpijpelink reviewed Jan 16, 2023

View reviewed changes

add code reviews for docs

1cd0127

ruflin marked this pull request as ready for review January 16, 2023 15:32

elasticsearchmachine added the needs:triage Requires assignment of a team area label label Jan 16, 2023

ruflin added 3 commits January 16, 2023 16:47

add JSON to source blocks

29a34fb

apply formatting rules

71ae211

fix tests, cleanup log message

23559c0

ruflin commented Jan 18, 2023

View reviewed changes

.../src/yamlRestTest/resources/rest-api-spec/test/indices.put_index_template/15_composition.yml Outdated Show resolved Hide resolved

ruflin added 2 commits January 18, 2023 12:53

fix changelog area entry

7a66bf3

fix yaml test

cc9cc5c

dakrone self-requested a review January 18, 2023 16:23

joshdover mentioned this pull request Jan 25, 2023

[Fleet] Add support for customizing integration data streams at more levels of granularity elastic/kibana#149484

Open

dakrone requested changes Jan 27, 2023

View reviewed changes

ruflin and others added 3 commits January 30, 2023 10:12

Update rest-api-spec/src/yamlRestTest/resources/rest-api-spec/test/in…

1de49ca

…dices.put_index_template/15_composition.yml Co-authored-by: Lee Hinman <dakrone@users.noreply.github.com>

Update server/src/test/java/org/elasticsearch/cluster/metadata/Compos…

973c49f

…ableIndexTemplateTests.java Co-authored-by: Lee Hinman <dakrone@users.noreply.github.com>

add initial pr review cleanup

072c81a

ruflin commented Jan 30, 2023

View reviewed changes

update more tests

da7e66d

ruflin commented Jan 30, 2023

View reviewed changes

ruflin added 5 commits January 30, 2023 13:17

remove comment from changelog and remove old exception code

cc824c5

cleanup code / spotless

2bb57f5

fix yaml test to use indices again

f12afe8

Merge branch 'main' into ignore-missing-component-template

57a9ce9

fix import after merge

55ef4bc

This comment was marked as outdated.

Sign in to view

Merge branch 'main' into ignore-missing-component-template

0f64578

dakrone approved these changes Jan 31, 2023

View reviewed changes

dakrone merged commit 9f4d7fa into elastic:main Jan 31, 2023

ruflin deleted the ignore-missing-component-template branch January 31, 2023 18:27

felixbarny mentioned this pull request Apr 23, 2023

[Logs+] Add support for logs@custom component template #95481

Merged

eyalkoren mentioned this pull request Apr 25, 2023

Fix issues with ignore_missing_component_templates #95527

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add `ignore_missing_component_templates` config option #92436

Add `ignore_missing_component_templates` config option #92436

ruflin commented Dec 19, 2022 •

edited

Loading

abdonpijpelink left a comment •

edited

Loading

ruflin commented Jan 16, 2023

ruflin commented Jan 17, 2023

ruflin commented Jan 18, 2023 •

edited

Loading

ruflin commented Jan 18, 2023 •

edited

Loading

dakrone left a comment

dakrone Jan 27, 2023

ruflin Jan 30, 2023

dakrone Jan 31, 2023

ruflin Jan 30, 2023

dakrone commented Jan 30, 2023

This comment was marked as outdated.

dakrone left a comment

ruflin commented Jan 31, 2023

Add ignore_missing_component_templates config option #92436

Add ignore_missing_component_templates config option #92436

Conversation

ruflin commented Dec 19, 2022 • edited Loading

Implementation

Testing

New behaviour, missing component template

Old behaviour, with all component templates

Check old behaviour

abdonpijpelink left a comment • edited Loading

Choose a reason for hiding this comment

ruflin commented Jan 16, 2023

ruflin commented Jan 17, 2023

ruflin commented Jan 18, 2023 • edited Loading

ruflin commented Jan 18, 2023 • edited Loading

dakrone left a comment

Choose a reason for hiding this comment

dakrone Jan 27, 2023

Choose a reason for hiding this comment

ruflin Jan 30, 2023

Choose a reason for hiding this comment

dakrone Jan 31, 2023

Choose a reason for hiding this comment

ruflin Jan 30, 2023

Choose a reason for hiding this comment

dakrone commented Jan 30, 2023

This comment was marked as outdated.

dakrone left a comment

Choose a reason for hiding this comment

ruflin commented Jan 31, 2023

Add `ignore_missing_component_templates` config option #92436

Add `ignore_missing_component_templates` config option #92436

ruflin commented Dec 19, 2022 •

edited

Loading

abdonpijpelink left a comment •

edited

Loading

ruflin commented Jan 18, 2023 •

edited

Loading

ruflin commented Jan 18, 2023 •

edited

Loading