-
Notifications
You must be signed in to change notification settings - Fork 24.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Make timestamp field mapping part of data stream definition. #58642
Closed
martijnvg
wants to merge
13
commits into
elastic:master
from
martijnvg:add_mapping_data_stream_template
Closed
Make timestamp field mapping part of data stream definition. #58642
martijnvg
wants to merge
13
commits into
elastic:master
from
martijnvg:add_mapping_data_stream_template
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Instead of requiring that the field mapping for the timestamp field is defined in the mapping of the composable index template for data streams, the mapping should also be definable in the data stream definition itself, since the field name of the data stream is already defined there. The timestamp field mapping will be applied when a new backing index of a data stream is being created. When a new data stream is created and composable index templates are evaluated then the mapping are applied in the following order: * Optional mappings from component templates that are associated with the matching template * Optional mappings from the matching template itself. * Optional timestamp field mapping from data stream definition of the matching template. If no mapping for timestamp field is defined in a component template, composable index template, data stream definition inside a composable index template then a default mapping for the timestamp field will be generated at data stream creation time. Before this commit the field mapping of the timestamp field should be defined as mapping in the composable template: ``` PUT /_index_template/logs_data_stream { "index_patterns": [ "logs*" ], "data_stream": { "timestamp_field": "@timestamp" }, "template": { "mappings": { "properties": { "@timestamp": { "type": "date" } } } } } ``` With this commit, the timestamp field mapping may also be defined as part of the data stream definition: ``` PUT /_index_template/logs_data_stream { "index_patterns": [ "logs*" ], "data_stream": { "timestamp_field": "@timestamp", "timestamp_mapping": { "type": "date" } } } ``` The field mapping is now also optional: ``` PUT /_index_template/logs_data_stream { "index_patterns": [ "logs*" ], "data_stream": { "timestamp_field": "@timestamp" } } ``` In the latter case the following mapping snippet is used as default: {"type": "date"} Closes to elastic#58583
martijnvg
added
the
:Data Management/Data streams
Data streams and their lifecycles
label
Jun 28, 2020
Pinging @elastic/es-core-features (:Core/Features/Data streams) |
elasticmachine
added
the
Team:Data Management
Meta label for data/management team
label
Jun 28, 2020
dakrone
approved these changes
Jul 1, 2020
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This LGTM, thanks Martijn
martijnvg
added a commit
to martijnvg/elasticsearch
that referenced
this pull request
Jul 6, 2020
The commit makes the following changes: * The timestamp field of a data stream definition in a composable index template can only be set to '@timestamp'. * Removed custom data stream timestamp field validation and reuse the validation from `TimestampFieldMapper` and instead only check that the _timestamp field mapping has been defined on a backing index of a data stream. * Moved code that injects _timestamp meta field mapping from `MetadataCreateIndexService#applyCreateIndexRequestWithV2Template58956(...)` method to `MetadataIndexTemplateService#collectMappings(...)` method. * Fixed a bug (elastic#58956) that cases timestamp field validation to be performed for each template and instead of the final mappings that is created. Relates to elastic#58642 Relates to elastic#53100 Closes elastic#58956 Closes elastic#58583
martijnvg
added a commit
that referenced
this pull request
Jul 8, 2020
The commit makes the following changes: * The timestamp field of a data stream definition in a composable index template can only be set to '@timestamp'. * Removed custom data stream timestamp field validation and reuse the validation from `TimestampFieldMapper` and instead only check that the _timestamp field mapping has been defined on a backing index of a data stream. * Moved code that injects _timestamp meta field mapping from `MetadataCreateIndexService#applyCreateIndexRequestWithV2Template58956(...)` method to `MetadataIndexTemplateService#collectMappings(...)` method. * Fixed a bug (#58956) that cases timestamp field validation to be performed for each template and instead of the final mappings that is created. * only apply _timestamp meta field if index is created as part of a data stream or data stream rollover, this fixes a docs test, where a regular index creation matches (logs-*) with a template with a data stream definition. Relates to #58642 Relates to #53100 Closes #58956 Closes #58583
martijnvg
added a commit
to martijnvg/elasticsearch
that referenced
this pull request
Jul 8, 2020
Backport elastic#59076 of to 7.x branch. The commit makes the following changes: * The timestamp field of a data stream definition in a composable index template can only be set to '@timestamp'. * Removed custom data stream timestamp field validation and reuse the validation from `TimestampFieldMapper` and instead only check that the _timestamp field mapping has been defined on a backing index of a data stream. * Moved code that injects _timestamp meta field mapping from `MetadataCreateIndexService#applyCreateIndexRequestWithV2Template58956(...)` method to `MetadataIndexTemplateService#collectMappings(...)` method. * Fixed a bug (elastic#58956) that cases timestamp field validation to be performed for each template and instead of the final mappings that is created. * only apply _timestamp meta field if index is created as part of a data stream or data stream rollover, this fixes a docs test, where a regular index creation matches (logs-*) with a template with a data stream definition. Relates to elastic#58642 Relates to elastic#53100 Closes elastic#58956 Closes elastic#58583
martijnvg
added a commit
that referenced
this pull request
Jul 8, 2020
Backport of #59076 to 7.x branch. The commit makes the following changes: * The timestamp field of a data stream definition in a composable index template can only be set to '@timestamp'. * Removed custom data stream timestamp field validation and reuse the validation from `TimestampFieldMapper` and instead only check that the _timestamp field mapping has been defined on a backing index of a data stream. * Moved code that injects _timestamp meta field mapping from `MetadataCreateIndexService#applyCreateIndexRequestWithV2Template58956(...)` method to `MetadataIndexTemplateService#collectMappings(...)` method. * Fixed a bug (#58956) that cases timestamp field validation to be performed for each template and instead of the final mappings that is created. * only apply _timestamp meta field if index is created as part of a data stream or data stream rollover, this fixes a docs test, where a regular index creation matches (logs-*) with a template with a data stream definition. Relates to #58642 Relates to #53100 Closes #58956 Closes #58583
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
:Data Management/Data streams
Data streams and their lifecycles
>non-issue
Team:Data Management
Meta label for data/management team
v7.9.0
v8.0.0-alpha1
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Instead of requiring that the field mapping for the timestamp field is defined
in the mapping of the composable index template for data streams, the mapping
should also be definable in the data stream definition itself, since the field
name of the data stream is already defined there.
The timestamp field mapping will be applied when a new backing index of a
data stream is being created.
When a new data stream is created and composable index templates are evaluated then
the mapping are applied in the following order:
If no mapping for timestamp field is defined in a component template, composable index template,
data stream definition inside a composable index template then a default mapping for the timestamp
field will be generated at data stream creation time.
Before this commit the field mapping of the timestamp field should be defined as mapping
in the composable template:
With this commit, the timestamp field mapping may also be defined as part of the data stream definition:
The field mapping is now also optional:
In the latter case the following mapping snippet is used as default:
{"type": "date"}
Closes to #58583