Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make timestamp field mapping part of data stream definition. #58642

Closed

Conversation

martijnvg
Copy link
Member

@martijnvg martijnvg commented Jun 28, 2020

Instead of requiring that the field mapping for the timestamp field is defined
in the mapping of the composable index template for data streams, the mapping
should also be definable in the data stream definition itself, since the field
name of the data stream is already defined there.

The timestamp field mapping will be applied when a new backing index of a
data stream is being created.

When a new data stream is created and composable index templates are evaluated then
the mapping are applied in the following order:

  • Optional mappings from component templates that are associated with the matching template
  • Optional mappings from the matching template itself.
  • Optional timestamp field mapping from data stream definition of the matching template.

If no mapping for timestamp field is defined in a component template, composable index template,
data stream definition inside a composable index template then a default mapping for the timestamp
field will be generated at data stream creation time.

Before this commit the field mapping of the timestamp field should be defined as mapping
in the composable template:

PUT /_index_template/logs_data_stream
{
  "index_patterns": [ "logs*" ],
  "data_stream": {
    "timestamp_field": "@timestamp"
  },
  "template": {
    "mappings": {
      "properties": {
        "@timestamp": {
          "type": "date"
        }
      }
    }
  }
}

With this commit, the timestamp field mapping may also be defined as part of the data stream definition:

PUT /_index_template/logs_data_stream
{
  "index_patterns": [ "logs*" ],
  "data_stream": {
    "timestamp_field": "@timestamp",
    "timestamp_mapping": {
      "type": "date"
    }
  }
}

The field mapping is now also optional:

PUT /_index_template/logs_data_stream
{
  "index_patterns": [ "logs*" ],
  "data_stream": {
    "timestamp_field": "@timestamp"
  }
}

In the latter case the following mapping snippet is used as default:
{"type": "date"}

Closes to #58583

Instead of requiring that the field mapping for the timestamp field is defined
in the mapping of the composable index template for data streams, the mapping
should also be definable in the data stream definition itself, since the field
name of the data stream is already defined there.

The timestamp field mapping will be applied when a new backing index of a
data stream is being created.

When a new data stream is created and composable index templates are evaluated then
the mapping are applied in the following order:
* Optional mappings from component templates that are associated with the matching template
* Optional mappings from the matching template itself.
* Optional timestamp field mapping from data stream definition of the matching template.

If no mapping for timestamp field is defined in a component template, composable index template,
data stream definition inside a composable index template then a default mapping for the timestamp
field will be generated at data stream creation time.

Before this commit the field mapping of the timestamp field should be defined as mapping
in the composable template:

```
PUT /_index_template/logs_data_stream
{
  "index_patterns": [ "logs*" ],
  "data_stream": {
    "timestamp_field": "@timestamp"
  },
  "template": {
    "mappings": {
      "properties": {
        "@timestamp": {
          "type": "date"
        }
      }
    }
  }
}
```

With this commit, the timestamp field mapping may also be defined as part of the data stream definition:

```
PUT /_index_template/logs_data_stream
{
  "index_patterns": [ "logs*" ],
  "data_stream": {
    "timestamp_field": "@timestamp",
    "timestamp_mapping": {
      "type": "date"
    }
  }
}
```

The field mapping is now also optional:

```
PUT /_index_template/logs_data_stream
{
  "index_patterns": [ "logs*" ],
  "data_stream": {
    "timestamp_field": "@timestamp"
  }
}
```

In the latter case the following mapping snippet is used as default:
{"type": "date"}

Closes to elastic#58583
@martijnvg martijnvg added the :Data Management/Data streams Data streams and their lifecycles label Jun 28, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-core-features (:Core/Features/Data streams)

@elasticmachine elasticmachine added the Team:Data Management Meta label for data/management team label Jun 28, 2020
Copy link
Member

@dakrone dakrone left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This LGTM, thanks Martijn

martijnvg added a commit to martijnvg/elasticsearch that referenced this pull request Jul 6, 2020
The commit makes the following changes:
* The timestamp field of a data stream definition in a composable
  index template can only be set to '@timestamp'.
* Removed custom data stream timestamp field validation and reuse the validation from `TimestampFieldMapper` and
  instead only check that the _timestamp field mapping has been defined on a backing index of a data stream.
* Moved code that injects _timestamp meta field mapping from `MetadataCreateIndexService#applyCreateIndexRequestWithV2Template58956(...)` method
  to `MetadataIndexTemplateService#collectMappings(...)` method.
* Fixed a bug (elastic#58956) that cases timestamp field validation to be performed
  for each template and instead of the final mappings that is created.

Relates to elastic#58642
Relates to elastic#53100
Closes elastic#58956
Closes elastic#58583
martijnvg added a commit that referenced this pull request Jul 8, 2020
The commit makes the following changes:
* The timestamp field of a data stream definition in a composable
  index template can only be set to '@timestamp'.
* Removed custom data stream timestamp field validation and reuse the validation from `TimestampFieldMapper` and
  instead only check that the _timestamp field mapping has been defined on a backing index of a data stream.
* Moved code that injects _timestamp meta field mapping from `MetadataCreateIndexService#applyCreateIndexRequestWithV2Template58956(...)` method
  to `MetadataIndexTemplateService#collectMappings(...)` method.
* Fixed a bug (#58956) that cases timestamp field validation to be performed
  for each template and instead of the final mappings that is created.
* only apply _timestamp meta field if index is created as part of a data stream or data stream rollover,
this fixes a docs test, where a regular index creation matches (logs-*) with a template with a data stream definition.

Relates to #58642
Relates to #53100
Closes #58956
Closes #58583
martijnvg added a commit to martijnvg/elasticsearch that referenced this pull request Jul 8, 2020
Backport elastic#59076 of to 7.x branch.

The commit makes the following changes:
* The timestamp field of a data stream definition in a composable
  index template can only be set to '@timestamp'.
* Removed custom data stream timestamp field validation and reuse the validation from `TimestampFieldMapper` and
  instead only check that the _timestamp field mapping has been defined on a backing index of a data stream.
* Moved code that injects _timestamp meta field mapping from `MetadataCreateIndexService#applyCreateIndexRequestWithV2Template58956(...)` method
  to `MetadataIndexTemplateService#collectMappings(...)` method.
* Fixed a bug (elastic#58956) that cases timestamp field validation to be performed
  for each template and instead of the final mappings that is created.
* only apply _timestamp meta field if index is created as part of a data stream or data stream rollover,
this fixes a docs test, where a regular index creation matches (logs-*) with a template with a data stream definition.

Relates to elastic#58642
Relates to elastic#53100
Closes elastic#58956
Closes elastic#58583
martijnvg added a commit that referenced this pull request Jul 8, 2020
Backport of #59076 to 7.x branch.

The commit makes the following changes:
* The timestamp field of a data stream definition in a composable
  index template can only be set to '@timestamp'.
* Removed custom data stream timestamp field validation and reuse the validation from `TimestampFieldMapper` and
  instead only check that the _timestamp field mapping has been defined on a backing index of a data stream.
* Moved code that injects _timestamp meta field mapping from `MetadataCreateIndexService#applyCreateIndexRequestWithV2Template58956(...)` method
  to `MetadataIndexTemplateService#collectMappings(...)` method.
* Fixed a bug (#58956) that cases timestamp field validation to be performed
  for each template and instead of the final mappings that is created.
* only apply _timestamp meta field if index is created as part of a data stream or data stream rollover,
this fixes a docs test, where a regular index creation matches (logs-*) with a template with a data stream definition.

Relates to #58642
Relates to #53100
Closes #58956
Closes #58583
@martijnvg
Copy link
Member Author

Superseded by #59076 and #59244

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Data Management/Data streams Data streams and their lifecycles >non-issue Team:Data Management Meta label for data/management team v7.9.0 v8.0.0-alpha1
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants