Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make timestamp field mapping part of data stream definition. #58642

Closed

Commits on Jun 28, 2020

  1. Make timestamp field mapping part of data stream definition.

    Instead of requiring that the field mapping for the timestamp field is defined
    in the mapping of the composable index template for data streams, the mapping
    should also be definable in the data stream definition itself, since the field
    name of the data stream is already defined there.
    
    The timestamp field mapping will be applied when a new backing index of a
    data stream is being created.
    
    When a new data stream is created and composable index templates are evaluated then
    the mapping are applied in the following order:
    * Optional mappings from component templates that are associated with the matching template
    * Optional mappings from the matching template itself.
    * Optional timestamp field mapping from data stream definition of the matching template.
    
    If no mapping for timestamp field is defined in a component template, composable index template,
    data stream definition inside a composable index template then a default mapping for the timestamp
    field will be generated at data stream creation time.
    
    Before this commit the field mapping of the timestamp field should be defined as mapping
    in the composable template:
    
    ```
    PUT /_index_template/logs_data_stream
    {
      "index_patterns": [ "logs*" ],
      "data_stream": {
        "timestamp_field": "@timestamp"
      },
      "template": {
        "mappings": {
          "properties": {
            "@timestamp": {
              "type": "date"
            }
          }
        }
      }
    }
    ```
    
    With this commit, the timestamp field mapping may also be defined as part of the data stream definition:
    
    ```
    PUT /_index_template/logs_data_stream
    {
      "index_patterns": [ "logs*" ],
      "data_stream": {
        "timestamp_field": "@timestamp",
        "timestamp_mapping": {
          "type": "date"
        }
      }
    }
    ```
    
    The field mapping is now also optional:
    
    ```
    PUT /_index_template/logs_data_stream
    {
      "index_patterns": [ "logs*" ],
      "data_stream": {
        "timestamp_field": "@timestamp"
      }
    }
    ```
    
    In the latter case the following mapping snippet is used as default:
    {"type": "date"}
    
    Closes to elastic#58583
    martijnvg committed Jun 28, 2020
    Configuration menu
    Copy the full SHA
    7ee2839 View commit details
    Browse the repository at this point in the history

Commits on Jun 29, 2020

  1. fixed checkstyle violation

    martijnvg committed Jun 29, 2020
    Configuration menu
    Copy the full SHA
    a23cc1a View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    aeba6f7 View commit details
    Browse the repository at this point in the history
  3. added more tests

    martijnvg committed Jun 29, 2020
    Configuration menu
    Copy the full SHA
    d5b79a1 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    d3a8faa View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    ed41377 View commit details
    Browse the repository at this point in the history

Commits on Jun 30, 2020

  1. fixed checkstyle violation

    martijnvg committed Jun 30, 2020
    Configuration menu
    Copy the full SHA
    344df52 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    da2df6b View commit details
    Browse the repository at this point in the history
  3. iter after merge

    martijnvg committed Jun 30, 2020
    Configuration menu
    Copy the full SHA
    9255942 View commit details
    Browse the repository at this point in the history
  4. added unit test

    martijnvg committed Jun 30, 2020
    Configuration menu
    Copy the full SHA
    39d22cf View commit details
    Browse the repository at this point in the history
  5. validate ts field once

    martijnvg committed Jun 30, 2020
    Configuration menu
    Copy the full SHA
    32c7c68 View commit details
    Browse the repository at this point in the history
  6. fixed checkstyle violation

    martijnvg committed Jun 30, 2020
    Configuration menu
    Copy the full SHA
    e9f2bc9 View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    3a50c36 View commit details
    Browse the repository at this point in the history