Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add time_series_metric parameter #76766

Merged
merged 17 commits into from
Sep 20, 2021
Merged

Conversation

csoulios
Copy link
Contributor

This PR adds the time_series_metric parameter to the following field types:

  • Numeric field types
  • histogram
  • aggregate_metric_double

The time_series_metric parameter is of type string (default is null) and is used to mark that a field is a time series metric field with a specific metric type. The following metric types are supported for each Elasticsearch field type:

Metric type ES field type
gauge number, aggregate_metric_double
counter number, aggregate_metric_double
histogram histogram
summary aggregate_metric_double

Example of setting the time_series_metric parameter to a field can be seen here:

PUT my-index-000001
{
  "mappings": {
    "properties": {
      "free_memory": {
        "type": "double",
        "time_series_metric": "gauge"
      },
      "net_bytes_in": {
        "type": "double",
        "time_series_metric": "counter"
      }
    }
  }
}

This PR together with #74450 annotate which Elasticsearch fields are time series dimensions or metrics.

Closes #74014

@csoulios csoulios added >enhancement :Search Foundations/Mapping Index mappings, including merging and defining field types backport pending :StorageEngine/Rollup Turn fine-grained time-based data into coarser-grained data v8.0.0-alpha1 v7.16.0 labels Aug 20, 2021
@csoulios csoulios requested review from nik9000 and imotov August 20, 2021 14:02
@elasticmachine elasticmachine added Team:Search Meta label for search team Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) labels Aug 20, 2021
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-search (Team:Search)

@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-analytics-geo (Team:Analytics)

@jrodewig jrodewig added the :StorageEngine/TSDB You know, for Metrics label Aug 21, 2021
Copy link
Contributor

@imotov imotov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks good to me in general. I have only a couple of comments. I think some integration level tests, maybe even rest test could be helpful, so we have some clear examples how this is used. It also looks like long fields can be metrics and dimensions at the same time, not sure if this is intentional. It would be great if @nik9000 could review it since he is working with this at the moment.

@nik9000
Copy link
Member

nik9000 commented Aug 23, 2021

It would be great if @nik9000 could review it since he is working with this at the moment.

I'm not 100% sure we'll keep allowing long to be a dimension. But I guess it's easy enough to add the assertion that you can't do that so we may as well.

@imotov
Copy link
Contributor

imotov commented Aug 23, 2021

We will not keep it as part of routing... but nothing prevents us from keeping it as non-routing dimension.

@csoulios csoulios requested a review from nik9000 September 3, 2021 15:02
@csoulios csoulios requested a review from nik9000 September 15, 2021 13:24
Copy link
Member

@nik9000 nik9000 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

Sorry for the delay!

@csoulios csoulios merged commit b2ed599 into elastic:master Sep 20, 2021
@csoulios csoulios deleted the metric-mapping-param branch September 20, 2021 14:40
csoulios added a commit that referenced this pull request Sep 20, 2021
This PR renames dimension mapping parameter to time_series_dimension to make it consistent with time_series_metric parameter (#76766)

Relates to #74450 and #74014
jrodewig added a commit that referenced this pull request Sep 23, 2021
Changes:
* Documents the `time_series_metric` mapping parameter for PR #76766.
* Renames the `dimension` parameter to `time_series_dimension` for PR #78012.
* Adds support for `unsigned_long` to `time_series_dimension` for PR #78204.
jrodewig added a commit that referenced this pull request Sep 23, 2021
Changes:
* Documents the `time_series_metric` mapping parameter for PR #76766.
* Renames the `dimension` parameter to `time_series_dimension` for PR #78012.
* Adds support for `unsigned_long` to `time_series_dimension` for PR #78204.
csoulios added a commit that referenced this pull request Sep 23, 2021
    Added the time_series_metric mapping parameter to the unsigned_long and scaled_float field types
    Added the time_series_dimension mapping parameter to the unsigned_long field type

Fixes #78100

Relates to #76766, #74450 and #74014
csoulios added a commit to csoulios/elasticsearch that referenced this pull request Sep 23, 2021
This PR adds the time_series_metric parameter to the following field types:

    Numeric field types
    histogram
    aggregate_metric_double
csoulios added a commit to csoulios/elasticsearch that referenced this pull request Sep 23, 2021
…tic#78012)

This PR renames dimension mapping parameter to time_series_dimension to make it consistent with time_series_metric parameter (elastic#76766)

Relates to elastic#74450 and elastic#74014
csoulios added a commit to csoulios/elasticsearch that referenced this pull request Sep 23, 2021
…#78204)

    Added the time_series_metric mapping parameter to the unsigned_long and scaled_float field types
    Added the time_series_dimension mapping parameter to the unsigned_long field type

Fixes elastic#78100

Relates to elastic#76766, elastic#74450 and elastic#74014
csoulios added a commit that referenced this pull request Sep 27, 2021
…rameters (#78265)

Backports the following PRs:

* Add dimension mapping parameter (#74450)

Added the dimension parameter to the following field types:

    keyword
    ip
    Numeric field types (integer, long, byte, short)

The dimension parameter is of type boolean (default: false) and is used
to mark that a field is a time series dimension field.

Relates to #74014

* Add constraints to dimension fields (#74939)

This PR adds the following constraints to dimension fields:

    It must be an indexed field and must has doc values
    It cannot be multi-valued
    The number of dimension fields in the index mapping must not be more than 16. This should be configurable through an index property (index.mapping.dimension_fields.limit)
    keyword fields cannot be more than 1024 bytes long
    keyword fields must not use a normalizer

Based on the code added in PR #74450
Relates to #74660

* Expand DocumentMapperTests (#76368)

Adds a test for setting the maximum number of dimensions setting and
tests the names and types of the metadata fields in the index.
Previously we just asserted the count of metadata fields. That made it
hard to read failures.

* Fix broken test for dimension keywords (#75408)

Test was failing because it was testing 1024 bytes long keyword and assertion was failing.

Closes #75225

* Checkstyle

* Add time_series_metric parameter (#76766)

This PR adds the time_series_metric parameter to the following field types:

    Numeric field types
    histogram
    aggregate_metric_double

* Rename `dimension` mapping parameter to `time_series_dimension` (#78012)

This PR renames dimension mapping parameter to time_series_dimension to make it consistent with time_series_metric parameter (#76766)

Relates to #74450 and #74014

* Add time series params to `unsigned_long` and `scaled_float` (#78204)

    Added the time_series_metric mapping parameter to the unsigned_long and scaled_float field types
    Added the time_series_dimension mapping parameter to the unsigned_long field type

Fixes #78100

Relates to #76766, #74450 and #74014

Co-authored-by: Nik Everett <nik9000@gmail.com>
@wchaparro wchaparro assigned csoulios and unassigned csoulios Dec 16, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
>enhancement :Search Foundations/Mapping Index mappings, including merging and defining field types :StorageEngine/Rollup Turn fine-grained time-based data into coarser-grained data :StorageEngine/TSDB You know, for Metrics Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) Team:Search Meta label for search team v7.16.0 v8.0.0-alpha1 v8.0.0-beta1
Projects
None yet
Development

Successfully merging this pull request may close these issues.

New mapping parameters to annotate dimensions and metrics in timeseries data
8 participants