Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DOCS] Clarify supported rollup fields #70045

Merged
merged 3 commits into from
Mar 9, 2021
Merged

[DOCS] Clarify supported rollup fields #70045

merged 3 commits into from
Mar 9, 2021

Conversation

jrodewig
Copy link
Contributor

@jrodewig jrodewig commented Mar 6, 2021

Notes that in a rollup config:

  • terms support text fields with fielddata enabled.
  • metrics support date and date_nanos fields.
  • unsigned_long fields are not supported.

Relates to #69582

CC @jloleysens. This may impact field type validation for the potential rollup UI.

Preview

https://elasticsearch_70045.docs-preview.app.elstc.co/guide/en/elasticsearch/reference/master/rollup-api.html

@jrodewig jrodewig marked this pull request as ready for review March 8, 2021 13:40
Comment on lines +187 to +188
<<number,Numeric>>, <<date,`date`>>, or <<date_nanos,`date_nanos`>> field to
collect metrics for. `unsigned_long` fields are not supported.
Copy link
Contributor Author

@jrodewig jrodewig Mar 8, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There seems to be a minor bug with date_nanos fields. Here's reproduction steps:

// Map a date_nanos field.
PUT my-index-000001
{
  "mappings": {
    "properties": {
      "my-date-nanos-field": {
        "type": "date_nanos" 
      }
    }
  }
}

// Index a document with the date_nanos field.
POST my-index-000001/_doc
{ 
  "@timestamp": "2099-11-15T13:12:00",
  "my-date-nanos-field": "2015-01-01T12:10:30.123456789Z"
}

// Attempt to roll up the index with the date_nanos field as a metric.
POST my-index-000001/_rollup/rollup-my-index-000001
{
  "groups": {
    "date_histogram": {
      "field": "@timestamp",
      "calendar_interval": "1d"
    }
  },
  "metrics": [
    {
      "field": "my-date-nanos-field",
      "metrics": [
        "min",
        "max"
      ]
    }
  ]
}

The rollup attempt returns an error:

{
  "error" : {
    "root_cause" : [
      {
        "type" : "illegal_argument_exception",
        "reason" : "trying to convert a known date time formatter to a nanosecond one, wrong field used?"
      }
    ],
    "type" : "exception",
    "reason" : "Unable to rollup index [my-index-000001]",
    "caused_by" : {
      "type" : "exception",
      "reason" : "[my-index-000001/9DveCFw8RR67S2tul3Qt_w][[my-index-000001][0]] org.elasticsearch.action.support.broadcast.BroadcastShardOperationFailedException:",
      "caused_by" : {
        "type" : "illegal_argument_exception",
        "reason" : "trying to convert a known date time formatter to a nanosecond one, wrong field used?"
      }
    }
  },
  "status" : 500
}

This seems to be the case even if an epoch_millis format is used.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

^^ @talevy

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will take a look. thank you!

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I opened #70161

@jrodewig jrodewig requested a review from talevy March 8, 2021 13:45
@jrodewig jrodewig added :StorageEngine/Rollup Turn fine-grained time-based data into coarser-grained data >docs General docs changes v7.13.0 v8.0.0 labels Mar 8, 2021
@elasticmachine elasticmachine added Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) Team:Docs Meta label for docs team labels Mar 8, 2021
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-analytics-geo (Team:Analytics)

@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-docs (Team:Docs)

** A <<date,`date`>> or <<date_nanos,`date_nanos`>> timestamp field
** One or more <<number,numeric>> fields
** A <<date,`date`>> or <<date_nanos,`date_nanos`>> timestamp field.
** At least one <<number,numeric>>, `date`, or `date_nanos` field other than the
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we highlight what this additional field would be used for? as in, why it is required: metrics.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the suggestion. Added with 983eb07

Comment on lines +187 to +188
<<number,Numeric>>, <<date,`date`>>, or <<date_nanos,`date_nanos`>> field to
collect metrics for. `unsigned_long` fields are not supported.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will take a look. thank you!

Comment on lines +187 to +188
<<number,Numeric>>, <<date,`date`>>, or <<date_nanos,`date_nanos`>> field to
collect metrics for. `unsigned_long` fields are not supported.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I opened #70161

@talevy
Copy link
Contributor

talevy commented Mar 9, 2021

@jrodewig also spoke on #70161. I am not entirely sure if we should even support date or date_nanos as metrics. So maybe we should watch this space. it may change

1 similar comment
@talevy
Copy link
Contributor

talevy commented Mar 9, 2021

@jrodewig also spoke on #70161. I am not entirely sure if we should even support date or date_nanos as metrics. So maybe we should watch this space. it may change

@jrodewig
Copy link
Contributor Author

jrodewig commented Mar 9, 2021

Thanks @talevy. I'll go ahead and merge this in for now. We can easily remove docs for date and date_nanos metrics once #70161 is resolved.

@jrodewig jrodewig merged commit d68b140 into elastic:master Mar 9, 2021
@jrodewig jrodewig deleted the docs__clarify-supported-rollup-field-types branch March 9, 2021 17:16
@jloleysens
Copy link
Contributor

jloleysens commented Mar 10, 2021

Thanks for the heads up @jrodewig !

The added information with respect to the required "metric" field is very useful!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
>docs General docs changes :StorageEngine/Rollup Turn fine-grained time-based data into coarser-grained data Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) Team:Docs Meta label for docs team v7.13.0 v8.0.0-alpha1
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants