Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rollup action bug unable to rollup date_nanos metrics #70161

Closed
talevy opened this issue Mar 9, 2021 · 7 comments
Closed

Rollup action bug unable to rollup date_nanos metrics #70161

talevy opened this issue Mar 9, 2021 · 7 comments
Assignees
Labels
>bug :StorageEngine/Rollup Turn fine-grained time-based data into coarser-grained data Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo)

Comments

@talevy
Copy link
Contributor

talevy commented Mar 9, 2021

Rollup action is supposed to support date_nanos as a metric for rolling up. Instead, there is a field formatter conflict when resolving the field.

reproduction:

// Map a date_nanos field.
PUT my-index-000001
{
  "mappings": {
    "properties": {
      "my-date-nanos-field": {
        "type": "date_nanos" 
      }
    }
  }
}

// Index a document with the date_nanos field.
POST my-index-000001/_doc
{ 
  "@timestamp": "2099-11-15T13:12:00",
  "my-date-nanos-field": "2015-01-01T12:10:30.123456789Z"
}

// Attempt to roll up the index with the date_nanos field as a metric.
POST my-index-000001/_rollup/rollup-my-index-000001
{
  "groups": {
    "date_histogram": {
      "field": "@timestamp",
      "calendar_interval": "1d"
    }
  },
  "metrics": [
    {
      "field": "my-date-nanos-field",
      "metrics": [
        "min",
        "max"
      ]
    }
  ]
}

exception:

{
  "error" : {
    "root_cause" : [
      {
        "type" : "illegal_argument_exception",
        "reason" : "trying to convert a known date time formatter to a nanosecond one, wrong field used?"
      }
    ],
    "type" : "exception",
    "reason" : "Unable to rollup index [my-index-000001]",
    "caused_by" : {
      "type" : "exception",
      "reason" : "[my-index-000001/9DveCFw8RR67S2tul3Qt_w][[my-index-000001][0]] org.elasticsearch.action.support.broadcast.BroadcastShardOperationFailedException:",
      "caused_by" : {
        "type" : "illegal_argument_exception",
        "reason" : "trying to convert a known date time formatter to a nanosecond one, wrong field used?"
      }
    }
  },
  "status" : 500
}
@talevy talevy added >bug :StorageEngine/Rollup Turn fine-grained time-based data into coarser-grained data labels Mar 9, 2021
@elasticmachine elasticmachine added the Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) label Mar 9, 2021
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-analytics-geo (Team:Analytics)

@jimczi
Copy link
Contributor

jimczi commented Mar 9, 2021

Rollup action is supposed to support date_nanos as a metric for rolling up.

I don't see why ? We use doubles for metrics so that doesn't work out of the box. Can we disable date_nanos as metrics instead ? I am not even sure that handling date is required.

@talevy
Copy link
Contributor Author

talevy commented Mar 9, 2021

I believe that is how things were? I see no reason to support date or date_nanos as metrics either. I can update accordingly

@jimczi jimczi added >docs General docs changes and removed >bug labels Mar 9, 2021
@elasticmachine elasticmachine added the Team:Docs Meta label for docs team label Mar 9, 2021
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-docs (Team:Docs)

@jimczi jimczi added >bug and removed >docs General docs changes labels Mar 9, 2021
@elasticmachine elasticmachine removed the Team:Docs Meta label for docs team label Mar 9, 2021
@not-napoleon
Copy link
Member

@jimczi - Does this still need work? Do we just need to update the docs? Can we close it?

@jrodewig
Copy link
Contributor

jrodewig commented Jun 9, 2021

@jimczi - Does this still need work? Do we just need to update the docs? Can we close it?

I removed the docs for the refactored rollup API + ILM action with #70885 as it seemed unlikely that we'll move forward with that functionality. I don't think updating the docs is likely the fix here.

@csoulios csoulios self-assigned this Apr 15, 2022
@csoulios
Copy link
Contributor

In the current implementation of downsampling for time-series indices, field type date_nanos does not support the time_series_metric mapping parameter. So, fields of type date_nanos cannot be a metric field.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
>bug :StorageEngine/Rollup Turn fine-grained time-based data into coarser-grained data Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo)
Projects
None yet
Development

No branches or pull requests

6 participants