date_histogram aggregation fails for date_nanos and noon utc timezone #39107

flash1293 · 2019-02-19T11:09:40Z

When trying to apply a date_histogram aggregation with a timezone which isn't utc (tested with Europe/Berlin) on a date_nanos field, elasticsearch responds with a unsupported_operation_exception.

Tested version: 7.0 snapshot

Steps to reproduce:

Create an index like described in the docs: https://www.elastic.co/guide/en/elasticsearch/reference/7.0/date_nanos.html
Execute the following query:

GET my_index/_search
{
  "version": true,
  "size": 500,
  "sort": [{ "date": { "order": "desc", "unmapped_type": "boolean" } }],
  "_source": { "excludes": [] },
  "aggs": {
    "2": {
      "date_histogram": {
        "field": "date",
        "interval": "30s",
        "time_zone": "Europe/Berlin",
        "min_doc_count": 1
      }
    }
  },
  "docvalue_fields": [{ "field": "date", "format": "date_time" }],
  "query": {
    "bool": {
      "must": [
        {
          "range": {
            "date": { "gte": 1050572204889, "lte": 1550573104889, "format": "epoch_millis" }
          }
        }
      ]
    }
  },
  "timeout": "30000ms"
}

expected response: status code 200 and hits in the payload
actual response:

{
  "error": {
    "root_cause": [
      {
        "type": "unsupported_operation_exception",
        "reason": null
      }
    ],
    "type": "search_phase_execution_exception",
    "reason": "all shards failed",
    "phase": "query",
    "grouped": true,
    "failed_shards": [
      {
        "shard": 0,
        "index": "my_index",
        "node": "mTfaCthlT8uvuCLbfUtPCQ",
        "reason": {
          "type": "unsupported_operation_exception",
          "reason": null
        }
      }
    ],
    "caused_by": {
      "type": "unsupported_operation_exception",
      "reason": null,
      "caused_by": {
        "type": "unsupported_operation_exception",
        "reason": null
      }
    }
  },
  "status": 500
}

The same query with "time_zone": "UTC", works correctly.

The text was updated successfully, but these errors were encountered:

elasticmachine · 2019-02-19T11:55:36Z

Pinging @elastic/es-search

colings86 · 2019-02-19T11:56:40Z

@flash1293 can you look int eh ES server logs and paste the stack trace for the UnsupportedOperationException please?

flash1293 · 2019-02-19T12:28:27Z

@colings86

   │ info [o.e.a.s.TransportSearchAction] [Johanness-MacBook-Pro.local] [my_index][0], node[JkZl0SGpRwONG1FLu1KAew], [P], s[STARTED], a[id=x7r15bL1Rtq2dRd0LR8-bg]: Failed to execute [SearchRequest{searchType=QUERY_THEN_FETCH, indices=[my_index], indicesOptions=IndicesOptions[ignore_unavailable=false, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, allow_aliases_to_multiple_indices=true, forbid_closed_indices=true, ignore_aliases=false, ignore_throttled=true], types=[], routing='null', preference='null', requestCache=null, scroll=null, maxConcurrentShardRequests=0, batchedReduceSize=512, preFilterShardSize=128, allowPartialSearchResults=true, localClusterAlias=null, getOrCreateAbsoluteStartMillis=-1, ccsMinimizeRoundtrips=true, source={"size":500,"timeout":"30000ms","query":{"bool":{"must":[{"range":{"date":{"from":1050572204889,"to":1550573104889,"include_lower":true,"include_upper":true,"format":"epoch_millis","boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},"version":true,"_source":{"includes":[],"excludes":[]},"docvalue_fields":[{"field":"date","format":"date_time"}],"sort":[{"date":{"order":"desc","unmapped_type":"boolean"}}],"aggregations":{"2":{"date_histogram":{"field":"date","time_zone":"Europe/Berlin","interval":"30s","offset":0,"order":{"_key":"asc"},"keyed":false,"min_doc_count":1}}}}}]
   │      org.elasticsearch.transport.RemoteTransportException: [Johanness-MacBook-Pro.local][127.0.0.1:9300][indices:data/read/search[phase/query]]
   │      Caused by: java.lang.UnsupportedOperationException
   │      	at org.elasticsearch.index.fielddata.AbstractSortedNumericDocValues.nextDoc(AbstractSortedNumericDocValues.java:42) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.search.aggregations.bucket.histogram.DateHistogramAggregationBuilder.rewriteTimeZone(DateHistogramAggregationBuilder.java:389) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.search.aggregations.bucket.histogram.DateHistogramAggregationBuilder.innerBuild(DateHistogramAggregationBuilder.java:444) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.search.aggregations.support.ValuesSourceAggregationBuilder.doBuild(ValuesSourceAggregationBuilder.java:315) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.search.aggregations.support.ValuesSourceAggregationBuilder.doBuild(ValuesSourceAggregationBuilder.java:39) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.search.aggregations.AbstractAggregationBuilder.build(AbstractAggregationBuilder.java:139) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.search.aggregations.AggregatorFactories$Builder.build(AggregatorFactories.java:333) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.search.SearchService.parseSource(SearchService.java:800) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.search.SearchService.createContext(SearchService.java:607) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:582) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:385) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.search.SearchService.access$100(SearchService.java:123) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:357) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:353) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.search.SearchService$4.doRun(SearchService.java:1068) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
   │      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
   │      	at java.lang.Thread.run(Thread.java:834) [?:?]
   │ info [o.e.a.s.TransportSearchAction] [Johanness-MacBook-Pro.local] All shards failed for phase: [query]
   │      org.elasticsearch.ElasticsearchException$1: null
   │      	at org.elasticsearch.ElasticsearchException.guessRootCauses(ElasticsearchException.java:639) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:137) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:259) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:105) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.action.search.InitialSearchPhase.access$200(InitialSearchPhase.java:50) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.action.search.InitialSearchPhase$2.onFailure(InitialSearchPhase.java:273) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:441) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1118) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1230) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1204) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:60) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.action.support.ChannelActionListener.onFailure(ChannelActionListener.java:56) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.search.SearchService$2.onFailure(SearchService.java:365) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:359) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:353) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.search.SearchService$4.doRun(SearchService.java:1068) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
   │      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
   │      	at java.lang.Thread.run(Thread.java:834) [?:?]
   │      Caused by: java.lang.UnsupportedOperationException
   │      	at org.elasticsearch.index.fielddata.AbstractSortedNumericDocValues.nextDoc(AbstractSortedNumericDocValues.java:42) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.search.aggregations.bucket.histogram.DateHistogramAggregationBuilder.rewriteTimeZone(DateHistogramAggregationBuilder.java:389) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.search.aggregations.bucket.histogram.DateHistogramAggregationBuilder.innerBuild(DateHistogramAggregationBuilder.java:444) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.search.aggregations.support.ValuesSourceAggregationBuilder.doBuild(ValuesSourceAggregationBuilder.java:315) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.search.aggregations.support.ValuesSourceAggregationBuilder.doBuild(ValuesSourceAggregationBuilder.java:39) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.search.aggregations.AbstractAggregationBuilder.build(AbstractAggregationBuilder.java:139) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.search.aggregations.AggregatorFactories$Builder.build(AggregatorFactories.java:333) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.search.SearchService.parseSource(SearchService.java:800) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.search.SearchService.createContext(SearchService.java:607) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:582) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:385) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.search.SearchService.access$100(SearchService.java:123) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:357) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	... 9 more
   │ info [r.suppressed] [Johanness-MacBook-Pro.local] path: /my_index/_search, params: {pretty=, index=my_index}
   │      org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed
   │      	at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:296) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:139) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:259) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:105) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.action.search.InitialSearchPhase.access$200(InitialSearchPhase.java:50) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.action.search.InitialSearchPhase$2.onFailure(InitialSearchPhase.java:273) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:441) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1118) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1230) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1204) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:60) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.action.support.ChannelActionListener.onFailure(ChannelActionListener.java:56) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.search.SearchService$2.onFailure(SearchService.java:365) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:359) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:353) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.search.SearchService$4.doRun(SearchService.java:1068) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
   │      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
   │      	at java.lang.Thread.run(Thread.java:834) [?:?]
   │      Caused by: org.elasticsearch.ElasticsearchException$1
   │      	at org.elasticsearch.ElasticsearchException.guessRootCauses(ElasticsearchException.java:639) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:137) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	... 23 more
   │      Caused by: java.lang.UnsupportedOperationException
   │      	at org.elasticsearch.index.fielddata.AbstractSortedNumericDocValues.nextDoc(AbstractSortedNumericDocValues.java:42) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.search.aggregations.bucket.histogram.DateHistogramAggregationBuilder.rewriteTimeZone(DateHistogramAggregationBuilder.java:389) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.search.aggregations.bucket.histogram.DateHistogramAggregationBuilder.innerBuild(DateHistogramAggregationBuilder.java:444) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.search.aggregations.support.ValuesSourceAggregationBuilder.doBuild(ValuesSourceAggregationBuilder.java:315) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.search.aggregations.support.ValuesSourceAggregationBuilder.doBuild(ValuesSourceAggregationBuilder.java:39) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.search.aggregations.AbstractAggregationBuilder.build(AbstractAggregationBuilder.java:139) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.search.aggregations.AggregatorFactories$Builder.build(AggregatorFactories.java:333) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.search.SearchService.parseSource(SearchService.java:800) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.search.SearchService.createContext(SearchService.java:607) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:582) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:385) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.search.SearchService.access$100(SearchService.java:123) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:357) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	... 9 more

colings86 · 2019-02-19T13:47:48Z

@jpountz @polyfractal could one of you take a look at this one?

elasticmachine · 2019-02-19T13:47:56Z

Pinging @elastic/es-analytics-geo

jpountz · 2019-02-19T14:18:00Z

So we were assuming that date fields only produced direct doc-value iterators, which is no longer the case with date_nanos due to the fact that they need to divide all values by 1M. In my opinion the right fix would be to extend nextDoc to delegate to the wrapped doc-value iterator (the one that stores nanos).

tsullivan · 2019-07-16T17:02:27Z

Hi, I am seeing a bug very similar to this pop up: elastic/kibana#40699

This query:

GET /my_index/_search
{
  "aggs": {
    "2": {
      "date_histogram": {
        "field": "date",
        "fixed_interval": "30d",
        "min_doc_count": 1,
        "time_zone": "America/Phoenix"
      }
    }
  }
}

Is causing a failure with:

{
  "error": {
    "caused_by": {
      "caused_by": {
        "reason": "date[1967-10-29T08:00:30Z] is before the epoch in 1970 and cannot be stored in nanosecond resolution",
        "type": "illegal_argument_exception"
      },
      "reason": "date[1967-10-29T08:00:30Z] is before the epoch in 1970 and cannot be stored in nanosecond resolution",
      "type": "illegal_argument_exception"
    },
    "failed_shards": [
      {
        "index": "my_index",
        "node": "nMrM4ZZTSh68Xg5kWsl8iA",
        "reason": {
          "reason": "date[1967-10-29T08:00:30Z] is before the epoch in 1970 and cannot be stored in nanosecond resolution",
          "type": "illegal_argument_exception"
        },
        "shard": 0
      }
    ],
    "grouped": true,
    "phase": "query",
    "reason": "all shards failed",
    "root_cause": [
      {
        "reason": "date[1967-10-29T08:00:30Z] is before the epoch in 1970 and cannot be stored in nanosecond resolution",
        "type": "illegal_argument_exception"
      }
    ],
    "type": "search_phase_execution_exception"
  },
  "status": 400
}

The same query without the timezone field in the date_histogram seems to work.

I'm re-opening this for visibility, but if that's wrong, please offer the corrections. Thanks!

kertal · 2019-07-18T18:44:32Z

We also had problems reviewing another PR, because also South Africa Timezone caused problems:
elastic/kibana#41353

romain-chanu · 2020-04-28T08:48:03Z

@tsullivan @kertal I believe the errors mentioned in #39107 (comment) and #39107 (comment) have been resolved by #52016. Please confirm my assessment. Thank you.

salvatore-campagna · 2022-03-21T09:17:51Z

I added the YAML test in PR #85149 to confirm we don't see the issue anymore.

flash1293 mentioned this issue Feb 19, 2019

Support date_nanos in Kibana elastic/kibana#31424

Closed

4 tasks

colings86 added the :Search Foundations/Mapping Index mappings, including merging and defining field types label Feb 19, 2019

colings86 added the feedback_needed label Feb 19, 2019

colings86 added :Analytics/Aggregations Aggregations and removed feedback_needed labels Feb 19, 2019

iverase self-assigned this Feb 20, 2019

iverase mentioned this issue Feb 20, 2019

Extend nextDoc to delegate to the wrapped doc-value iterator for date_nanos #39176

Merged

iverase closed this as completed in #39176 Feb 21, 2019

tsullivan reopened this Jul 16, 2019

tsullivan mentioned this issue Jul 16, 2019

A date field with date_nanos format can not be used as an index pattern time field. elastic/kibana#40699

Closed

kertal mentioned this issue Sep 18, 2019

Add numeric_type option for correct sort order on mixed date and date_nanos fields elastic/kibana#44212

Merged

2 tasks

jtibshirani added >bug and removed :Search Foundations/Mapping Index mappings, including merging and defining field types labels Nov 27, 2019

rjernst added the Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) label May 4, 2020

fixmebot bot referenced this issue in VectorXz/elasticsearch Apr 22, 2021

Create TestFixMe.md

a9fae03

fixmebot bot referenced this issue in VectorXz/elasticsearch May 28, 2021

Create Helloworld.md

1398a04

fixmebot bot referenced this issue in VectorXz/elasticsearch Aug 4, 2021

Update Helloworld.md

f68abab

wchaparro unassigned iverase Jan 11, 2022

salvatore-campagna mentioned this issue Mar 21, 2022

test: date_histogram with time zone on date_nanos field #85149

Merged

salvatore-campagna self-assigned this Mar 21, 2022

salvatore-campagna closed this as completed Apr 20, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

date_histogram aggregation fails for date_nanos and noon utc timezone #39107

date_histogram aggregation fails for date_nanos and noon utc timezone #39107

flash1293 commented Feb 19, 2019

elasticmachine commented Feb 19, 2019

colings86 commented Feb 19, 2019

flash1293 commented Feb 19, 2019

colings86 commented Feb 19, 2019

elasticmachine commented Feb 19, 2019

jpountz commented Feb 19, 2019

tsullivan commented Jul 16, 2019

kertal commented Jul 18, 2019

romain-chanu commented Apr 28, 2020

salvatore-campagna commented Mar 21, 2022

date_histogram aggregation fails for date_nanos and noon utc timezone #39107

date_histogram aggregation fails for date_nanos and noon utc timezone #39107

Comments

flash1293 commented Feb 19, 2019

elasticmachine commented Feb 19, 2019

colings86 commented Feb 19, 2019

flash1293 commented Feb 19, 2019

colings86 commented Feb 19, 2019

elasticmachine commented Feb 19, 2019

jpountz commented Feb 19, 2019

tsullivan commented Jul 16, 2019

kertal commented Jul 18, 2019

romain-chanu commented Apr 28, 2020

salvatore-campagna commented Mar 21, 2022