Illegal instant due to time zone offset transition

[greenboxal]

When issuing a date_histogram with America/Sao_Paulo as timezone, the query fails with the following stack trace:

Stack Trace:

org.elasticsearch.common.joda.time.IllegalInstantException: Illegal instant due to time zone offset transition (daylight savings time 'gap'): 2014-10-19T00:00:00.000 (America/Sao_Paulo)
        at org.elasticsearch.common.joda.time.DateTimeZone.convertLocalToUTC(DateTimeZone.java:1025)
        at org.elasticsearch.common.rounding.TimeZoneRounding$TimeTimeZoneRoundingFloor.nextRoundingValue(TimeZoneRounding.java:175)
        at org.elasticsearch.search.aggregations.bucket.histogram.InternalHistogram.reduce(InternalHistogram.java:323)
        at org.elasticsearch.search.aggregations.InternalAggregations.reduce(InternalAggregations.java:140)
        at org.elasticsearch.search.controller.SearchPhaseController.merge(SearchPhaseController.java:407)
        at org.elasticsearch.action.search.type.TransportSearchCountAction$AsyncAction.moveToSecondPhase(TransportSearchCountAction.java:77)
        at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.innerMoveToSecondPhase(TransportSearchTypeAction.java:397)
        at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.onFirstPhaseResult(TransportSearchTypeAction.java:198)
        at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$1.onResult(TransportSearchTypeAction.java:174)
        at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$1.onResult(TransportSearchTypeAction.java:171)
        at org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:568)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)

Please note that this is a very critical bug, we're already running on master due to another ElasticSearch bug(#9491).

[rjernst]

@cbuescher can you take a look?

@greenboxal You should absolutely never be running on master (or 1.x). These are based on snapshots of Lucene, and those snapshots use as yet unreleased versions of Lucene's index format. There are zero backcompat guarantees when you do this.

Only use officially released versions of Elasticsearch (which are based on officially released version of Lucene).

[greenboxal]

I can't use a stable release as #9491 fix didn't land in any released version(at least when we started using master).
Without that fix when the DST went off all date histograms(interval=1d) broke showing one day with 24 hours and another day on the same date with 1 hour of duration.

[cbuescher]

We started to use Joda times utility method for local to UTC time conversions in the Rounding classes. It seems that doing this with the "strict" option set to true makes Joda time to throw exceptions when using this conversion on local time stamps that would fall in the DST gaps when DST is switched on. I could reproduce this on 1.x, but it also seems to affect the 1.4 and master branch where we now use DateTimeTone#convertLocalToUTC().
I think it is save to switch the "strict" option when using this method to "false" to prevent it from throwing the exception. I will open a PR with tests and the changes in the Rounding classes.

[nullbuddy1243]

@cbuescher Do you think that the commit referenced above would handle this exception as well?

org.elasticsearch.index.mapper.MapperParsingException: failed to parse [day_timeinterval]
        at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:416)
        at org.elasticsearch.index.mapper.object.ObjectMapper.serializeValue(ObjectMapper.java:709)
        at org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:500)
        at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:542)
        at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:491)
        at org.elasticsearch.index.shard.service.InternalIndexShard.prepareCreate(InternalIndexShard.java:392)
        at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:444)
        at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:150)
        at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:512)
        at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:419)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.mapper.MapperParsingException: failed to parse date field [2015-03-08 02:15:00 EST], tried both date format [yyyy-MM-dd HH:mm:ss z], and timestamp number with locale []
        at org.elasticsearch.index.mapper.core.DateFieldMapper.parseStringValue(DateFieldMapper.java:621)
        at org.elasticsearch.index.mapper.core.DateFieldMapper.innerParseCreateField(DateFieldMapper.java:549)
        at org.elasticsearch.index.mapper.core.NumberFieldMapper.parseCreateField(NumberFieldMapper.java:235)
        at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:406)
        ... 12 more
Caused by: org.elasticsearch.common.joda.time.IllegalInstantException: Cannot parse "2015-03-08 02:15:00 EST": Illegal instant due to time zone offset transition (America/New_York)
        at org.elasticsearch.common.joda.time.format.DateTimeParserBucket.computeMillis(DateTimeParserBucket.java:390)
        at org.elasticsearch.common.joda.time.format.DateTimeFormatter.parseMillis(DateTimeFormatter.java:749)
        at org.elasticsearch.index.mapper.core.DateFieldMapper.parseStringValue(DateFieldMapper.java:615)

I know its pretty similar, but I want to make sure it would be solved by upgrading ES before I go about doing that.

[cbuescher]

@deusofnull Thanks, from a first glance I would not say that this is related. IllegaInstantException is used in various places in joda time. This issue concerns a case where it is raised when converting time zones in DateTimeZone that fall into the DST gap. Your issue seems to be related to parsing a date that falls into the DST gap, thats why they look similar, but the rest of the stack trace is different. It would be great if you could open a separate issue and report what you are doing and which version of ES you are running there.

[nullbuddy1243]

@cbuescher Ill bring up a new issue then! Thanks!

What is strange is that I was parsing 90 days back of timestamps and I got this error for all of them. All 15k of the timestamps. If I'm correct, DST started about a month ago... So why did those dates still elicit the error?