AmbiguousTimeError using drop on DatetimeIndex with DST change

[JrtPec]

Code Sample, a copy-pastable example if possible

import pandas as pd
start = pd.Timestamp('201710290100', tz='Europe/Brussels')
end = pd.Timestamp('201710290300', tz='Europe/Brussels')
index = pd.date_range(start=start, end=end, freq='15min')

index.drop(index[0])

AmbiguousTimeError: Cannot infer dst time from Timestamp('2017-10-29 02:00:00'), try using the 'ambiguous' argument

Problem description

When dropping from a localized DateTimeIndex, behind the scenes the index is converted to UTC, the drop is performed, and then it is converted back to the original timezone. (This is what I make of the traceback).
This second conversion is ambiguous because the original index contains a DST change.

Expected Output

Output of index.drop(index[0])

DatetimeIndex(['2017-10-29 01:15:00+02:00', '2017-10-29 01:30:00+02:00',
'2017-10-29 01:45:00+02:00', '2017-10-29 02:00:00+02:00',
'2017-10-29 02:15:00+02:00', '2017-10-29 02:30:00+02:00',
'2017-10-29 02:45:00+02:00', '2017-10-29 02:00:00+01:00',
'2017-10-29 02:15:00+01:00', '2017-10-29 02:30:00+01:00',
'2017-10-29 02:45:00+01:00', '2017-10-29 03:00:00+01:00'],
dtype='datetime64[ns, Europe/Brussels]', freq='15T')

[gfyoung]

@JrtPec : Thanks for reporting this! This indeed does look a little strange. Feel free to investigate and put up a PR for this.

For reference, do you mind posting the entire stacktrace?

[jreback]

we don't have the ambiguous parameter for .drop (or .delete) which it is calling, but wouldn't be hard to add and pass it thru.

as a work-around

In [16]: index.tz_convert('UTC').drop(index[0].tz_convert('UTC')).tz_convert(index.tz)
Out[16]: 
DatetimeIndex(['2017-10-29 01:15:00+02:00', '2017-10-29 01:30:00+02:00', '2017-10-29 01:45:00+02:00', '2017-10-29 02:00:00+02:00', '2017-10-29 02:15:00+02:00', '2017-10-29 02:30:00+02:00',
               '2017-10-29 02:45:00+02:00', '2017-10-29 02:00:00+01:00', '2017-10-29 02:15:00+01:00', '2017-10-29 02:30:00+01:00', '2017-10-29 02:45:00+01:00', '2017-10-29 03:00:00+01:00'],
              dtype='datetime64[ns, Europe/Brussels]', freq='15T')

[jreback]

pull requests welcome!

[JrtPec]

Full traceback:

AmbiguousTimeError                        Traceback (most recent call last)
<ipython-input-19-7d5e95692da7> in <module>()
----> 1 index.drop(index[0])

/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/pandas/core/indexes/base.py in drop(self, labels, errors)
   3741                                  labels[mask])
   3742             indexer = indexer[~mask]
-> 3743         return self.delete(indexer)
   3744 
   3745     @Appender(base._shared_docs['unique'] % _index_doc_kwargs)

/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/pandas/core/indexes/datetimes.py in delete(self, loc)
   1773         if self.tz is not None:
   1774             new_dates = libts.tz_convert(new_dates, 'UTC', self.tz)
-> 1775         return DatetimeIndex(new_dates, name=self.name, freq=freq, tz=self.tz)
   1776 
   1777     def tz_convert(self, tz):

/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/pandas/util/_decorators.py in wrapper(*args, **kwargs)
    116                 else:
    117                     kwargs[new_arg_name] = new_arg_value
--> 118             return func(*args, **kwargs)
    119         return wrapper
    120     return _deprecate_kwarg

/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/pandas/core/indexes/datetimes.py in __new__(cls, data, freq, start, end, periods, copy, name, tz, verify_integrity, normalize, closed, ambiguous, dtype, **kwargs)
    388                     ints = subarr.view('i8')
    389                     subarr = libts.tz_localize_to_utc(ints, tz,
--> 390                                                       ambiguous=ambiguous)
    391                 subarr = subarr.view(_NS_DTYPE)
    392 

pandas/_libs/tslib.pyx in pandas._libs.tslib.tz_localize_to_utc()

AmbiguousTimeError: Cannot infer dst time from Timestamp('2017-10-29 02:00:00'), try using the 'ambiguous' argument

[matthiasLevy]

Hy,
I face the same error with ceil('T') method... any workaround seen, except converting to UTC if freq is below hour ? (Not working for 'D')
ts.tz_convert('UTC').ceil('T').tz_convert(ts.tz)
Thanks in advance...
Matt

[mroeschke]

This looks fixed on master if anyone wants to put up a test:

In [1]: pd.__version__
Out[1]: '0.24.0.dev0+371.g0b7a08b70'

In [2]: start = pd.Timestamp('201710290100', tz='Europe/Brussels')
   ...: end = pd.Timestamp('201710290300', tz='Europe/Brussels')
   ...: index = pd.date_range(start=start, end=end, freq='15min')
   ...:
   ...:

In [3]: index
Out[3]:
DatetimeIndex(['2017-10-29 01:00:00+02:00', '2017-10-29 01:15:00+02:00',
               '2017-10-29 01:30:00+02:00', '2017-10-29 01:45:00+02:00',
               '2017-10-29 02:00:00+02:00', '2017-10-29 02:15:00+02:00',
               '2017-10-29 02:30:00+02:00', '2017-10-29 02:45:00+02:00',
               '2017-10-29 02:00:00+01:00', '2017-10-29 02:15:00+01:00',
               '2017-10-29 02:30:00+01:00', '2017-10-29 02:45:00+01:00',
               '2017-10-29 03:00:00+01:00'],
              dtype='datetime64[ns, Europe/Brussels]', freq='15T')

In [4]: index.drop(index[0])
   ...:
Out[4]:
DatetimeIndex(['2017-10-29 01:15:00+02:00', '2017-10-29 01:30:00+02:00',
               '2017-10-29 01:45:00+02:00', '2017-10-29 02:00:00+02:00',
               '2017-10-29 02:15:00+02:00', '2017-10-29 02:30:00+02:00',
               '2017-10-29 02:45:00+02:00', '2017-10-29 02:00:00+01:00',
               '2017-10-29 02:15:00+01:00', '2017-10-29 02:30:00+01:00',
               '2017-10-29 02:45:00+01:00', '2017-10-29 03:00:00+01:00'],
              dtype='datetime64[ns, Europe/Brussels]', freq='15T')