bug in tz_localize when summer/winter time change

[gbakalian]

Works with datetime.datetime, does not work with pandas.tslib.Timestamp.
Encounter the issue when trying to resample hourly a series.

import pandas as pd
from datetime import datetime as dt
import pytz


tz = 'Europe/Berlin'
d = dt(2014,10,26,02,00)
tz = pytz.timezone(tz)
tz.localize(d)

x = pd.Series(0., [d]).index[0]
x.tz_localize(tz)

[jreback]

try this using 0.15.0rc1 http://pandas.pydata.org/

their is now an ambiguous option to do this (which has several options for it)

In [16]: x.tz_localize(tz,ambiguous=True)
Out[16]: Timestamp('2014-10-26 02:00:00+0200', tz='Europe/Berlin')

In [17]: x.tz_localize(tz,ambiguous='NaT')
Out[17]: NaT

In [18]: x.tz_localize(tz,ambiguous=False)
Out[18]: Timestamp('2014-10-26 02:00:00+0100', tz='Europe/Berlin')

cc @rockg
cc @ischwabacher

[gbakalian]

I have updated the version - your example work fine.
But i still have an issue when trying to resample:

import pandas as pd
from datetime import datetime as dt
import pytz


tz = 'Europe/Berlin'
d = dt(2014,10,25,03,00)
tz = pytz.timezone(tz)
dates = pd.date_range(tz.localize(d), periods=24, freq='H')
pd.Series(5,dates).resample('H')

[gbakalian]

How can I 'set' ambiguous equal True in my previous example ?

[gbakalian]

Also:

dates = pd.date_range(dt(2014,10,25,03,00), periods=24, freq='H')
ts = pd.Series(5,dates)
ts.tz_localize(tz, ambiguous=True)

[jreback]

@gbakalian out of curiosity why are you resampling over a DST transition with hourly data?

[gbakalian]

LOL indeed can sounds weird !

In my database I received several curves with different frequencies and different timezones from different weather providers. I need to get them at the same time zone and resample them hourly.

but it can happen that among all the curves I retrieve some which are already hourly.

if you prefer the following example also crash :

import pandas as pd
from datetime import datetime as dt
import pytz


tz = 'Europe/Berlin'
d = dt(2014,10,25,04,00)
tz = pytz.timezone(tz)
dates = pd.date_range(tz.localize(d), periods=12, freq='2H')
pd.Series(5,dates).resample('H')

[jreback]

thanks! that sounds like a good use case.

if you'd like to dig-in would welcome a PR to fix! It prob just not passing the argument thru

[gbakalian]

May I ask you a question:

The ambiguous parameters is set in which element ? - and is it kept ? (sorry if I am asking dumb questions)

because in the previous example I do not tell him anywhere that I want ambiguous=True, as I made the tz localisation using pytz and not tz_localize on pd.Series. So maybe we should be able to pass ambiguous through resample ?

dates = pd.date_range(d, periods=12, freq='2H')
ts = pd.Series(5, dates)
ts.tz_localize(tz, ambiguous=True)

this does not work neither so I guess in tz_localize_to_utc in the tslib.pyx file the ambiguous parameters is not passed or read correctly ?

Also I guess the 4 tz_localize between lines 418-428 in the index.py file should have ambiguous=ambiguous.

[jreback]

that sounds about right. here's some help for doing a pull-request: https://github.com/pydata/pandas/wiki

this prob has several aspects which are not handling this: date_range (manifesting in _generate) as you have seen, and resample so create appropriate tests (you can use the examples above).

have it fail, then fix.