vectorised setting of timestamp columns fails with python datetime and numpy datetime64

[seanv507]

import pandas as pd
import numpy as np
import datetime as dt
z=dt.date(2010,11,1)
zs=[z+dt.timedelta(days=r) for r in range(5)]
df=pd.DataFrame({'obj':zs, 'b':pd.Timestamp('2010-10-01'),'c':pd.Timestamp('2010-10-01')})
df.dtypes

#df.loc[0:2,'c']=dt.date(2010,10,12) # causes error: long() argument must be a string or a number, not 'datetime.date
df.loc[0:2,'c']=np.datetime64('2010-10-12') # sets to 1970...
df.at[4,'c']=np.datetime64('2010-10-12') # works
df.loc[0:2,'obj']=np.datetime64('2010-10-12') #works

df.loc[0:2,'obj']=dt.date(2010,10,12)

df

ind	b	c	obj
0	2010-10-01	1970-01-01	2010-10-12
1	2010-10-01	1970-01-01	2010-10-12
2	2010-10-01	1970-01-01	2010-10-12
3	2010-10-01	2010-10-01	2010-11-04
4	2010-10-01	2010-10-12	2010-11-05

I am using Pandas 0.16.2

[jreback]

datetime.date are normally not supported for most datetime operations, simply use datetime.datetime and this will work. datetime.date are stored as object dtypes and thus are not very efficient. If you want to use a pure date object, the you might find Period objects useful.

I will mark this as a bug, and if you'd like to dig-in you are welcome. This fix is actually pretty straightforward.

[seanv507]

@jreback, you understood there is also an issue with numpy datetime64 'dates' not just python datetime.dates. (ie I get 1970 when I use a numpy datetime64 date)

[jreback]

@seanv507 you must have an older version of pandas/numpy. In current this works with np.datetime (see issue #9516), which is in 0.16.0.

In [13]: df.loc[0:2,'c']=np.datetime64('2010-10-12') # sets to 1970...

In [14]: df
Out[14]: 
           b                             c         obj
0 2010-10-01 1970-01-01 00:00:00.000014894  2010-11-01
1 2010-10-01 1970-01-01 00:00:00.000014894  2010-11-02
2 2010-10-01 1970-01-01 00:00:00.000014894  2010-11-03
3 2010-10-01 2010-10-01 00:00:00.000000000  2010-11-04
4 2010-10-01 2010-10-01 00:00:00.000000000  2010-11-05

[jorisvandenbossche]

@jreback I can confirm this also with pandas 0.16.2 / numpy 1.9.2

In [36]: np.__version__
Out[36]: '1.9.2'

In [37]: pd.__version__
Out[37]: '0.16.2'

In [38]: df.loc[0:2,'c'] = np.datetime64('2010-10-12')

In [39]: df
Out[39]:
           b                             c         obj
0 2010-10-01 1970-01-01 00:00:00.000014894  2010-11-01
1 2010-10-01 1970-01-01 00:00:00.000014894  2010-11-02
2 2010-10-01 1970-01-01 00:00:00.000014894  2010-11-03
3 2010-10-01 2010-10-01 00:00:00.000000000  2010-11-04
4 2010-10-01 2010-10-01 00:00:00.000000000  2010-11-05

[seanv507]

which is the version i am using as I mentioned in my original bug report
(forgot to state the numpy version 1.9.2)

On Tue, Jun 23, 2015 at 12:52 PM, Joris Van den Bossche <
notifications@github.com> wrote:

@jreback https://github.com/jreback I can confirm this also with pandas
0.16.2 / numpy 1.9.2

In [36]: np.version
Out[36]: '1.9.2'

In [37]: pd.version
Out[37]: '0.16.2'

In [38]: df.loc[0:2,'c'] = np.datetime64('2010-10-12')

In [39]: df
Out[39]:
b c obj
0 2010-10-01 1970-01-01 00:00:00.000014894 2010-11-01
1 2010-10-01 1970-01-01 00:00:00.000014894 2010-11-02
2 2010-10-01 1970-01-01 00:00:00.000014894 2010-11-03
3 2010-10-01 2010-10-01 00:00:00.000000000 2010-11-04
4 2010-10-01 2010-10-01 00:00:00.000000000 2010-11-05

—
Reply to this email directly or view it on GitHub
#10408 (comment).

[jreback]

Ahh, ok, seems that the test was insufficient; e.g. we are testing the equivalent of the [21], you are doing [22]

In [21]: np.datetime64(Timestamp('2010-10-12'))
Out[21]: numpy.datetime64('2010-10-11T20:00:00.000000-0400')

In [22]: np.datetime64('2010-10-12')
Out[22]: numpy.datetime64('2010-10-12')

ok, easy enough prob to fix, want to take a crack at it?