vectorised setting of timestamp columns fails with python datetime and numpy datetime64

[seanv507]

import pandas as pd
import numpy as np
import datetime as dt
z=dt.date(2010,11,1)
zs=[z+dt.timedelta(days=r) for r in range(5)]
df=pd.DataFrame({'obj':zs, 'b':pd.Timestamp('2010-10-01'),'c':pd.Timestamp('2010-10-01')})
df.dtypes

#df.loc[0:2,'c']=dt.date(2010,10,12) # causes error: long() argument must be a string or a number, not 'datetime.date
df.loc[0:2,'c']=np.datetime64('2010-10-12') # sets to 1970...
df.at[4,'c']=np.datetime64('2010-10-12') # works
df.loc[0:2,'obj']=np.datetime64('2010-10-12') #works

df.loc[0:2,'obj']=dt.date(2010,10,12)

df

ind	b	c	obj
0	2010-10-01	1970-01-01	2010-10-12
1	2010-10-01	1970-01-01	2010-10-12
2	2010-10-01	1970-01-01	2010-10-12
3	2010-10-01	2010-10-01	2010-11-04
4	2010-10-01	2010-10-12	2010-11-05

I am using Pandas 0.16.2

[jreback]

datetime.date are normally not supported for most datetime operations, simply use datetime.datetime and this will work. datetime.date are stored as object dtypes and thus are not very efficient. If you want to use a pure date object, the you might find Period objects useful.

I will mark this as a bug, and if you'd like to dig-in you are welcome. This fix is actually pretty straightforward.

[seanv507]

@jreback, you understood there is also an issue with numpy datetime64 'dates' not just python datetime.dates. (ie I get 1970 when I use a numpy datetime64 date)

[jreback]

@seanv507 you must have an older version of pandas/numpy. In current this works with np.datetime (see issue #9516), which is in 0.16.0.

In [13]: df.loc[0:2,'c']=np.datetime64('2010-10-12') # sets to 1970...

In [14]: df
Out[14]: 
           b                             c         obj
0 2010-10-01 1970-01-01 00:00:00.000014894  2010-11-01
1 2010-10-01 1970-01-01 00:00:00.000014894  2010-11-02
2 2010-10-01 1970-01-01 00:00:00.000014894  2010-11-03
3 2010-10-01 2010-10-01 00:00:00.000000000  2010-11-04
4 2010-10-01 2010-10-01 00:00:00.000000000  2010-11-05

[jorisvandenbossche]

@jreback I can confirm this also with pandas 0.16.2 / numpy 1.9.2

In [36]: np.__version__
Out[36]: '1.9.2'

In [37]: pd.__version__
Out[37]: '0.16.2'

In [38]: df.loc[0:2,'c'] = np.datetime64('2010-10-12')

In [39]: df
Out[39]:
           b                             c         obj
0 2010-10-01 1970-01-01 00:00:00.000014894  2010-11-01
1 2010-10-01 1970-01-01 00:00:00.000014894  2010-11-02
2 2010-10-01 1970-01-01 00:00:00.000014894  2010-11-03
3 2010-10-01 2010-10-01 00:00:00.000000000  2010-11-04
4 2010-10-01 2010-10-01 00:00:00.000000000  2010-11-05

[seanv507]

which is the version i am using as I mentioned in my original bug report
(forgot to state the numpy version 1.9.2)

On Tue, Jun 23, 2015 at 12:52 PM, Joris Van den Bossche <
notifications@github.com> wrote:

@jreback https://github.com/jreback I can confirm this also with pandas
0.16.2 / numpy 1.9.2

In [36]: np.version
Out[36]: '1.9.2'

In [37]: pd.version
Out[37]: '0.16.2'

In [38]: df.loc[0:2,'c'] = np.datetime64('2010-10-12')

In [39]: df
Out[39]:
b c obj
0 2010-10-01 1970-01-01 00:00:00.000014894 2010-11-01
1 2010-10-01 1970-01-01 00:00:00.000014894 2010-11-02
2 2010-10-01 1970-01-01 00:00:00.000014894 2010-11-03
3 2010-10-01 2010-10-01 00:00:00.000000000 2010-11-04
4 2010-10-01 2010-10-01 00:00:00.000000000 2010-11-05

—
Reply to this email directly or view it on GitHub
#10408 (comment).

[jreback]

Ahh, ok, seems that the test was insufficient; e.g. we are testing the equivalent of the [21], you are doing [22]

In [21]: np.datetime64(Timestamp('2010-10-12'))
Out[21]: numpy.datetime64('2010-10-11T20:00:00.000000-0400')

In [22]: np.datetime64('2010-10-12')
Out[22]: numpy.datetime64('2010-10-12')

ok, easy enough prob to fix, want to take a crack at it?

[jreback]

fix was in #9522 (original)

[jorisvandenbossche]

@jreback the PR you link to is about a datetime64 in the left-hand-side (inside the loc), while here it is the value being assigned, so don't know if this is related

[jreback]

ahh right - ok should be straightforward in any event

[jorisvandenbossche]

yep, indeed. But just to be sure this does not get lost, I will open a separate issue, and leave this one for the date error problem

-> #10412

[seanv507]

@jreback - yes I will give it a go!

[jreback]

@seanv507 gr8! here are the contributing docs. shout if you need help.

[schettino72]

PR #10644