DataFrame.groupby() causes loss of precision on large integer values

[nisaggarwal]

Here's what I've in a dataframe to start with:

In [298]: g
Out[298]: 
                     ts  level
33  1382016600011617669     32
34  1382016600011625872     32
44  1382016600013590377     43
45  1382016600013598606     43

Running groupby() causes my 'ts' value to change and causes some loss of precision in the smallest units. My guess would be somewhere internally this column
is converted to a np.float64 and then returned back as an np.int64.

In [299]: g.groupby('level').last()
Out[299]: 
                        ts
level                     
32     1382016600011625984
43     1382016600013598720

Type information:

In [300]: g.dtypes
Out[300]: 
ts       int64
level    int64
dtype: object

Version info:

In [303]: pd.__version__
Out[303]: '0.12.0'

In [304]: np.__version__
Out[304]: '1.7.1'

[jreback]

yep that's exactly what happens
related #3007

not hard to fix by expanding the code generator for more types and then
directly them properly

would welcome a PR for this

[jtratner]

how would fused types help this?

[jreback]

its not fused types, but for example, need a definition for group_last_int64 (this part is easy, just a slight modification in the code generator). The harder part is then executing the group block-wise (e.g. by dtype) and dispatching, rather than just casting to floa64 (and then back). A bit tricky because of the splitting/recombing, but more so because you have to handle errors at the cython level when you have overflow (then upcast it). and do it again with a newer type. In some cases this can be predicted so its not a big deal (e.g. in the case of last, you know that you can always have the same return type, but sum could cause upcasting).

[jreback]

this was fixed by #9311

In [5]: df.groupby('level').last()
Out[5]: 
                        ts
level                     
32     1382016600011625872
43     1382016600013598606