API/ENH: master issue for pd.rolling_apply

[leeong05]

Catchall for rolling_* issues:

• ENH: window functions need to exclude nuiscance columns #12537 exclude nuiscance columns
• ENH: support timedeltas with window functions #12536 fully support timedeltas
• Non-Reducing Return Values in Rolling .apply #4130 (return type of rolling_apply)
• PERF/ENH: np.apply_along_axis -> reduce in moments.py/nanops.py #3185 (stop using np.apply_along_axis)
• ENH/API: rolling_apply to pass frames to the rolled function (rather than ndarrays) #5071, Functions applied on .expanding() receive ndarrays rather than pandas objects #12950 (pass frames, operate 2-D)
• BUG: pandas rolling_quantile does not use interpolation #9413 rolling_quantile
• Implement high performance rolling_rank #9481 rolling_idxmax
• Rolling qcut #10759 rolling_qcut
• ENH: Support returning the same dtype as the caller for window ops (including extension dtypes) #11446, rolling.apply or rolling.agg doesn't work on string columns #20773 casting in .apply
• Implementing rolling min/max functions that can retain the original type #12595 casting in all window functions (min/max are good)
• ENH: Add the moment function as DataFrame and Series method WITH namespacing #10702 namespacing (though prob just create a Rolling object), API: provide Rolling/Expanding/EWM objects for deferred rolling type calculations #10702 #11603
• TST: test_window/Datetimelike adjustments #12535 testing of datetimelike with nans
• BUG: window function count should count anything non-null #12541 count should be integer dtype (and deal with infs)
• API: Table-wise rolling / expanding / EWM function application #15095 Table-wise

---

Hi all,

I intended to apply a function that gives on each day a ranking based on means calculated from previous n-day's data. The natural way is to use pd.rolling_apply. A toy example:

In [93]: df = pd.DataFrame(np.random.randint(10, size=20).reshape(4, 5))

In [94]: df
Out[94]: 
   0  1  2  3  4
0  2  0  0  2  0
1  9  5  5  6  1
2  2  3  6  8  8
3  5  1  2  9  0

In [95]: import bottleneck as bn

In [96]: bn.nanrankdata(df.mean())
Out[96]: array([ 4. ,  1.5,  3. ,  5. ,  1.5])

Up to now, it is cool. Then:

In [97]: pd.rolling_apply(df, 2, lambda x: bn.nanrankdata(bn.nanmean(x, axis=0)))
Out[97]: 
    0   1   2   3   4
0 NaN NaN NaN NaN NaN
1   1   1   1   1   1
2   1   1   1   1   1
3   1   1   1   1   1

This is clearly wrong. Is this a bug?

[jreback]

import pandas as pd
import numpy as np
import bottleneck as bn

df = pd.DataFrame(np.random.randint(10, size=20).reshape(4, 5))

def f(x):
    import pdb; pdb.set_trace()
    result = bn.nanrankdata(bn.nanmean(x, axis=0))
    print result
    return result

pd.rolling_apply(df, 2, f)

rankdata is being passed a scalar (the result of the mean). The input is only 1-d, NOT 2-d

whenever I have something like this I create a function and step thru

[leeong05]

Thanks. With pdb, the pd.rolling_apply works in this way: i) it works column by column in order ii) on each column, it splits this long 1-D array into small array with size window and apply the function to it.

What I had in mind is that pd.rolling_apply splits the 2-D array into chunks along axis=0 and apply the function to the small 2-D array.

For this specific question, the workaround is:

pd.rolling_apply(df, 2, bn.nanmean).rank(axis=1)

But is the 2-D to 2-D rolling feature not desirable?

[jreback]

you can just do pd.rolling_mean(df,2).rank(axis=1) FYI (bottleneck mean is dispatched if its available on appropriate dtypes anyhow)

yes, I don't think this case is 'handled per se'. But this is really a rolling-grid type of things yes?

It is implemented column by column for simplicity ATM (and uses np.apply_along_axis which is pretty inefficient IMHO).

a couple of open issues w.r.t. rolling_*, see: #4964, #4130, #3185 and I think you want #5071

so yes I would agree that is needed.

like to work on it?

cc @seth-p whom has done quite a bit of work on fixing some of these as well

[wesm]

What is the status of this issue?

[jreback]

its just tracking many open issues here

[mroeschke]

I don't think we need this high level tracking issue. There are not too many windowing issues and most are enhancement requests for other non apply methods. Closing.