Skip to content

ENH: Support all Scipy window types in rolling(..., win_type) #37204

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
85d1a30
CLN: Weighted rolling window methods
Oct 17, 2020
bc4d44b
Merge remote-tracking branch 'upstream/master' into cln/weighted_roll…
Oct 17, 2020
2ead3d5
typing
Oct 17, 2020
4466110
Merge remote-tracking branch 'upstream/master' into cln/weighted_roll…
Oct 17, 2020
bc698d1
Support all scipy windows with arguments
Oct 19, 2020
66bdb26
clarify whatsnew
Oct 19, 2020
5214f1b
flake8
Oct 19, 2020
3ed2bc7
clarify documentation
Oct 19, 2020
2427317
Add supported windows as of scipy 1.2.0
Oct 19, 2020
377d225
Fix test and mypy error
Oct 19, 2020
64dd172
Merge remote-tracking branch 'upstream/master' into cln/weighted_roll…
Oct 19, 2020
c056370
Use valid window in test
Oct 19, 2020
b2be373
Simplify documentation
Oct 19, 2020
047b0af
Merge remote-tracking branch 'upstream/master' into cln/weighted_roll…
Oct 20, 2020
135283c
Change test to use public constructor
Oct 20, 2020
97d5e6c
Merge remote-tracking branch 'upstream/master' into cln/weighted_roll…
Oct 21, 2020
aff1497
Merge remote-tracking branch 'upstream/master' into cln/weighted_roll…
Oct 22, 2020
231fc78
Merge remote-tracking branch 'upstream/master' into cln/weighted_roll…
Oct 22, 2020
0646a46
Merge remote-tracking branch 'upstream/master' into cln/weighted_roll…
Oct 25, 2020
5e4c886
Add test for invalid scipy arg
Oct 25, 2020
1460599
Merge remote-tracking branch 'upstream/master' into cln/weighted_roll…
Oct 27, 2020
36d9b1b
Merge remote-tracking branch 'upstream/master' into cln/weighted_roll…
Oct 27, 2020
e93de0c
Merge remote-tracking branch 'upstream/master' into cln/weighted_roll…
Oct 28, 2020
d47c0b2
Merge remote-tracking branch 'upstream/master' into cln/weighted_roll…
Oct 29, 2020
a160aa9
Merge remote-tracking branch 'upstream/master' into cln/weighted_roll…
Oct 30, 2020
a8a17f9
Merge remote-tracking branch 'upstream/master' into cln/weighted_roll…
Oct 30, 2020
126808e
Merge remote-tracking branch 'upstream/master' into cln/weighted_roll…
Oct 30, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions doc/source/user_guide/computation.rst
Original file line number Diff line number Diff line change
Expand Up @@ -451,6 +451,10 @@ The list of recognized types are the `scipy.signal window functions
* ``slepian`` (needs width)
* ``exponential`` (needs tau).

.. versionadded:: 1.2.0

All Scipy window types, concurrent with your installed version, are recognized ``win_types``.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

which functions do we now support that we did not before? can u enumerate them

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added. Eventually, we should remove all the enumerated supported window types as it will get stale with scipy developments

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe
but we do parameter introspection now -
would be ok to fix this up i think and just point to the documentation of the actual window functions


.. ipython:: python

ser = pd.Series(np.random.randn(10), index=pd.date_range("1/1/2000", periods=10))
Expand Down
1 change: 1 addition & 0 deletions doc/source/whatsnew/v1.2.0.rst
Original file line number Diff line number Diff line change
Expand Up @@ -226,6 +226,7 @@ Other enhancements
- :meth:`DataFrame.to_parquet` now returns a ``bytes`` object when no ``path`` argument is passed (:issue:`37105`)
- :class:`Rolling` now supports the ``closed`` argument for fixed windows (:issue:`34315`)
- :class:`DatetimeIndex` and :class:`Series` with ``datetime64`` or ``datetime64tz`` dtypes now support ``std`` (:issue:`37436`)
- :class:`Window` now supports all Scipy window types in ``win_type`` with flexible keyword argument support (:issue:`34556`)

.. _whatsnew_120.api_breaking.python:

Expand Down
123 changes: 18 additions & 105 deletions pandas/core/window/rolling.py
Original file line number Diff line number Diff line change
Expand Up @@ -873,30 +873,14 @@ class Window(BaseWindow):
To learn more about the offsets & frequency strings, please see `this link
<https://pandas.pydata.org/pandas-docs/stable/user_guide/timeseries.html#offset-aliases>`__.

The recognized win_types are:

* ``boxcar``
* ``triang``
* ``blackman``
* ``hamming``
* ``bartlett``
* ``parzen``
* ``bohman``
* ``blackmanharris``
* ``nuttall``
* ``barthann``
* ``kaiser`` (needs parameter: beta)
* ``gaussian`` (needs parameter: std)
* ``general_gaussian`` (needs parameters: power, width)
* ``slepian`` (needs parameter: width)
* ``exponential`` (needs parameter: tau), center is set to None.

If ``win_type=None`` all points are evenly weighted. To learn more about
different window types see `scipy.signal window functions
If ``win_type=None``, all points are evenly weighted; otherwise, ``win_type``
can accept a string of any `scipy.signal window function
<https://docs.scipy.org/doc/scipy/reference/signal.windows.html#module-scipy.signal.windows>`__.

Certain window types require additional parameters to be passed. Please see
the third example below on how to add the additional parameters.
Certain Scipy window types require additional parameters to be passed
in the aggregation function. The additional parameters must match
the keywords specified in the Scipy window type method signature.
Please see the third example below on how to add the additional parameters.

Examples
--------
Expand Down Expand Up @@ -1000,71 +984,22 @@ def _constructor(self):
def validate(self):
super().validate()

window = self.window
if isinstance(window, BaseIndexer):
if isinstance(self.window, BaseIndexer):
raise NotImplementedError(
"BaseIndexer subclasses not implemented with win_types."
)
elif isinstance(window, (list, tuple, np.ndarray)):
pass
elif is_integer(window):
if window <= 0:
elif is_integer(self.window):
if self.window <= 0:
raise ValueError("window must be > 0 ")
import_optional_dependency(
"scipy", extra="Scipy is required to generate window weight."
sig = import_optional_dependency(
"scipy.signal", extra="Scipy is required to generate window weight."
)
import scipy.signal as sig

if not isinstance(self.win_type, str):
raise ValueError(f"Invalid win_type {self.win_type}")
if getattr(sig, self.win_type, None) is None:
raise ValueError(f"Invalid win_type {self.win_type}")
else:
raise ValueError(f"Invalid window {window}")

def _get_win_type(self, kwargs: Dict[str, Any]) -> Union[str, Tuple]:
"""
Extract arguments for the window type, provide validation for it
and return the validated window type.

Parameters
----------
kwargs : dict

Returns
-------
win_type : str, or tuple
"""
# the below may pop from kwargs
def _validate_win_type(win_type, kwargs):
arg_map = {
"kaiser": ["beta"],
"gaussian": ["std"],
"general_gaussian": ["power", "width"],
"slepian": ["width"],
"exponential": ["tau"],
}

if win_type in arg_map:
win_args = _pop_args(win_type, arg_map[win_type], kwargs)
if win_type == "exponential":
# exponential window requires the first arg (center)
# to be set to None (necessary for symmetric window)
win_args.insert(0, None)

return tuple([win_type] + win_args)

return win_type

def _pop_args(win_type, arg_names, kwargs):
all_args = []
for n in arg_names:
if n not in kwargs:
raise ValueError(f"{win_type} window requires {n}")
all_args.append(kwargs.pop(n))
return all_args

return _validate_win_type(self.win_type, kwargs)
raise ValueError(f"Invalid window {self.window}")

def _center_window(self, result: np.ndarray, offset: int) -> np.ndarray:
"""
Expand All @@ -1079,31 +1014,6 @@ def _center_window(self, result: np.ndarray, offset: int) -> np.ndarray:
result = np.copy(result[tuple(lead_indexer)])
return result

def _get_window_weights(
self, win_type: Optional[Union[str, Tuple]] = None
) -> np.ndarray:
"""
Get the window, weights.

Parameters
----------
win_type : str, or tuple
type of window to create

Returns
-------
window : ndarray
the window, weights
"""
window = self.window
if isinstance(window, (list, tuple, np.ndarray)):
return com.asarray_tuplesafe(window).astype(float)
elif is_integer(window):
import scipy.signal as sig

# GH #15662. `False` makes symmetric window, rather than periodic.
return sig.get_window(win_type, window, False).astype(float)

def _apply(
self,
func: Callable[[np.ndarray, int, int], np.ndarray],
Expand All @@ -1124,14 +1034,17 @@ def _apply(
whether to cache a numba compiled function. Only available for numba
enabled methods (so far only apply)
**kwargs
additional arguments for rolling function and window function
additional arguments for scipy windows if necessary

Returns
-------
y : type of input
"""
win_type = self._get_win_type(kwargs)
window = self._get_window_weights(win_type=win_type)
signal = import_optional_dependency(
"scipy.signal", extra="Scipy is required to generate window weight."
)
assert self.win_type is not None # for mypy
window = getattr(signal, self.win_type)(self.window, **kwargs)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add a test if you pass bogus kwargs to a real scipy function.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

offset = (len(window) - 1) // 2 if self.center else 0

def homogeneous_func(values: np.ndarray):
Expand Down
4 changes: 2 additions & 2 deletions pandas/tests/window/moments/test_moments_rolling.py
Original file line number Diff line number Diff line change
Expand Up @@ -431,7 +431,7 @@ def test_cmov_window_special(win_types_special):
kwds = {
"kaiser": {"beta": 1.0},
"gaussian": {"std": 1.0},
"general_gaussian": {"power": 2.0, "width": 2.0},
"general_gaussian": {"p": 2.0, "sig": 2.0},
"exponential": {"tau": 10},
}

Expand Down Expand Up @@ -503,7 +503,7 @@ def test_cmov_window_special_linear_range(win_types_special):
kwds = {
"kaiser": {"beta": 1.0},
"gaussian": {"std": 1.0},
"general_gaussian": {"power": 2.0, "width": 2.0},
"general_gaussian": {"p": 2.0, "sig": 2.0},
"slepian": {"width": 0.5},
"exponential": {"tau": 10},
}
Expand Down
11 changes: 9 additions & 2 deletions pandas/tests/window/test_window.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,6 @@

import pandas as pd
from pandas import Series
from pandas.core.window import Window


@td.skip_if_no_scipy
Expand Down Expand Up @@ -50,7 +49,7 @@ def test_constructor_with_win_type(which, win_types):
@pytest.mark.parametrize("method", ["sum", "mean"])
def test_numpy_compat(method):
# see gh-12811
w = Window(Series([2, 4, 6]), window=[0, 2])
w = Series([2, 4, 6]).rolling(window=2)

msg = "numpy operations are not valid with window objects"

Expand All @@ -75,3 +74,11 @@ def test_agg_function_support(arg):

with pytest.raises(AttributeError, match=msg):
roll.agg({"A": arg})


@td.skip_if_no_scipy
def test_invalid_scipy_arg():
# This error is raised by scipy
msg = r"boxcar\(\) got an unexpected"
with pytest.raises(TypeError, match=msg):
Series(range(3)).rolling(1, win_type="boxcar").mean(foo="bar")