-
-
Notifications
You must be signed in to change notification settings - Fork 18.5k
ENH: Support all Scipy window types in rolling(..., win_type) #37204
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
jreback
merged 27 commits into
pandas-dev:master
from
mroeschke:cln/weighted_rolling_window_methods
Oct 31, 2020
Merged
Changes from all commits
Commits
Show all changes
27 commits
Select commit
Hold shift + click to select a range
85d1a30
CLN: Weighted rolling window methods
bc4d44b
Merge remote-tracking branch 'upstream/master' into cln/weighted_roll…
2ead3d5
typing
4466110
Merge remote-tracking branch 'upstream/master' into cln/weighted_roll…
bc698d1
Support all scipy windows with arguments
66bdb26
clarify whatsnew
5214f1b
flake8
3ed2bc7
clarify documentation
2427317
Add supported windows as of scipy 1.2.0
377d225
Fix test and mypy error
64dd172
Merge remote-tracking branch 'upstream/master' into cln/weighted_roll…
c056370
Use valid window in test
b2be373
Simplify documentation
047b0af
Merge remote-tracking branch 'upstream/master' into cln/weighted_roll…
135283c
Change test to use public constructor
97d5e6c
Merge remote-tracking branch 'upstream/master' into cln/weighted_roll…
aff1497
Merge remote-tracking branch 'upstream/master' into cln/weighted_roll…
231fc78
Merge remote-tracking branch 'upstream/master' into cln/weighted_roll…
0646a46
Merge remote-tracking branch 'upstream/master' into cln/weighted_roll…
5e4c886
Add test for invalid scipy arg
1460599
Merge remote-tracking branch 'upstream/master' into cln/weighted_roll…
36d9b1b
Merge remote-tracking branch 'upstream/master' into cln/weighted_roll…
e93de0c
Merge remote-tracking branch 'upstream/master' into cln/weighted_roll…
d47c0b2
Merge remote-tracking branch 'upstream/master' into cln/weighted_roll…
a160aa9
Merge remote-tracking branch 'upstream/master' into cln/weighted_roll…
a8a17f9
Merge remote-tracking branch 'upstream/master' into cln/weighted_roll…
126808e
Merge remote-tracking branch 'upstream/master' into cln/weighted_roll…
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -873,30 +873,14 @@ class Window(BaseWindow): | |
To learn more about the offsets & frequency strings, please see `this link | ||
<https://pandas.pydata.org/pandas-docs/stable/user_guide/timeseries.html#offset-aliases>`__. | ||
|
||
The recognized win_types are: | ||
|
||
* ``boxcar`` | ||
* ``triang`` | ||
* ``blackman`` | ||
* ``hamming`` | ||
* ``bartlett`` | ||
* ``parzen`` | ||
* ``bohman`` | ||
* ``blackmanharris`` | ||
* ``nuttall`` | ||
* ``barthann`` | ||
* ``kaiser`` (needs parameter: beta) | ||
* ``gaussian`` (needs parameter: std) | ||
* ``general_gaussian`` (needs parameters: power, width) | ||
* ``slepian`` (needs parameter: width) | ||
* ``exponential`` (needs parameter: tau), center is set to None. | ||
|
||
If ``win_type=None`` all points are evenly weighted. To learn more about | ||
different window types see `scipy.signal window functions | ||
If ``win_type=None``, all points are evenly weighted; otherwise, ``win_type`` | ||
can accept a string of any `scipy.signal window function | ||
<https://docs.scipy.org/doc/scipy/reference/signal.windows.html#module-scipy.signal.windows>`__. | ||
|
||
Certain window types require additional parameters to be passed. Please see | ||
the third example below on how to add the additional parameters. | ||
Certain Scipy window types require additional parameters to be passed | ||
in the aggregation function. The additional parameters must match | ||
the keywords specified in the Scipy window type method signature. | ||
Please see the third example below on how to add the additional parameters. | ||
|
||
Examples | ||
-------- | ||
|
@@ -1000,71 +984,22 @@ def _constructor(self): | |
def validate(self): | ||
super().validate() | ||
|
||
window = self.window | ||
if isinstance(window, BaseIndexer): | ||
if isinstance(self.window, BaseIndexer): | ||
raise NotImplementedError( | ||
"BaseIndexer subclasses not implemented with win_types." | ||
) | ||
elif isinstance(window, (list, tuple, np.ndarray)): | ||
pass | ||
elif is_integer(window): | ||
if window <= 0: | ||
elif is_integer(self.window): | ||
if self.window <= 0: | ||
raise ValueError("window must be > 0 ") | ||
import_optional_dependency( | ||
"scipy", extra="Scipy is required to generate window weight." | ||
sig = import_optional_dependency( | ||
"scipy.signal", extra="Scipy is required to generate window weight." | ||
) | ||
import scipy.signal as sig | ||
|
||
if not isinstance(self.win_type, str): | ||
raise ValueError(f"Invalid win_type {self.win_type}") | ||
if getattr(sig, self.win_type, None) is None: | ||
raise ValueError(f"Invalid win_type {self.win_type}") | ||
else: | ||
raise ValueError(f"Invalid window {window}") | ||
|
||
def _get_win_type(self, kwargs: Dict[str, Any]) -> Union[str, Tuple]: | ||
""" | ||
Extract arguments for the window type, provide validation for it | ||
and return the validated window type. | ||
|
||
Parameters | ||
---------- | ||
kwargs : dict | ||
|
||
Returns | ||
------- | ||
win_type : str, or tuple | ||
""" | ||
# the below may pop from kwargs | ||
def _validate_win_type(win_type, kwargs): | ||
arg_map = { | ||
"kaiser": ["beta"], | ||
"gaussian": ["std"], | ||
"general_gaussian": ["power", "width"], | ||
"slepian": ["width"], | ||
"exponential": ["tau"], | ||
} | ||
|
||
if win_type in arg_map: | ||
win_args = _pop_args(win_type, arg_map[win_type], kwargs) | ||
if win_type == "exponential": | ||
# exponential window requires the first arg (center) | ||
# to be set to None (necessary for symmetric window) | ||
win_args.insert(0, None) | ||
|
||
return tuple([win_type] + win_args) | ||
|
||
return win_type | ||
|
||
def _pop_args(win_type, arg_names, kwargs): | ||
all_args = [] | ||
for n in arg_names: | ||
if n not in kwargs: | ||
raise ValueError(f"{win_type} window requires {n}") | ||
all_args.append(kwargs.pop(n)) | ||
return all_args | ||
|
||
return _validate_win_type(self.win_type, kwargs) | ||
raise ValueError(f"Invalid window {self.window}") | ||
|
||
def _center_window(self, result: np.ndarray, offset: int) -> np.ndarray: | ||
""" | ||
|
@@ -1079,31 +1014,6 @@ def _center_window(self, result: np.ndarray, offset: int) -> np.ndarray: | |
result = np.copy(result[tuple(lead_indexer)]) | ||
return result | ||
|
||
def _get_window_weights( | ||
self, win_type: Optional[Union[str, Tuple]] = None | ||
) -> np.ndarray: | ||
""" | ||
Get the window, weights. | ||
|
||
Parameters | ||
---------- | ||
win_type : str, or tuple | ||
type of window to create | ||
|
||
Returns | ||
------- | ||
window : ndarray | ||
the window, weights | ||
""" | ||
window = self.window | ||
if isinstance(window, (list, tuple, np.ndarray)): | ||
return com.asarray_tuplesafe(window).astype(float) | ||
elif is_integer(window): | ||
import scipy.signal as sig | ||
|
||
# GH #15662. `False` makes symmetric window, rather than periodic. | ||
return sig.get_window(win_type, window, False).astype(float) | ||
|
||
def _apply( | ||
self, | ||
func: Callable[[np.ndarray, int, int], np.ndarray], | ||
|
@@ -1124,14 +1034,17 @@ def _apply( | |
whether to cache a numba compiled function. Only available for numba | ||
enabled methods (so far only apply) | ||
**kwargs | ||
additional arguments for rolling function and window function | ||
additional arguments for scipy windows if necessary | ||
|
||
Returns | ||
------- | ||
y : type of input | ||
""" | ||
win_type = self._get_win_type(kwargs) | ||
window = self._get_window_weights(win_type=win_type) | ||
signal = import_optional_dependency( | ||
"scipy.signal", extra="Scipy is required to generate window weight." | ||
) | ||
assert self.win_type is not None # for mypy | ||
window = getattr(signal, self.win_type)(self.window, **kwargs) | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. add a test if you pass bogus kwargs to a real scipy function. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Done |
||
offset = (len(window) - 1) // 2 if self.center else 0 | ||
|
||
def homogeneous_func(values: np.ndarray): | ||
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
which functions do we now support that we did not before? can u enumerate them
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added. Eventually, we should remove all the enumerated supported window types as it will get stale with scipy developments
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
maybe
but we do parameter introspection now -
would be ok to fix this up i think and just point to the documentation of the actual window functions