Issue #10174. Add 'interpolation' keyword in DataFrame.quantile and Series.quantile #10204

mayankasthana · 2015-05-24T23:06:36Z

Closes #10174.
I have added the new argument to the doc too.

shoyer · 2015-05-24T23:24:29Z

This need some tests.

mayankasthana · 2015-05-24T23:32:11Z

I'll add some tests too.
Meanwhile I am trying to fix the build fail due to a stray \xe2 character.

mayankasthana · 2015-05-25T08:20:45Z

The interpolation argument was added to np.percentile() only in numpy version 1.9.0. The tests don't pass where numpy version < 1.9.0. How to handle this?

jorisvandenbossche · 2015-05-25T10:47:12Z

pandas/core/frame.py

@@ -4479,6 +4479,14 @@ def quantile(self, q=0.5, axis=0, numeric_only=True):
            0 <= q <= 1, the quantile(s) to compute
        axis : {0, 1}
            0 for row-wise, 1 for column-wise
+        interpolation : {'linear', 'lower', 'higher', 'midpoint', 'nearest'}
+            Specifies the interpolation method to use, when the desired quantile lies between two data points i and j:


Can you keep here to PEP8 line length?

Further, better to make a bullet points list of the options (newlines are ignored when rendered to html)

BTW, you can just copy it from the source from the numpy.percentile function (using np.percentile?? in IPython):

interpolation : {'linear', 'lower', 'higher', 'midpoint', 'nearest'} This optional parameter specifies the interpolation method to use, when the desired quantile lies between two data points `i` and `j`: * linear: `i + (j - i) * fraction`, where `fraction` is the fractional part of the index surrounded by `i` and `j`. * lower: `i`. * higher: `j`. * nearest: `i` or `j` whichever is nearest. * midpoint: (`i` + `j`) / 2. .. versionadded:: 1.9.0

jorisvandenbossche · 2015-05-25T11:02:01Z

For the tests to handle the numpy version, you can do something like this:

if _np_version_under1p9:
    raise nose.SkipTest('numpy >= 1.9 required')

mayankasthana · 2015-05-25T12:46:55Z

What should happen when numpy version is < 1.9.0?

Should quantile() still work and ignore the interpolation argument? OR
should it fail saying that pd > 1.9 is required?

if 1. , then I can put the if _np_version_under1p9: check inside quantile() and handle the different numpy versions transparently. We can throw a warning if the interpolation keyword is specified, but not being used (np < 1.9).

shoyer · 2015-05-25T21:05:39Z

The default interpolation argument should work on all supported versions of NumPy.

For numpy < 1.9.0, it's OK not to support other interpolation options. In these cases, we should raise a ValueError with an informative error message. We should certainly not ignore the interpolation argument or merely raise a warning -- that could lead to unanticipated bugs.

mayankasthana · 2015-05-25T22:08:46Z

Now the default interpolation argument works on all supported versions of NumPy.
And raised ValueError if an interpolation method other than linear is specified when numpy versions <1.9.

Will add tests soon.

mayankasthana · 2015-05-26T14:05:48Z

I added tests for Dataframe.quantile . This is my first time writing tests, so please verify if it is done right.
If someone verifies these tests, I'll write the tests for Series.quantile too.

jorisvandenbossche · 2015-05-26T14:18:48Z

pandas/tests/test_frame.py

+        from numpy import percentile
+        #interpolation = linear (default case)
+        q = self.tsframe.quantile(0.1, axis=0,interpolation='linear')
+        self.assertEqual(q['A'], percentile(self.tsframe['A'], 10))


you can also use np.percentile here, instead of importing percentile explicitely from numpy (np is already available)

jorisvandenbossche · 2015-05-26T14:21:48Z

Tests look good! Most important things to be tested is that the keyword is passed to percentile correctly, that it generate an error for older numpy versions, and that the default is not changed. And those are included, so OK!

mayankasthana · 2015-05-26T14:47:25Z

I added the explicit test that the result with interpolation='linear' and without specifying it is the same.
Also added the github issue number.

jreback · 2015-05-26T14:50:58Z

this is getting to be lots of duplicated code. I would prefer that this all be moved to core/generic.py, see #10207

jreback · 2015-08-05T21:14:43Z

can you rebase?

mayankasthana · 2015-08-09T21:04:48Z

@jreback rebase done!

jreback · 2015-08-10T11:15:26Z

not really sure what you did. you should have 1 commit.

pls see contributing docs: http://pandas.pydata.org/pandas-docs/stable/contributing.html

mayankasthana · 2015-08-10T12:42:08Z

I screwed up a lot. I fetched upstream and then rebased on that or something.
It's better I close this and create a new pull request following the contributing guide.
@jreback Is there any better way?

jorisvandenbossche · 2015-08-10T13:25:30Z

@mayankasthana No, there is no need to close this and create a new PR.

Normally, doing this:

git fetch upstream
git rebase upstream/master
git push -f origin interpolationQuantile

should be all that is needed to clean this up

mayankasthana · 2015-08-10T21:57:46Z

@jreback Thanks for the help with git. Here is the expected 1 commit.

jreback · 2015-08-10T22:09:59Z

looks pretty good. pls add a release note.

jorisvandenbossche · 2015-08-11T07:18:59Z

pandas/core/frame.py

+                * lower: `i`.
+                * higher: `j`.
+                * nearest: `i` or `j` whichever is nearest.
+                * midpoint: (`i` + `j`) / 2.



Can you add here the line:

.. versionadded:: 0.17.0

(same indentation as 'This optional ...')

mayankasthana · 2015-08-12T14:00:10Z

@jreback @jorisvandenbossche Where should I add the feature description in v0.17.0.txt?
Under Other API Changes or Other enhancements? And does it require a usage example?

jorisvandenbossche · 2015-08-12T14:45:35Z

You can put it under 'Other enhancements'. A one line sentence explaining it is enough I think.

mayankasthana · 2015-09-04T07:08:51Z

@jreback Rebased. Test is green.

mayankasthana · 2015-09-18T19:13:00Z

@jreback Rebased. Test is green.

jreback · 2015-11-10T01:22:51Z

@mayankasthana I'd really like to see #10207 but that can be done after. can you rebase.

mayankasthana · 2015-11-10T20:58:48Z

@jreback Rebased. Test is green.

jreback · 2015-12-06T19:11:10Z

can you rebase / update whats new to 0.18.0

mayankasthana · 2015-12-11T09:00:04Z

@jreback Rebased and updated whats new to 0.18.0.

jreback · 2015-12-12T14:28:30Z

pandas/core/series.py

+        def multi(values, qs, interpolation):
+            if _np_version_under1p9:
+                if com.is_list_like(qs):
+                    values = [_quantile(values, x*100) for x in qs]


just create a kwargs depending on the version and pass it to the functions, instead of duplicating all of this code

jreback · 2015-12-19T16:49:30Z

pandas/tests/test_frame.py

+        #test with and without interpolation keyword
+        assert_series_equal(q,q1)   
+
+        #interpolation method other than default linear


make this 2 tests, 1 for the version checking, and the other that would skip at the top if under verion 1.9

I separated <np 1.9 tests and >1.9 tests. Is this what you meant?

no, make them 2 separately tests, skipping on each one respectively if the numpy version not what you need it to be. Its simpler / easier to read that way.

jreback · 2015-12-19T16:49:50Z

comments updated

jreback · 2016-01-02T23:19:58Z

pandas/tests/test_series.py

+
+        #interpolation method other than default linear
+        if _np_version_under1p9:
+            expErrMsg = "Interpolation methods other than linear not supported in numpy < 1.9"


same here, make the numpy < 1.9 testsin a separate test

jreback · 2016-01-02T23:20:10Z

can you run: git diff master | flake8 --diff and fix anything. going to shortly be enforcing PEP8.

…rame.quantile and Series.quantile

mayankasthana · 2016-01-06T21:34:58Z

Created separate tests for different numpy versions.
Ran git diff master | flake8 --diff
Rebased.

jreback · 2016-01-07T00:20:48Z

merged via e05f66a

thanks!

mayankasthana · 2016-01-07T05:46:05Z

Thanks @jreback @shoyer @jorisvandenbossche

jorisvandenbossche reviewed May 25, 2015
View reviewed changes

jreback added API Design Numeric Operations Arithmetic, Comparison, and Logical operations labels May 26, 2015

jreback added this to the 0.17.0 milestone May 26, 2015

jorisvandenbossche reviewed May 26, 2015
View reviewed changes

mayankasthana force-pushed the interpolationQuantile branch 2 times, most recently from 175eafa to b365dcc Compare August 10, 2015 20:44

jorisvandenbossche reviewed Aug 11, 2015
View reviewed changes

mayankasthana force-pushed the interpolationQuantile branch from 715e787 to 6e03017 Compare September 18, 2015 18:33

mayankasthana force-pushed the interpolationQuantile branch from 6e03017 to 748a692 Compare November 10, 2015 18:58

mayankasthana force-pushed the interpolationQuantile branch 2 times, most recently from 3f32fcf to 324d811 Compare December 10, 2015 20:18

jreback reviewed Dec 12, 2015
View reviewed changes

mayankasthana force-pushed the interpolationQuantile branch 2 times, most recently from 2ff1ac5 to 7eeeb33 Compare December 16, 2015 20:11

jreback reviewed Dec 19, 2015
View reviewed changes

mayankasthana force-pushed the interpolationQuantile branch 2 times, most recently from 697ccae to 3cb69dd Compare December 20, 2015 22:03

jreback reviewed Jan 2, 2016
View reviewed changes

mayankasthana force-pushed the interpolationQuantile branch from 217535f to f8e4b5a Compare January 6, 2016 19:49

Closes issue pandas-dev#10174. Added 'interpolation' keyword in Dataf…

55836e6

…rame.quantile and Series.quantile

mayankasthana force-pushed the interpolationQuantile branch from f8e4b5a to 55836e6 Compare January 6, 2016 20:58

jreback modified the milestones: 0.18.0, Next Major Release Jan 6, 2016

jreback closed this Jan 7, 2016

mayankasthana deleted the interpolationQuantile branch September 18, 2016 20:42

Issue #10174. Add 'interpolation' keyword in DataFrame.quantile and Series.quantile #10204

Issue #10174. Add 'interpolation' keyword in DataFrame.quantile and Series.quantile #10204

Conversation

mayankasthana commented May 24, 2015

shoyer commented May 24, 2015

mayankasthana commented May 24, 2015

mayankasthana commented May 25, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jorisvandenbossche commented May 25, 2015

mayankasthana commented May 25, 2015

shoyer commented May 25, 2015

mayankasthana commented May 25, 2015

mayankasthana commented May 26, 2015

Choose a reason for hiding this comment

jorisvandenbossche commented May 26, 2015

mayankasthana commented May 26, 2015

jreback commented May 26, 2015

jreback commented Aug 5, 2015

mayankasthana commented Aug 9, 2015

jreback commented Aug 10, 2015

mayankasthana commented Aug 10, 2015

jorisvandenbossche commented Aug 10, 2015

mayankasthana commented Aug 10, 2015

jreback commented Aug 10, 2015

Choose a reason for hiding this comment

mayankasthana commented Aug 12, 2015

jorisvandenbossche commented Aug 12, 2015

mayankasthana commented Sep 4, 2015

mayankasthana commented Sep 18, 2015

jreback commented Nov 10, 2015

mayankasthana commented Nov 10, 2015

jreback commented Dec 6, 2015

mayankasthana commented Dec 11, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented Dec 19, 2015

Choose a reason for hiding this comment

jreback commented Jan 2, 2016

mayankasthana commented Jan 6, 2016

jreback commented Jan 7, 2016

mayankasthana commented Jan 7, 2016