[DataFrame] Implemented nunique, skew #1995

hsubbaraj · 2018-05-03T06:59:01Z

Implemented nunique, skew

…test suite

AmplabJenkins · 2018-05-03T08:02:37Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5165/
Test PASSed.

kunalgosar

Looks really good! Just a few minor comments.

kunalgosar · 2018-05-03T23:14:16Z

python/ray/dataframe/dataframe.py

@@ -604,7 +604,8 @@ def groupby(self, by=None, axis=0, level=None, as_index=True, sort=True,
        return DataFrameGroupBy(self, by, axis, level, as_index, sort,
                                group_keys, squeeze, **kwargs)

-    def sum(self, axis=None, skipna=True, level=None, numeric_only=None):
+    def sum(self, axis=None, skipna=True, level=None, numeric_only=None, 


Add **kwargs to header too.

kunalgosar · 2018-05-03T23:15:15Z

python/ray/dataframe/dataframe.py

@@ -604,7 +604,8 @@ def groupby(self, by=None, axis=0, level=None, as_index=True, sort=True,
        return DataFrameGroupBy(self, by, axis, level, as_index, sort,
                                group_keys, squeeze, **kwargs)

-    def sum(self, axis=None, skipna=True, level=None, numeric_only=None):
+    def sum(self, axis=None, skipna=True, level=None, numeric_only=None, 
+            min_count=0):


From Pandas docs: "New in version 0.22.0: Added with the default being 1. This means the sum or product of an all-NA or empty series is NaN."

kunalgosar · 2018-05-03T23:17:20Z

python/ray/dataframe/dataframe.py

+            return df.prod(axis=axis, skipna=skipna, level=level,
+                          numeric_only=numeric_only, min_count=min_count)
+
+        return self._arithmetic_helper(remote_func, axis, level)

    def product(self, axis=None, skipna=None, level=None, numeric_only=None,


Better to call prod above instead of duplicating code.

kunalgosar · 2018-05-03T23:17:56Z

python/ray/dataframe/dataframe.py

+        """Perform a product across the DataFrame.
+
+        Args:
+            axis (int): The axis to product on.


Needs complete method docs. Seen in a few more functions in this PR.

kunalgosar · 2018-05-03T23:19:41Z

python/ray/dataframe/test/test_dataframe.py

-
+@pytest.fixture
+def test_prod(ray_df, pandas_df):
+    assert(ray_df.prod().sort_index().equals(pandas_df.prod().sort_index()))


use the ray_df_equals_pandas helper.

p-yang · 2018-05-04T00:23:18Z

You may want to separate changes out from those of #1994, so when it comes time we can merge both PRs at the same time.

hsubbaraj · 2018-05-04T04:16:42Z

I took out the changes from #1994 as well

AmplabJenkins · 2018-05-04T05:18:53Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5192/
Test PASSed.

kunalgosar

Few minor testing comments; generally, looks really good! Many of the early comments were addressed in #1994 after code was moved over.

kunalgosar · 2018-05-04T05:25:27Z

python/ray/dataframe/test/test_dataframe.py

@@ -2205,14 +2203,14 @@ def test_pow():
    test_inter_df_math("pow", simple=False)


-def test_prod():
+def test_prod(ray_df, pandas_df):


These changes should be moved to #1994

kunalgosar · 2018-05-04T05:25:56Z

python/ray/dataframe/test/test_dataframe.py

-        ray_df.nunique()
+@pytest.fixture
+def test_nunique(ray_df, pandas_df):
+    assert(ray_df_equals_pandas(ray_df.nunique(), pandas_df.nunique()))


Test both axis here.

kunalgosar

Looks great, thanks!

devin-petersohn

Looks great! Thanks @hsubbaraj!

AmplabJenkins · 2018-05-04T06:25:59Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5202/
Test FAILed.

AmplabJenkins · 2018-05-04T07:14:24Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5203/
Test PASSed.

AmplabJenkins · 2018-05-04T08:29:43Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5207/
Test PASSed.

AmplabJenkins · 2018-05-04T18:56:05Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5219/
Test PASSed.

devin-petersohn · 2018-05-04T19:22:15Z

Passes private-travis. Merged, thanks @hsubbaraj!

* master: (21 commits) Expand local_dir in Trial init (ray-project#2013) Fixing ascii error for Python2 (ray-project#2009) [DataFrame] Implements df.update (ray-project#1997) [DataFrame] Implements df.as_matrix (ray-project#2001) [DataFrame] Implement quantile (ray-project#1992) [DataFrame] Impement sort_values and sort_index (ray-project#1977) [DataFrame] Implement rank (ray-project#1991) [DataFrame] Implemented prod, product, added test suite (ray-project#1994) [DataFrame] Implemented __setitem__, select_dtypes, and astype (ray-project#1941) [DataFrame] Implement diff (ray-project#1996) [DataFrame] Implemented nunique, skew (ray-project#1995) [DataFrame] Implements filter and dropna (ray-project#1959) [DataFrame] Implements df.pipe (ray-project#1999) [DataFrame] Apply() for Lists and Dicts (ray-project#1973) Clean up syntax for supported Python versions. (ray-project#1963) [DataFrame] Implements mode, to_datetime, and get_dummies (ray-project#1956) [DataFrame] Fix dtypes (ray-project#1930) keep_dims -> keepdims (ray-project#1980) add pthread linking (ray-project#1986) [DataFrame] Add layer of abstraction to allow OID instantiation (ray-project#1984) ...

* master: (25 commits) [DataFrame] Add direct pandas imports for MVP (ray-project#1960) Make ActorHandles pickleable, also make proper ActorHandle and ActorC… (ray-project#2007) Expand local_dir in Trial init (ray-project#2013) Fixing ascii error for Python2 (ray-project#2009) [DataFrame] Implements df.update (ray-project#1997) [DataFrame] Implements df.as_matrix (ray-project#2001) [DataFrame] Implement quantile (ray-project#1992) [DataFrame] Impement sort_values and sort_index (ray-project#1977) [DataFrame] Implement rank (ray-project#1991) [DataFrame] Implemented prod, product, added test suite (ray-project#1994) [DataFrame] Implemented __setitem__, select_dtypes, and astype (ray-project#1941) [DataFrame] Implement diff (ray-project#1996) [DataFrame] Implemented nunique, skew (ray-project#1995) [DataFrame] Implements filter and dropna (ray-project#1959) [DataFrame] Implements df.pipe (ray-project#1999) [DataFrame] Apply() for Lists and Dicts (ray-project#1973) Clean up syntax for supported Python versions. (ray-project#1963) [DataFrame] Implements mode, to_datetime, and get_dummies (ray-project#1956) [DataFrame] Fix dtypes (ray-project#1930) keep_dims -> keepdims (ray-project#1980) ...

* master: [DataFrame] Add direct pandas imports for MVP (ray-project#1960) Make ActorHandles pickleable, also make proper ActorHandle and ActorC… (ray-project#2007) Expand local_dir in Trial init (ray-project#2013) Fixing ascii error for Python2 (ray-project#2009) [DataFrame] Implements df.update (ray-project#1997) [DataFrame] Implements df.as_matrix (ray-project#2001) [DataFrame] Implement quantile (ray-project#1992) [DataFrame] Impement sort_values and sort_index (ray-project#1977) [DataFrame] Implement rank (ray-project#1991) [DataFrame] Implemented prod, product, added test suite (ray-project#1994) [DataFrame] Implemented __setitem__, select_dtypes, and astype (ray-project#1941) [DataFrame] Implement diff (ray-project#1996) [DataFrame] Implemented nunique, skew (ray-project#1995) [DataFrame] Implements filter and dropna (ray-project#1959) [DataFrame] Implements df.pipe (ray-project#1999) [DataFrame] Apply() for Lists and Dicts (ray-project#1973)

hsubbaraj added 4 commits May 2, 2018 22:21

implemented prod/product, modified declaration for sum, added pandas …

4b2a830

…test suite

fixed tests

f9c5faf

removed test_analytics file

6d08249

implemented nunique, skew

f98ac8f

devin-petersohn changed the title ~~Implemented nunique, skew~~ [DataFrame] Implemented nunique, skew May 3, 2018

kunalgosar suggested changes May 3, 2018

View reviewed changes

fixed changes for nunique, skew

ca17d36

kunalgosar approved these changes May 4, 2018

View reviewed changes

hsubbaraj added 2 commits May 3, 2018 22:54

fixed nunique test

5761618

added axis=1 test to skew

79971e5

kunalgosar approved these changes May 4, 2018

View reviewed changes

devin-petersohn approved these changes May 4, 2018

View reviewed changes

hsubbaraj added 2 commits May 4, 2018 00:25

flake8 issues

bafb02e

more flake8 issues

bce1867

resolve some flake issues

80bedb1

devin-petersohn merged commit a58629f into ray-project:master May 4, 2018

simon-mo mentioned this pull request May 9, 2018

[DataFrame] Refactor indexers and implement setitem #2020

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[DataFrame] Implemented nunique, skew #1995

[DataFrame] Implemented nunique, skew #1995

hsubbaraj commented May 3, 2018

AmplabJenkins commented May 3, 2018

kunalgosar left a comment

kunalgosar May 3, 2018

kunalgosar May 3, 2018

kunalgosar May 3, 2018

kunalgosar May 3, 2018

kunalgosar May 3, 2018

p-yang commented May 4, 2018

hsubbaraj commented May 4, 2018

AmplabJenkins commented May 4, 2018

kunalgosar left a comment

kunalgosar May 4, 2018

kunalgosar May 4, 2018

kunalgosar left a comment

devin-petersohn left a comment

AmplabJenkins commented May 4, 2018

AmplabJenkins commented May 4, 2018

AmplabJenkins commented May 4, 2018

AmplabJenkins commented May 4, 2018

devin-petersohn commented May 4, 2018

[DataFrame] Implemented nunique, skew #1995

[DataFrame] Implemented nunique, skew #1995

Conversation

hsubbaraj commented May 3, 2018

AmplabJenkins commented May 3, 2018

kunalgosar left a comment

Choose a reason for hiding this comment

kunalgosar May 3, 2018

Choose a reason for hiding this comment

kunalgosar May 3, 2018

Choose a reason for hiding this comment

kunalgosar May 3, 2018

Choose a reason for hiding this comment

kunalgosar May 3, 2018

Choose a reason for hiding this comment

kunalgosar May 3, 2018

Choose a reason for hiding this comment

p-yang commented May 4, 2018

hsubbaraj commented May 4, 2018

AmplabJenkins commented May 4, 2018

kunalgosar left a comment

Choose a reason for hiding this comment

kunalgosar May 4, 2018

Choose a reason for hiding this comment

kunalgosar May 4, 2018

Choose a reason for hiding this comment

kunalgosar left a comment

Choose a reason for hiding this comment

devin-petersohn left a comment

Choose a reason for hiding this comment

AmplabJenkins commented May 4, 2018

AmplabJenkins commented May 4, 2018

AmplabJenkins commented May 4, 2018

AmplabJenkins commented May 4, 2018

devin-petersohn commented May 4, 2018