[DataFrame] Implemented prod, product, added test suite #1994

hsubbaraj · 2018-05-03T06:28:05Z

Implemented prod, product
pandas test suite for min, max, cumulative functions, sum, prod, product
added an additional missing parameter to sum

…test suite

devin-petersohn · 2018-05-03T06:32:00Z

python/ray/dataframe/test/test_analytics.py

@@ -0,0 +1,353 @@
+from __future__ import absolute_import


We should remove this file.

AmplabJenkins · 2018-05-03T06:59:32Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5161/
Test FAILed.

AmplabJenkins · 2018-05-03T07:52:37Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5163/
Test PASSed.

p-yang · 2018-05-04T00:20:37Z

python/ray/dataframe/dataframe.py

+            return df.prod(axis=axis, skipna=skipna, level=level,
+                          numeric_only=numeric_only, min_count=min_count)
+
+        return self._arithmetic_helper(remote_func, axis, level)

    def product(self, axis=None, skipna=None, level=None, numeric_only=None,


Pandas defines product as an alias of prod, so we should do the same.

AmplabJenkins · 2018-05-04T04:39:58Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5188/
Test PASSed.

devin-petersohn

Looks great! Just a couple of minor changes.

devin-petersohn · 2018-05-04T04:40:42Z

python/ray/dataframe/test/test_dataframe.py

-
+@pytest.fixture
+def test_prod(ray_df, pandas_df):
+    assert(ray_df_equals_pandas(ray_df.prod().sort_index(), pandas_df.prod().sort_index()))


Please remove sort_index() from each side of this. The index should come out the same.

devin-petersohn · 2018-05-04T04:40:55Z

python/ray/dataframe/test/test_dataframe.py

-        ray_df.product()
+@pytest.fixture
+def test_product(ray_df, pandas_df):
+    assert(ray_df_equals_pandas(ray_df.product().sort_index(), pandas_df.product().sort_index()))


Same as above regarding sort_index().

devin-petersohn

After you rebase and replace the newlines, looks good!

devin-petersohn · 2018-05-04T06:20:25Z

python/ray/dataframe/test/test_dataframe.py

@@ -3105,3 +3105,4 @@ def test__doc__():
            pd_obj = getattr(pd.DataFrame, attr, None)
            if callable(pd_obj) or isinstance(pd_obj, property):
                assert obj.__doc__ == pd_obj.__doc__
+


Please replace newline.

devin-petersohn · 2018-05-04T06:20:44Z

python/ray/dataframe/dataframe.py

@@ -4424,4 +4453,4 @@ def _merge_columns(left_columns, right_columns, *args):
    """
    return pd.DataFrame(columns=left_columns, index=[0], dtype='uint8').merge(
        pd.DataFrame(columns=right_columns, index=[0], dtype='uint8'),
-        *args).columns
+        *args).columns


Please replace newline.

AmplabJenkins · 2018-05-04T06:21:34Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5201/
Test FAILed.

AmplabJenkins · 2018-05-04T07:15:40Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5204/
Test PASSed.

AmplabJenkins · 2018-05-04T07:35:29Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5205/
Test PASSed.

AmplabJenkins · 2018-05-04T16:50:53Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5215/
Test PASSed.

hsubbaraj · 2018-05-04T19:43:02Z

Prod returns a Series, but I was using a dataframe check instead. This commit should resolve the issue

AmplabJenkins · 2018-05-04T20:47:12Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5224/
Test PASSed.

devin-petersohn · 2018-05-06T04:25:51Z

Merged, thanks @hsubbaraj!

* master: (21 commits) Expand local_dir in Trial init (ray-project#2013) Fixing ascii error for Python2 (ray-project#2009) [DataFrame] Implements df.update (ray-project#1997) [DataFrame] Implements df.as_matrix (ray-project#2001) [DataFrame] Implement quantile (ray-project#1992) [DataFrame] Impement sort_values and sort_index (ray-project#1977) [DataFrame] Implement rank (ray-project#1991) [DataFrame] Implemented prod, product, added test suite (ray-project#1994) [DataFrame] Implemented __setitem__, select_dtypes, and astype (ray-project#1941) [DataFrame] Implement diff (ray-project#1996) [DataFrame] Implemented nunique, skew (ray-project#1995) [DataFrame] Implements filter and dropna (ray-project#1959) [DataFrame] Implements df.pipe (ray-project#1999) [DataFrame] Apply() for Lists and Dicts (ray-project#1973) Clean up syntax for supported Python versions. (ray-project#1963) [DataFrame] Implements mode, to_datetime, and get_dummies (ray-project#1956) [DataFrame] Fix dtypes (ray-project#1930) keep_dims -> keepdims (ray-project#1980) add pthread linking (ray-project#1986) [DataFrame] Add layer of abstraction to allow OID instantiation (ray-project#1984) ...

* master: (25 commits) [DataFrame] Add direct pandas imports for MVP (ray-project#1960) Make ActorHandles pickleable, also make proper ActorHandle and ActorC… (ray-project#2007) Expand local_dir in Trial init (ray-project#2013) Fixing ascii error for Python2 (ray-project#2009) [DataFrame] Implements df.update (ray-project#1997) [DataFrame] Implements df.as_matrix (ray-project#2001) [DataFrame] Implement quantile (ray-project#1992) [DataFrame] Impement sort_values and sort_index (ray-project#1977) [DataFrame] Implement rank (ray-project#1991) [DataFrame] Implemented prod, product, added test suite (ray-project#1994) [DataFrame] Implemented __setitem__, select_dtypes, and astype (ray-project#1941) [DataFrame] Implement diff (ray-project#1996) [DataFrame] Implemented nunique, skew (ray-project#1995) [DataFrame] Implements filter and dropna (ray-project#1959) [DataFrame] Implements df.pipe (ray-project#1999) [DataFrame] Apply() for Lists and Dicts (ray-project#1973) Clean up syntax for supported Python versions. (ray-project#1963) [DataFrame] Implements mode, to_datetime, and get_dummies (ray-project#1956) [DataFrame] Fix dtypes (ray-project#1930) keep_dims -> keepdims (ray-project#1980) ...

* master: [DataFrame] Add direct pandas imports for MVP (ray-project#1960) Make ActorHandles pickleable, also make proper ActorHandle and ActorC… (ray-project#2007) Expand local_dir in Trial init (ray-project#2013) Fixing ascii error for Python2 (ray-project#2009) [DataFrame] Implements df.update (ray-project#1997) [DataFrame] Implements df.as_matrix (ray-project#2001) [DataFrame] Implement quantile (ray-project#1992) [DataFrame] Impement sort_values and sort_index (ray-project#1977) [DataFrame] Implement rank (ray-project#1991) [DataFrame] Implemented prod, product, added test suite (ray-project#1994) [DataFrame] Implemented __setitem__, select_dtypes, and astype (ray-project#1941) [DataFrame] Implement diff (ray-project#1996) [DataFrame] Implemented nunique, skew (ray-project#1995) [DataFrame] Implements filter and dropna (ray-project#1959) [DataFrame] Implements df.pipe (ray-project#1999) [DataFrame] Apply() for Lists and Dicts (ray-project#1973)

hsubbaraj added 2 commits May 2, 2018 22:21

implemented prod/product, modified declaration for sum, added pandas …

4b2a830

…test suite

fixed tests

f9c5faf

devin-petersohn changed the title ~~Implemented prod, product, added test suite~~ [DataFrame] Implemented prod, product, added test suite May 3, 2018

devin-petersohn reviewed May 3, 2018

View reviewed changes

hsubbaraj added 2 commits May 2, 2018 23:44

removed test_analytics file

6d08249

implemented nunique, skew

f98ac8f

p-yang suggested changes May 4, 2018

View reviewed changes

p-yang mentioned this pull request May 4, 2018

[DataFrame] Implemented nunique, skew #1995

Merged

hsubbaraj added 2 commits May 3, 2018 20:36

fixed requested changes

3b29c96

added nunique, skew

c16109f

devin-petersohn reviewed May 4, 2018

View reviewed changes

hsubbaraj added 2 commits May 3, 2018 22:51

fixed tests in request

4962ad5

added newline back

11a77b6

devin-petersohn reviewed May 4, 2018

View reviewed changes

fixed newlines hopefully

571171f

devin-petersohn approved these changes May 4, 2018

View reviewed changes

hsubbaraj added 2 commits May 4, 2018 08:44

fixed flake8 issues

4601e93

more flake8 issues

15f58f1

fixed test for prod

82bde0e

devin-petersohn merged commit 857458c into ray-project:master May 6, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[DataFrame] Implemented prod, product, added test suite #1994

[DataFrame] Implemented prod, product, added test suite #1994

hsubbaraj commented May 3, 2018

devin-petersohn May 3, 2018

AmplabJenkins commented May 3, 2018

AmplabJenkins commented May 3, 2018

p-yang May 4, 2018

AmplabJenkins commented May 4, 2018

devin-petersohn left a comment

devin-petersohn May 4, 2018

devin-petersohn May 4, 2018

devin-petersohn left a comment

devin-petersohn May 4, 2018

devin-petersohn May 4, 2018

AmplabJenkins commented May 4, 2018

AmplabJenkins commented May 4, 2018

AmplabJenkins commented May 4, 2018

AmplabJenkins commented May 4, 2018

hsubbaraj commented May 4, 2018

AmplabJenkins commented May 4, 2018

devin-petersohn commented May 6, 2018

[DataFrame] Implemented prod, product, added test suite #1994

[DataFrame] Implemented prod, product, added test suite #1994

Conversation

hsubbaraj commented May 3, 2018

devin-petersohn May 3, 2018

Choose a reason for hiding this comment

AmplabJenkins commented May 3, 2018

AmplabJenkins commented May 3, 2018

p-yang May 4, 2018

Choose a reason for hiding this comment

AmplabJenkins commented May 4, 2018

devin-petersohn left a comment

Choose a reason for hiding this comment

devin-petersohn May 4, 2018

Choose a reason for hiding this comment

devin-petersohn May 4, 2018

Choose a reason for hiding this comment

devin-petersohn left a comment

Choose a reason for hiding this comment

devin-petersohn May 4, 2018

Choose a reason for hiding this comment

devin-petersohn May 4, 2018

Choose a reason for hiding this comment

AmplabJenkins commented May 4, 2018

AmplabJenkins commented May 4, 2018

AmplabJenkins commented May 4, 2018

AmplabJenkins commented May 4, 2018

hsubbaraj commented May 4, 2018

AmplabJenkins commented May 4, 2018

devin-petersohn commented May 6, 2018