[DataFrame] Implements df.update #1997
Conversation
|
Test PASSed. |
There was a problem hiding this comment.
This function requires more nuanced handling. This will require logic similar to the join function implemented by @devin-petersohn.
python/ray/dataframe/dataframe.py
Outdated
| "To contribute to Pandas on Ray, please visit " | ||
| "github.com/ray-project/ray.") | ||
|
|
||
| if join != 'left': # pragma: no cover |
There was a problem hiding this comment.
This # pragma: no cover is an artifact from pandas integration with code coverage tools. Unnecessary for our case.
python/ray/dataframe/dataframe.py
Outdated
|
|
||
| def _update_part(x, y, axis, level): | ||
| new_df = x.copy() | ||
| new_df.update( |
There was a problem hiding this comment.
Inner partition index does not match outer index in our architecture. This will not work here, you will need to handle the difference in the outer and inner index when calling the underlying pandas update function.
There was a problem hiding this comment.
Ah, I see. I noticed in the pandas source, that they align the indexes up before performing the update. Do you think it would be best for me to wait unitl reindex/reindex_like get's merged in?
There was a problem hiding this comment.
Actually, i'll look at devin's join to see if I can do it without using reindex
There was a problem hiding this comment.
There is a _reindex_helper in utils. It is only used for join currently.
python/ray/dataframe/dataframe.py
Outdated
| return new_df | ||
|
|
||
| result = self._operator_helper( | ||
| _update_part, other, 0, None |
There was a problem hiding this comment.
other here can potentially be an entire DataFrame which is potentially a huge object, this should not be passed to each remote call.
devin-petersohn
force-pushed
the
peter/update
branch
from
a0a26fc to
abade94
Compare
python/ray/dataframe/dataframe.py
Outdated
| return x | ||
|
|
||
| self._inter_df_op_helper( | ||
| lambda x, y: update_helper(x, y), |
There was a problem hiding this comment.
You can pass update_helper directly here; does not need a lambda.
|
Test PASSed. |
|
Test PASSed. |
|
Test PASSed. |
|
Passes on private-travis. Merged, thanks @SaladRaider! |
* master: (21 commits) Expand local_dir in Trial init (ray-project#2013) Fixing ascii error for Python2 (ray-project#2009) [DataFrame] Implements df.update (ray-project#1997) [DataFrame] Implements df.as_matrix (ray-project#2001) [DataFrame] Implement quantile (ray-project#1992) [DataFrame] Impement sort_values and sort_index (ray-project#1977) [DataFrame] Implement rank (ray-project#1991) [DataFrame] Implemented prod, product, added test suite (ray-project#1994) [DataFrame] Implemented __setitem__, select_dtypes, and astype (ray-project#1941) [DataFrame] Implement diff (ray-project#1996) [DataFrame] Implemented nunique, skew (ray-project#1995) [DataFrame] Implements filter and dropna (ray-project#1959) [DataFrame] Implements df.pipe (ray-project#1999) [DataFrame] Apply() for Lists and Dicts (ray-project#1973) Clean up syntax for supported Python versions. (ray-project#1963) [DataFrame] Implements mode, to_datetime, and get_dummies (ray-project#1956) [DataFrame] Fix dtypes (ray-project#1930) keep_dims -> keepdims (ray-project#1980) add pthread linking (ray-project#1986) [DataFrame] Add layer of abstraction to allow OID instantiation (ray-project#1984) ...
* master: (25 commits) [DataFrame] Add direct pandas imports for MVP (ray-project#1960) Make ActorHandles pickleable, also make proper ActorHandle and ActorC… (ray-project#2007) Expand local_dir in Trial init (ray-project#2013) Fixing ascii error for Python2 (ray-project#2009) [DataFrame] Implements df.update (ray-project#1997) [DataFrame] Implements df.as_matrix (ray-project#2001) [DataFrame] Implement quantile (ray-project#1992) [DataFrame] Impement sort_values and sort_index (ray-project#1977) [DataFrame] Implement rank (ray-project#1991) [DataFrame] Implemented prod, product, added test suite (ray-project#1994) [DataFrame] Implemented __setitem__, select_dtypes, and astype (ray-project#1941) [DataFrame] Implement diff (ray-project#1996) [DataFrame] Implemented nunique, skew (ray-project#1995) [DataFrame] Implements filter and dropna (ray-project#1959) [DataFrame] Implements df.pipe (ray-project#1999) [DataFrame] Apply() for Lists and Dicts (ray-project#1973) Clean up syntax for supported Python versions. (ray-project#1963) [DataFrame] Implements mode, to_datetime, and get_dummies (ray-project#1956) [DataFrame] Fix dtypes (ray-project#1930) keep_dims -> keepdims (ray-project#1980) ...
* master: [DataFrame] Add direct pandas imports for MVP (ray-project#1960) Make ActorHandles pickleable, also make proper ActorHandle and ActorC… (ray-project#2007) Expand local_dir in Trial init (ray-project#2013) Fixing ascii error for Python2 (ray-project#2009) [DataFrame] Implements df.update (ray-project#1997) [DataFrame] Implements df.as_matrix (ray-project#2001) [DataFrame] Implement quantile (ray-project#1992) [DataFrame] Impement sort_values and sort_index (ray-project#1977) [DataFrame] Implement rank (ray-project#1991) [DataFrame] Implemented prod, product, added test suite (ray-project#1994) [DataFrame] Implemented __setitem__, select_dtypes, and astype (ray-project#1941) [DataFrame] Implement diff (ray-project#1996) [DataFrame] Implemented nunique, skew (ray-project#1995) [DataFrame] Implements filter and dropna (ray-project#1959) [DataFrame] Implements df.pipe (ray-project#1999) [DataFrame] Apply() for Lists and Dicts (ray-project#1973)
What do these changes do?
Implements df.update