Removes empty partitions after dropping rows and splitting datasets #2328
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
for more information, see https://pre-commit.ci
arnavgarg1
approved these changes
Jul 28, 2022
…i/ludwig into remove-empty-partitions
for more information, see https://pre-commit.ci
…set <=> Dask conversions
for more information, see https://pre-commit.ci
for more information, see https://pre-commit.ci
ShreyaR
approved these changes
Aug 3, 2022
arnavgarg1
reviewed
Aug 3, 2022
ludwig/data/dataframe/dask.py
Outdated
| @@ -129,7 +149,24 @@ def to_ray_dataset(self, df): | |||
| return from_dask(df) | |||
|
|
|||
| def from_ray_dataset(self, dataset) -> dd.DataFrame: | |||
| return dataset.to_dask() | |||
| """Custom Ray to Dask conversion implementation to pass in meta during dd.DataFrame creation.""" | |||
There was a problem hiding this comment.
@geoffreyangus should we revert back to dataset.to_dask() if the empty partitions issue doesn't stem from a need to pass in meta?
There was a problem hiding this comment.
Yup, reverted!
|
LGTM! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR addresses two separate issues: #2324 and #2308.
The issues are addressed by culling empty partitions from the Dask DataFrame at two points: (1) after dropping rows with NaNs (part of the DROP_ROWS missing value strategy) and (2) after splitting the dataset into train/val/test.
In order to maintain/increase performance, we add a persist call at the end of
build_dataset, which makes it relatively inexpensive to compute the length of partitions repeatedly downstream.