Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dictarray to and from df #247

Merged
merged 3 commits into from
Dec 8, 2022
Merged

Dictarray to and from df #247

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
00a6dd6
add to and from df functionality to dictarray
jperez999 Dec 5, 2022
59e54a9
fix for to and from df functions
jperez999 Dec 6, 2022
c439ca5
Merge branch 'main' into dictarray-to-from-df
jperez999 Dec 6, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions merlin/systems/dag/dictarray.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@

import numpy as np

from merlin.core.dispatch import get_lib
from merlin.core.protocols import SeriesLike

try:
Expand Down Expand Up @@ -225,6 +226,25 @@ def copy(self):
"""
return DictArray(self._columns.copy())

def to_df(self):
"""
Create a DataFrame from the DictArray
"""
df = get_lib().DataFrame()
for col in self.columns:
df[col] = get_lib().Series(self[col])
return df

@classmethod
def from_df(cls, df):
"""
Create a DictArray from a DataFrame
"""
array_dict = {}
for col in df.columns:
array_dict[col] = df[col].to_numpy()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We might need some extra logic here to handle list values. There's currently a bug in cudf that results in this throwing a TypeError if we hit this line with a series with a list dtype.

rapidsai/cudf#12115

import cudf
cudf.Series([[1]]).to_numpy()
# => Raises TypeError: issubclass() arg 1 must be a class

And in the case of cudf would we want to keep the data on the same device when using this method?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a mini step that allows us to get the current tests working. We will not be keeping this in the long term. It will be modified once merlin dtypes and columns are in place in Core.

return cls(array_dict)


def _array_lib():
"""Dispatch to the appropriate library (cupy or numpy) for the current environment"""
Expand Down