Skip to content

ARROW-1639: [Python] Serialize RangeIndex as metadata via Table.from_pandas instead of converting to a column of integers #3868

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 4 commits into from
Closed

ARROW-1639: [Python] Serialize RangeIndex as metadata via Table.from_pandas instead of converting to a column of integers #3868

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
9ba4131
Serialize RangeIndex as metadata via Table.from_pandas instead of con…
wesm Mar 11, 2019
0ca1bfc
Add benchmark
wesm Mar 12, 2019
670dc6f
Add compatibility tests for pre-0.13 metadata. Add Arrow version to p…
wesm Mar 12, 2019
ec929ae
Add pandas_metadata attribute to pyarrow.Schema to make interactions …
wesm Mar 12, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions python/benchmarks/convert_pandas.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,3 +90,18 @@ def time_deserialize_from_buffer(self):

def time_deserialize_from_components(self):
pa.deserialize_components(self.as_components)


class SerializeDeserializePandas(object):

def setup(self):
# 10 million length
n = 10000000
self.df = pd.DataFrame({'data': np.random.randn(n)})
self.serialized = pa.serialize_pandas(self.df)

def time_serialize_pandas(self):
pa.serialize_pandas(self.df)

def time_deserialize_pandas(self):
pa.deserialize_pandas(self.serialized)
Loading