feat(python): add pyarrow to delta compatible schema conversion in writer/merge #1820

ion-elgreco · 2023-11-07T21:45:50Z

Description

This ports some functionality that @stinodego and I had worked on in Polars. Where we converted a pyarrow schema to a compatible delta schema. It converts the following:

uint -> int
timestamp(any timeunit) -> timestamp(us)

I adjusted the functionality to do schema conversion from large to normal when necessary, which is still needed in MERGE as workaround #1753.

Additional things I've added:

Schema conversion for every input in write_deltalake/merge
Add Pandas dataframe conversion
Add Pandas dataframe as input in merge

Related Issue(s)

ion-elgreco · 2023-11-07T22:25:12Z

With 843c1d8 it's robust against schema's that have mixed Large and normal types. The schema will either be down casted to normal types or everything to large types, depending on user input.

python/deltalake/schema.py

ion-elgreco · 2023-11-11T11:58:14Z

@wjones127 do you know how we can cast schemas onto a pyarrow dataset or recordbatchreader without materializing? replace_schema on Dataset doesn't work apparently.

Also I am not sure if it's even wrong to not materialize since we have the validate_batch to check the invariants at the end

wjones127

This looking good. Just a few minor changes and it's ready to ship :)

python/deltalake/schema.py

python/tests/test_schema.py

python/tests/test_writer.py

ion-elgreco · 2023-11-17T07:50:47Z

@wjones127 I've applied all the requested changes

roeap · 2023-11-18T20:46:04Z

python/deltalake/writer.py

should this file live at the same level as the regular license and be included via the pyproject.toml?

Here's what I would propose:

Move the license to python/licenses/polars_license.txt and reference that path in code comment.

Create a file python/licenses/readme.md and include a list of which license apply to which code.

Add MIT license to pyproject.toml

I think it's possible we will borrow other polars code in the futures, so nice to name the file polars_license.txt.

Does that sound good @ion-elgreco ?

just realized I commented on a completely unrelated file - luckily the question seems to have been understood regardless 😆.

Personally, I like @wjones127's suggestion.

@wjones127 can you take a look at point 3, I wasn't sure on how I could add another license file in the toml for it to work with maturin.

Are we ready to merge this? 😀

roeap

👍

Co-authored-by: Will Jones <willjones127@gmail.com>

ion-elgreco · 2023-11-24T19:20:04Z

@roeap had to force push :) can you approve again?

Changes are resolved

@stinodego

…iter/merge (delta-io#1820) This ports some functionality that @stinodego and I had worked on in Polars. Where we converted a pyarrow schema to a compatible delta schema. It converts the following: - uint -> int - timestamp(any timeunit) -> timestamp(us) I adjusted the functionality to do schema conversion from large to normal when necessary, which is still needed in MERGE as workaround delta-io#1753. Additional things I've added: - Schema conversion for every input in write_deltalake/merge - Add Pandas dataframe conversion - Add Pandas dataframe as input in merge - closes delta-io#686 - closes delta-io#1467 --------- Co-authored-by: Will Jones <willjones127@gmail.com>

github-actions bot added the binding/python Issues for the Python package label Nov 7, 2023

ion-elgreco changed the title ~~feat: add pyarrow to delta compatible schema conversion in writer and~~ feat: add pyarrow to delta compatible schema conversion in writer/merge Nov 7, 2023

ion-elgreco changed the title ~~feat: add pyarrow to delta compatible schema conversion in writer/merge~~ feat(python): add pyarrow to delta compatible schema conversion in writer/merge Nov 7, 2023

ion-elgreco force-pushed the feat/add_schema_conversion_helpers branch from 3b9eebc to 2302148 Compare November 7, 2023 21:59

ion-elgreco marked this pull request as ready for review November 7, 2023 21:59

ion-elgreco requested review from wjones127, fvaleye and roeap as code owners November 7, 2023 21:59

wjones127 reviewed Nov 10, 2023

View reviewed changes

python/deltalake/schema.py Show resolved Hide resolved

wjones127 reviewed Nov 10, 2023

View reviewed changes

python/deltalake/schema.py Show resolved Hide resolved

wjones127 reviewed Nov 10, 2023

View reviewed changes

python/deltalake/schema.py Outdated Show resolved Hide resolved

ion-elgreco force-pushed the feat/add_schema_conversion_helpers branch from 1e7e9b1 to 3a1a120 Compare November 11, 2023 09:30

ion-elgreco requested a review from wjones127 November 11, 2023 11:33

ion-elgreco force-pushed the feat/add_schema_conversion_helpers branch from e370ecb to 8e076eb Compare November 11, 2023 23:15

ion-elgreco mentioned this pull request Nov 12, 2023

feat(python): Add merge mode to write_delta and remove pyarrow to delta conversions pola-rs/polars#12392

Merged

ion-elgreco force-pushed the feat/add_schema_conversion_helpers branch from ea25d41 to 850a05a Compare November 12, 2023 09:45

This was referenced Nov 15, 2023

Deltalake: Issue with specified schema for large dtypes pola-rs/polars#12467

Closed

feat(python): expose rust writer as additional engine #1872

Closed

wjones127 previously requested changes Nov 17, 2023

View reviewed changes

ion-elgreco force-pushed the feat/add_schema_conversion_helpers branch from 0e29368 to 08cc31e Compare November 17, 2023 07:48

ion-elgreco requested a review from wjones127 November 17, 2023 07:48

roeap reviewed Nov 18, 2023

View reviewed changes

ion-elgreco force-pushed the feat/add_schema_conversion_helpers branch 3 times, most recently from 83cbe65 to 3a31fc7 Compare November 20, 2023 06:42

ion-elgreco mentioned this pull request Nov 20, 2023

feat(python): expose rust writer as additional engine v2 #1891

Merged

ion-elgreco force-pushed the feat/add_schema_conversion_helpers branch from 3a31fc7 to 06931fa Compare November 20, 2023 22:18

roeap approved these changes Nov 24, 2023

View reviewed changes

ion-elgreco and others added 23 commits November 24, 2023 20:18

port schema conversion from polars and update writer and merge

424a92f

change folder name

fcb8534

only overwrite schema variable if schema is none

dee4864

Also upcast from normal to large when large = True

d56a9d2

use struct direct indexing for backwards pyarrow compat

4c0a549

Fix and improve pyarrow conversions

863d3e4

add many tests for pyarrow to delta schema conversions

e5fca75

Add test for more write inputs with large types

c2ac4fa

don't return schema and use generator

3cb085c

move imports to the top

95a5ff2

use generator typehint

687bd24

Update python/tests/test_writer.py

a11c432

Co-authored-by: Will Jones <willjones127@gmail.com>

Update python/tests/test_writer.py

d6a0c4b

Co-authored-by: Will Jones <willjones127@gmail.com>

Update python/deltalake/schema.py

33cb863

Co-authored-by: Will Jones <willjones127@gmail.com>

Update python/deltalake/schema.py

09b4a19

Co-authored-by: Will Jones <willjones127@gmail.com>

add test to check if nullability is maintained

b351739

Update python/tests/test_writer.py

3deb15b

Co-authored-by: Will Jones <willjones127@gmail.com>

use parametrized constructor to reduce tests

a86c8f3

restructure licenses

cba6105

Update schema.py

9468dae

Update table.py

3478369

Update schema.py

50c4255

Update table.py

e2beb38

ion-elgreco force-pushed the feat/add_schema_conversion_helpers branch from 3356048 to e2beb38 Compare November 24, 2023 19:18

roeap approved these changes Nov 24, 2023

View reviewed changes

ion-elgreco merged commit fa6c513 into delta-io:main Nov 24, 2023
24 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(python): add pyarrow to delta compatible schema conversion in writer/merge #1820

feat(python): add pyarrow to delta compatible schema conversion in writer/merge #1820

ion-elgreco commented Nov 7, 2023 •

edited

Loading

ion-elgreco commented Nov 7, 2023

ion-elgreco commented Nov 11, 2023

wjones127 left a comment

ion-elgreco commented Nov 17, 2023

roeap Nov 18, 2023 •

edited

Loading

wjones127 Nov 19, 2023

roeap Nov 19, 2023

ion-elgreco Nov 19, 2023

ion-elgreco Nov 21, 2023

roeap left a comment

ion-elgreco commented Nov 24, 2023

feat(python): add pyarrow to delta compatible schema conversion in writer/merge #1820

feat(python): add pyarrow to delta compatible schema conversion in writer/merge #1820

Conversation

ion-elgreco commented Nov 7, 2023 • edited Loading

Description

Related Issue(s)

ion-elgreco commented Nov 7, 2023

ion-elgreco commented Nov 11, 2023

wjones127 left a comment

Choose a reason for hiding this comment

ion-elgreco commented Nov 17, 2023

roeap Nov 18, 2023 • edited Loading

Choose a reason for hiding this comment

wjones127 Nov 19, 2023

Choose a reason for hiding this comment

roeap Nov 19, 2023

Choose a reason for hiding this comment

ion-elgreco Nov 19, 2023

Choose a reason for hiding this comment

ion-elgreco Nov 21, 2023

Choose a reason for hiding this comment

roeap left a comment

Choose a reason for hiding this comment

ion-elgreco commented Nov 24, 2023

ion-elgreco commented Nov 7, 2023 •

edited

Loading

roeap Nov 18, 2023 •

edited

Loading