Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(python): expose large_dtype param in merge #2003

Merged
merged 2 commits into from
Jan 2, 2024
Merged

feat(python): expose large_dtype param in merge #2003

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
2590458
expose large_dtype
ion-elgreco Dec 30, 2023
9e0365f
Merge branch 'main' into feat/expose_large_dtype_merge
rtyler Jan 2, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 7 additions & 5 deletions python/deltalake/table.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -808,6 +808,7 @@ def merge(
target_alias: Optional[str] = None,
error_on_type_mismatch: bool = True,
writer_properties: Optional[WriterProperties] = None,
large_dtypes: bool = True,
) -> "TableMerger":
"""Pass the source data which you want to merge on the target delta table, providing a
predicate in SQL query like format. You can also specify on what to do when the underlying data types do not
Expand All @@ -820,6 +821,7 @@ def merge(
target_alias: Alias for the target table
error_on_type_mismatch: specify if merge will return error if data types are mismatching :default = True
writer_properties: Pass writer properties to the Rust parquet writer
large_dtypes: If True, the data schema is kept in large_dtypes.
Returns:
TableMerger: TableMerger Object
Expand All @@ -835,16 +837,16 @@ def merge(
)

if isinstance(source, pyarrow.RecordBatchReader):
source = convert_pyarrow_recordbatchreader(source, large_dtypes=True)
source = convert_pyarrow_recordbatchreader(source, large_dtypes)
elif isinstance(source, pyarrow.RecordBatch):
source = convert_pyarrow_recordbatch(source, large_dtypes=True)
source = convert_pyarrow_recordbatch(source, large_dtypes)
elif isinstance(source, pyarrow.Table):
source = convert_pyarrow_table(source, large_dtypes=True)
source = convert_pyarrow_table(source, large_dtypes)
elif isinstance(source, ds.Dataset):
source = convert_pyarrow_dataset(source, large_dtypes=True)
source = convert_pyarrow_dataset(source, large_dtypes)
elif isinstance(source, pandas.DataFrame):
source = convert_pyarrow_table(
pyarrow.Table.from_pandas(source), large_dtypes=True
pyarrow.Table.from_pandas(source), large_dtypes
)
else:
raise TypeError(
Expand Down
Loading