Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(rust): expose WriterProperties method on RecordBatchWriter and DeltaWriter #1497

Merged
merged 7 commits into from
Jun 28, 2023
Merged

feat(rust): expose WriterProperties method on RecordBatchWriter and DeltaWriter #1497

merged 7 commits into from
Jun 28, 2023

Conversation

theelderbeever
Copy link
Contributor

@theelderbeever theelderbeever commented Jun 27, 2023 •
edited
Loading

Description

Adds the capability to pass a configured WriterProperties to the RecordBatchWriter and DeltaWriter similar to how the OptimizeBuilder can be updated.

Related Issue(s)

Documentation

@github-actions github-actions bot added binding/rust Issues for the Rust crate rust labels Jun 27, 2023
@github-actions
Copy link

ACTION NEEDED

delta-rs follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

@theelderbeever
Copy link
Contributor Author

This is my first time contributing to a Rust repo so happy for any and all feedback and to be told I am on the completely wrong path.

@theelderbeever theelderbeever changed the title Expose WriterProperties method on RecordBatchWriter and DeltaWriter feat(rust): Expose WriterProperties method on RecordBatchWriter and DeltaWriter Jun 27, 2023
@theelderbeever theelderbeever changed the title feat(rust): Expose WriterProperties method on RecordBatchWriter and DeltaWriter feat(rust): expose WriterProperties method on RecordBatchWriter and DeltaWriter Jun 27, 2023
@theelderbeever
Copy link
Contributor Author

Not sure why the python lint is suddenly not passing... Seems to be a mypy issue with a # type: ignore that it is incorrectly saying isn't necessary. There is a global workaround by changing updating the make check-python command to run mypy --no-warn-unused-ignores.

@github-actions github-actions bot added the binding/python Issues for the Python package label Jun 27, 2023
@wjones127
Copy link
Collaborator

@theelderbeever don't worry about the mypy. It's from a new version of the checker. I have a fix in another PR, so that'll be green once that PR is merged.

@theelderbeever
Copy link
Contributor Author

@wjones127 Gotcha makes sense. I can revert my last commit that adds the ignore flag then.

@github-actions github-actions bot removed the binding/python Issues for the Python package label Jun 28, 2023
@wjones127 wjones127 force-pushed the enhancement/add-writer-properties-to-record-batch-writer branch from aaf39f3 to ad6c8f8 Compare June 28, 2023 20:59
wjones127
wjones127 approved these changes Jun 28, 2023
View reviewed changes
Copy link
Collaborator

@wjones127 wjones127 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for doing this. Looks good to me!

rust/examples/recordbatch-writer.rs
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for adding this to the example :)

@wjones127 wjones127 merged commit d00b620 into delta-io:main Jun 28, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@wjones127 wjones127 wjones127 approved these changes

@houqp houqp Awaiting requested review from houqp

@xianwill xianwill Awaiting requested review from xianwill

@fvaleye fvaleye Awaiting requested review from fvaleye

@roeap roeap Awaiting requested review from roeap

@rtyler rtyler Awaiting requested review from rtyler

@mosyp mosyp Awaiting requested review from mosyp

Assignees
No one assigned
Labels
binding/rust Issues for the Rust crate rust
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Expose Compression Options or WriterProperties for writing to Delta Support configuring parquet compression
2 participants
@theelderbeever @wjones127