-
Notifications
You must be signed in to change notification settings - Fork 378
feat: Support of branches in add_files #2485
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
@kevinjqliu @Fokko Request your review |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This pr is 🔥 ! I pulled it down and tested it on a few large tables in my own dev env - and it worked well. Really great work, I'm excited to use this in prod!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It might be nice to add a negative test that attempts to add files to a non-existent branch, just to make sure that exceptions are handled gracefully and that meaningful errors are surfaced.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It should be same as the test for appending to non-existing branch:
def test_append_to_non_existing_branch(session_catalog: Catalog, arrow_table_with_null: pa.Table) -> None: |
If seen from the POV of a snapshot, the flow for add_files and append operation is same as only a new snapshot with new files is being appended.
Since, we are not adding any different code, it just introduces another test which goes through the same flow thus increasing test time.
My suggestion would be to not bloat the test time with similar tests
Thoughts?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That makes sense to me!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Great work!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice one @vinjai 🙌 I've left one comment, but apart from that it is good to go 👍
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
slight nit on the test comment consistency
Closes #2428
Rationale for this change
Support to add files to iceberg branches. Currently, you can only add files to the main branch
Are these changes tested?
Yes
Are there any user-facing changes?
New optional paramater for
branch
in the add_files method of the Table API