-
Notifications
You must be signed in to change notification settings - Fork 413
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Added without_files flag to DeltaTable constructor #866
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for submitting this! I have a few suggested changes.
What's the behavior of DeltaTable.file_uris()
when without_files=True
? Does it return empty? Or raise an error? Or load the files? We might want to add a unit test for that in Python.
Just checked and it returns an empty list. We should document that for now, and consider changing that behavior in the future (#867). |
@MykhailoHevak looks like the merge you pushed overwrote the changes you accepted in the GitHub UI. Could you add those back? |
4d784a9
to
629f503
Compare
Oooops... I did it accidentally. Is it OK now? |
The docstring change is still missing. |
Do I have to do something else? |
Thanks @MykhailoHevak! |
@wjones127 is there an estimation on when the new release is planned? P.S. closes #860 |
Hey @MykhailoHevak I believe this is your first PR to the delta repo. Can you ping me (dennyglee) via the Delta Users slack so we can send you some swag? Thanks! |
Description
Added without_files flag to DeltaTable constructor. If the flag is set it will be called method without_files in the DeltaTableBuilder. And this allows users to control memory usage. Some append-only applications might have no need of tracking any files.
Documentation
Indicates whether DeltaTable should not track files. This defaults to "false" (files are tracked). Some append-only applications might have no need of tracking any files. So, DeltaTable will be loaded with significant memory reduction.