Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

without_files flag for RawDeltaTable #860

Closed
MykhailoHevak opened this issue Oct 4, 2022 · 3 comments
Closed

without_files flag for RawDeltaTable #860

MykhailoHevak opened this issue Oct 4, 2022 · 3 comments
Labels
binding/python Issues for the Python package bug Something isn't working good first issue Good for newcomers

Comments

@MykhailoHevak
Copy link
Contributor

There is the method without_files in the DeltaTableBuilder

pub fn without_files(mut self) -> Self {

The method sets require_files flag in DeltaTableLoadOptions. And this flag is very useful cause it can load DeltaTable with significant memory reduction.

But python's class RawDeltaTable doesn't have the flag! And it is impossibly to handle tracking files :(

fn new(

So, it's impossible to load big DeltaTable if python's library is used :(

@MykhailoHevak MykhailoHevak added the bug Something isn't working label Oct 4, 2022
@wjones127
Copy link
Collaborator

Yes I think we could add a new flag to the DeltaTable constructor for that. Would you be willing to make a PR?

BTW, DeltaTable is the public class. RawDeltaTable is meant to be internal.

@wjones127 wjones127 added good first issue Good for newcomers binding/python Issues for the Python package labels Oct 4, 2022
@MykhailoHevak
Copy link
Contributor Author

Sure, I will do PR

@roeap
Copy link
Collaborator

roeap commented Jan 24, 2023

closed by #866, feel free to re-open if needed.

@roeap roeap closed this as completed Jan 24, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
binding/python Issues for the Python package bug Something isn't working good first issue Good for newcomers
Projects
None yet
Development

No branches or pull requests

3 participants