-
Notifications
You must be signed in to change notification settings - Fork 416
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix(python): wrong batch size #2314
Conversation
I'm struggling to understand this change and how it relates to the issue, I don't see how WriterProperties are sent in the Rust writer, or how removing this argument fixes things, maybe it just clears up confusion? The |
@rtyler when I built the rust engine on the python side, I misinterpreted the max row group size as the batch size. The actual max_batch_size and max_row_group_size can be set with the python WriterProperies class. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is there a test on the current Rust writer for max_rows_per_group
? Is it being set through writer_properties: Option<HashMap<String, Option<String>>>,
?
@wjones127 There are some tests here: https://github.com/delta-io/delta-rs/blob/main/python/tests/test_writerproperties.py. Setting the max_row_group_size inside WriterProperties works fine, see here: #2309 (comment). I just by accident pass |
Description
Was passing the wrong param