Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DeltaBitPackEncoder Pads Miniblock BitWidths With Arbitrary Values #1416

Closed
tustvold opened this issue Mar 10, 2022 · 0 comments · Fixed by #1418
Closed

DeltaBitPackEncoder Pads Miniblock BitWidths With Arbitrary Values #1416

tustvold opened this issue Mar 10, 2022 · 0 comments · Fixed by #1418
Labels
bug parquet Changes to the parquet crate

Comments

@tustvold
Copy link
Contributor

Describe the bug

https://github.com/apache/arrow-rs/blob/master/parquet/src/encodings/encoding.rs#L577 skips over the miniblock bit widths, and then only goes back and writes a value for the miniblocks that contain a non-zero number of values. The empty miniblocks are left with whatever value happens to be in the encoder's buffer.

To Reproduce

This is one of the underlying bugs behind apache/datafusion#1976

Expected behavior

Whilst the specification technically allows for arbitrary padding, it seems like a good idea to avoid non-deterministic output where possible
@tustvold tustvold added the bug label Mar 10, 2022
@tustvold tustvold mentioned this issue Mar 10, 2022
Closed
tustvold added a commit to tustvold/arrow-rs that referenced this issue Mar 10, 2022
@tustvold
Consistent DeltaBitPackEncoder bit width padding (apache#1416)
173e897 
Ignore non-zero padded bit widths in DeltaBitPackDecoder (apache#1417)
tustvold added a commit to tustvold/arrow-rs that referenced this issue Mar 10, 2022
@tustvold
Consistent DeltaBitPackEncoder bit width padding (apache#1416)
5848020 
Ignore non-zero padded bit widths in DeltaBitPackDecoder (apache#1417)
tustvold added a commit to tustvold/arrow-rs that referenced this issue Mar 10, 2022
@tustvold
Consistent DeltaBitPackEncoder bit width padding (apache#1416)
416dedb 
Ignore non-zero padded bit widths in DeltaBitPackDecoder (apache#1417)
@tustvold tustvold mentioned this issue Mar 10, 2022
Merged
tustvold added a commit to tustvold/arrow-rs that referenced this issue Mar 10, 2022
@tustvold
Consistent DeltaBitPackEncoder bit width padding (apache#1416)
44d9d88 
Ignore non-zero padded bit widths in DeltaBitPackDecoder (apache#1417)
alamb pushed a commit that referenced this issue Mar 14, 2022
@tustvold
Fix DeltaBitPack MiniBlock Bit Width Padding (#1418)
40ef4aa 
* Consistent DeltaBitPackEncoder bit width padding (#1416)

Ignore non-zero padded bit widths in DeltaBitPackDecoder (#1417)

* chore: review feedback

* Add test of DeltaBitPackDecoder padding

* Revert formatting
@alamb alamb added the parquet Changes to the parquet crate label Mar 17, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug parquet Changes to the parquet crate
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix DeltaBitPack MiniBlock Bit Width Padding tustvold/arrow-rs
2 participants
@alamb @tustvold