Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Parquet Preserve BitMask #1037

Closed
tustvold opened this issue Dec 13, 2021 · 0 comments · Fixed by #1054
Closed

Parquet Preserve BitMask #1037

tustvold opened this issue Dec 13, 2021 · 0 comments · Fixed by #1054

Comments

@tustvold
Copy link
Contributor

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

Currently the parquet reader reads out definition levels into [i16] and then constructs a bitmask from this. In the case of a max definition level of 1, the data is likely encoded already as bitpacked with a bitwidth of 1. It is possible to just use this encoded representation as is, without decoding it to [i16] and then re-encoding it as a bitmask.

FWIW parquet2 performs this optimisation

Describe the solution you'd like

Reuse the already encoded bitmask directly
@tustvold tustvold added the enhancement Any new improvement worthy of a entry in the changelog label Dec 13, 2021
This was referenced Dec 13, 2021
Closed
Closed
Merged
Merged
Closed
tustvold added a commit to tustvold/arrow-rs that referenced this issue Dec 17, 2021
@tustvold
Preserve bitmask (apache#1037)
40a3c6b
@tustvold tustvold mentioned this issue Dec 17, 2021
Merged
tustvold added a commit to tustvold/arrow-rs that referenced this issue Jan 11, 2022
@tustvold
Preserve bitmask (apache#1037)
2dc5b80
tustvold added a commit to tustvold/arrow-rs that referenced this issue Jan 11, 2022
@tustvold
Preserve bitmask (apache#1037)
c92e79b
tustvold added a commit to tustvold/arrow-rs that referenced this issue Jan 11, 2022
@tustvold
Preserve bitmask (apache#1037)
94d66ad
alamb pushed a commit that referenced this issue Jan 13, 2022
@tustvold
Improve parquet reading performance for columns with nulls by preserv…
231cf78 
…ing bitmask when possible (#1037) (#1054)

* Preserve bitmask (#1037)

* Remove now unnecessary box (#1061)

* Fix handling of empty bitmasks

* More docs

* Add nested nullability test case

* Add packed decoder test
@tustvold tustvold mentioned this issue Jan 15, 2022
Merged
@alamb alamb removed the enhancement Any new improvement worthy of a entry in the changelog label Jan 20, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
2 participants
@alamb @tustvold