Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add Page Row Count Limit #2941

Closed
Tracked by #3462
tustvold opened this issue Oct 26, 2022 · 1 comment · Fixed by #2942
Closed
Tracked by #3462

Add Page Row Count Limit #2941

tustvold opened this issue Oct 26, 2022 · 1 comment · Fixed by #2942
Labels
enhancement Any new improvement worthy of a entry in the changelog parquet Changes to the parquet crate

Comments

@tustvold
Copy link
Contributor

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

Skipping pages is often significantly cheaper than skipping values within a page, as such we want a reasonable granularity of pages within a column chunk. However, parquet's encodings can be very efficient leading to a very large number of rows in a single page before hitting the page size limit

Describe the solution you'd like

Add an optional page row count limit to the parquet writer, that limits the maximum number of rows within a page.

Describe alternatives you've considered

Additional context

Impala has such an option for the same reason
@tustvold tustvold added parquet Changes to the parquet crate enhancement Any new improvement worthy of a entry in the changelog labels Oct 26, 2022
tustvold added a commit to tustvold/arrow-rs that referenced this issue Oct 26, 2022
@tustvold
Add page row count limit (apache#2941)
fdefedf
@tustvold tustvold mentioned this issue Oct 26, 2022
Merged
@alamb
Copy link
Contributor

alamb commented Oct 27, 2022

related to apache/datafusion#3462

@alamb alamb mentioned this issue Oct 27, 2022
Open
27 tasks
tustvold added a commit that referenced this issue Oct 27, 2022
@tustvold @alamb
Add optional page row count limit for parquet WriterProperties (#2941…
880c4d9 
…) (#2942)

* Add page row count limit (#2941)

* Apply suggestions from code review

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
@alamb alamb mentioned this issue Oct 28, 2022
Merged
14 tasks
@Ted-Jiang Ted-Jiang mentioned this issue Nov 4, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement Any new improvement worthy of a entry in the changelog parquet Changes to the parquet crate
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add optional page row count limit for parquet WriterProperties (#2941) tustvold/arrow-rs
2 participants
@alamb @tustvold