Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add benchmarks for BYTE_STREAM_SPLIT encoded Parquet FIXED_LEN_BYTE_ARRAY data #6203

Closed
etseidl opened this issue Aug 6, 2024 · 1 comment · Fixed by #6204
Closed

Add benchmarks for BYTE_STREAM_SPLIT encoded Parquet FIXED_LEN_BYTE_ARRAY data #6203

etseidl opened this issue Aug 6, 2024 · 1 comment · Fixed by #6204
Labels
enhancement Any new improvement worthy of a entry in the changelog parquet Changes to the parquet crate

Comments

@etseidl
Copy link
Contributor

etseidl commented Aug 6, 2024

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
#6159 added support for using BYTE_STREAM_SPLIT with FIXED_LEN_BYTE_ARRAY primitive types. While some effort was put into optimizing the encoding path, the decoding path is largely unoptimized (and seemingly quite slow). It would be nice to have some benchmarks for the new encodings to guide future optimization efforts.

Describe the solution you'd like
Benchmarks for Float16/FIXED_LEN_BYTE_ARRAY(2) and DECIMAL/FIXED_LEN_BYTE_ARRAY(16) would be a good start for some likely to be used data types.

Describe alternatives you've considered

Additional context
See #6159 (comment) and following.

@etseidl etseidl added the enhancement Any new improvement worthy of a entry in the changelog label Aug 6, 2024
@alamb alamb added the parquet Changes to the parquet crate label Aug 31, 2024
@alamb
Copy link
Contributor

alamb commented Aug 31, 2024

label_issue.py automatically added labels {'parquet'} from #6204

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Any new improvement worthy of a entry in the changelog parquet Changes to the parquet crate
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants