Specialize FixedLenByteArrayReader for Parquet #2318
Labels
enhancement
Any new improvement worthy of a entry in the changelog
parquet
Changes to the parquet crate
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
We have optimised implementations for reading most types directly to their arrow representations, however, we fallback to reading
FixedLenByteArray
and then usingComplexObjectArrayReader
to cast it to the desired type.This is problematic for a couple of reasons:
ComplexObjectArrayReader
- Remove ComplexObjectArrayReader #1661FixedLenByteArray
has aBox<[u8]>
which is very expensiveDescribe the solution you'd like
I would like an optimized
FixedLenByteArrayReader<const N: usize>
that can be used to efficiently read values where the type is known up front, e.g. decimals and intervals. We can fallback to usingFixedLenByteArray
for the general case.Describe alternatives you've considered
We could not do this
Additional context
The text was updated successfully, but these errors were encountered: