[SPARK-26509][SQL] Support DELTA_BYTE_ARRAY encoding for Parquet vectorized reader #27316
Conversation
… 2.x's Vectorized Reader
|
Test build #117210 has finished for PR 27316 at commit
|
|
cc @rdblue |
| */ | ||
| public class VectorizedDeltaBinaryPackedReader extends ValuesReader | ||
| implements VectorizedValuesReader { | ||
| private final DeltaBinaryPackingValuesReader valuesReader = new DeltaBinaryPackingValuesReader(); |
There was a problem hiding this comment.
Can we reuse DeltaBinaryPackingValuesReader multiple times while I am not familiar with Parquet?
When DeltaBinaryPackinginitFromPage.initFromPage is called, valuesBuffer is allocated every time at here. On the other hand, valuesBuffer is not initialized at DeltaBinaryPackinginitFromPage.initFromPage.
I am curious about what happens if initFromPage is called multiple times.
There was a problem hiding this comment.
Thanks for the comment.
The current usage keeps the same pattern with parquet inside, DeltaBinaryPackingValuesReader is also reused in the encoding DeltaByteArrayReader and DeltaLengthByteArrayValuesReader.
|
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable. |
5 participants
What changes were proposed in this pull request?
Implement Parquet delta encoding for the vectorized interface, which is needed for V2 pages. The implementation simply delegates the decoding to the Parquet implementation.
Reference:
Parquet encodings
DefaultV2ValuesWriterFactory
Why are the changes needed?
Support for parquet DataPageV2.
Does this PR introduce any user-facing change?
How was this patch tested?
New UT.
Credit to @nandorKollar, Closes #23988.