Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

parquet: Optimized ByteArrayReader, Add UTF-8 Validation (#1040) #1082

Merged
merged 18 commits into from
Jan 18, 2022
Merged

parquet: Optimized ByteArrayReader, Add UTF-8 Validation (#1040) #1082

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
585d0f5
Optimized ByteArrayReader (#1040)
tustvold Dec 19, 2021
503f31f
Fix arrow_array_reader benchmark
tustvold Jan 16, 2022
38da212
Allow running subset of arrow_array_reader benchmarks
tustvold Jan 16, 2022
3d59a4c
Faster UTF-8 validation
tustvold Jan 16, 2022
f510f8e
Tweak null handling
tustvold Jan 16, 2022
9626728
Add license
tustvold Jan 16, 2022
014be9f
Merge remote-tracking branch 'upstream/master' into string-array-reader
tustvold Jan 16, 2022
0ba1ca7
Refine `ValuesBuffer::pad_nulls`
tustvold Jan 16, 2022
23887f5
Tweak error handling
tustvold Jan 16, 2022
7812799
Use page null count if available
tustvold Jan 16, 2022
2057048
Merge remote-tracking branch 'upstream/master' into string-array-reader
tustvold Jan 16, 2022
9f6bcac
Doc comments
tustvold Jan 16, 2022
8e27ce2
Test DELTA_BYTE_ARRAY encoding
tustvold Jan 16, 2022
a12babb
Support legacy Encoding::PLAIN_DICTIONARY
tustvold Jan 16, 2022
971622c
Add OffsetBuffer unit tests
tustvold Jan 16, 2022
dbf7b56
More tests
tustvold Jan 17, 2022
cf099c3
Fix lint
tustvold Jan 17, 2022
c941606
Review feedback
tustvold Jan 17, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
146 changes: 72 additions & 74 deletions parquet/benches/arrow_array_reader.rs
View file
Open in desktop

Large diffs are not rendered by default.

92 changes: 41 additions & 51 deletions parquet/src/arrow/array_reader.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,8 +60,7 @@ use crate::arrow::converter::{
DecimalConverter, FixedLenBinaryConverter, FixedSizeArrayConverter,
Int96ArrayConverter, Int96Converter, IntervalDayTimeArrayConverter,
IntervalDayTimeConverter, IntervalYearMonthArrayConverter,
IntervalYearMonthConverter, LargeBinaryArrayConverter, LargeBinaryConverter,
LargeUtf8ArrayConverter, LargeUtf8Converter,
IntervalYearMonthConverter, Utf8ArrayConverter, Utf8Converter,
};
use crate::arrow::record_reader::buffer::{ScalarValue, ValuesBuffer};
use crate::arrow::record_reader::{GenericRecordReader, RecordReader};
Expand All @@ -81,6 +80,11 @@ use crate::schema::types::{
};
use crate::schema::visitor::TypeVisitor;

mod byte_array;
mod offset_buffer;

pub use byte_array::make_byte_array_reader;

/// Array reader reads parquet data into arrow array.
pub trait ArrayReader {
fn as_any(&self) -> &dyn Any;
Expand Down Expand Up @@ -1749,57 +1753,43 @@ impl<'a> ArrayReaderBuilder {
null_mask_only,
)?,
)),
PhysicalType::BYTE_ARRAY => {
if cur_type.get_basic_info().converted_type() == ConvertedType::UTF8 {
if let Some(ArrowType::LargeUtf8) = arrow_type {
let converter =
LargeUtf8Converter::new(LargeUtf8ArrayConverter {});
Ok(Box::new(ComplexObjectArrayReader::<
ByteArrayType,
LargeUtf8Converter,
>::new(
page_iterator,
column_desc,
converter,
arrow_type,
)?))
} else {
use crate::arrow::arrow_array_reader::{
ArrowArrayReader, StringArrayConverter,
};
let converter = StringArrayConverter::new();
Ok(Box::new(ArrowArrayReader::try_new(
*page_iterator,
column_desc,
converter,
arrow_type,
)?))
PhysicalType::BYTE_ARRAY => match arrow_type {
// TODO: Replace with optimised dictionary reader (#171)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See #1180

Some(ArrowType::Dictionary(_, _)) => {
match cur_type.get_basic_info().converted_type() {
ConvertedType::UTF8 => {
let converter = Utf8Converter::new(Utf8ArrayConverter {});
Ok(Box::new(ComplexObjectArrayReader::<
ByteArrayType,
Utf8Converter,
>::new(
page_iterator,
column_desc,
converter,
arrow_type,
)?))
}
_ => {
let converter = BinaryConverter::new(BinaryArrayConverter {});
Ok(Box::new(ComplexObjectArrayReader::<
ByteArrayType,
BinaryConverter,
>::new(
page_iterator,
column_desc,
converter,
arrow_type,
)?))
}
}
} else if let Some(ArrowType::LargeBinary) = arrow_type {
let converter =
LargeBinaryConverter::new(LargeBinaryArrayConverter {});
Ok(Box::new(ComplexObjectArrayReader::<
ByteArrayType,
LargeBinaryConverter,
>::new(
page_iterator,
column_desc,
converter,
arrow_type,
)?))
} else {
let converter = BinaryConverter::new(BinaryArrayConverter {});
Ok(Box::new(ComplexObjectArrayReader::<
ByteArrayType,
BinaryConverter,
>::new(
page_iterator,
column_desc,
converter,
arrow_type,
)?))
}
}
_ => make_byte_array_reader(
page_iterator,
column_desc,
arrow_type,
tustvold marked this conversation as resolved.
Show resolved Hide resolved
null_mask_only,
),
},
PhysicalType::FIXED_LEN_BYTE_ARRAY
if cur_type.get_basic_info().converted_type()
== ConvertedType::DECIMAL =>
Expand Down
Loading