WIP: add datafusion based parquet reader #312
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jiacai2050
force-pushed
the
new-parquet-reader
branch
from
2a50baf
to
b303920
Compare
jiacai2050
force-pushed
the
new-parquet-reader
branch
4 times, most recently
from
e38659a
to
4f4de4f
Compare
jiacai2050
force-pushed
the
new-parquet-reader
branch
6 times, most recently
from
13d99c0
to
c557007
Compare
jiacai2050
force-pushed
the
new-parquet-reader
branch
9 times, most recently
from
6115035
to
6243fcf
Compare
ShiKaiWi
reviewed
Oct 25, 2022
| @@ -264,7 +264,14 @@ mod tests { | |||
| }; | |||
|
|
|||
| let mut reader = ParquetSstReader::new(&sst_file_path, &store, &sst_reader_options); | |||
| assert_eq!(reader.meta_data().await.unwrap(), &sst_meta); | |||
| let sst_meta_readback = { | |||
| // size of SstMetaData is not what this file's size, so overwrite it | |||
|
|
||
| use super::encoding::{self, ParquetDecoder}; |
There was a problem hiding this comment.
Avoid the relative importing path.
| self.metadata_size_hint, self.cache_hit, self.cache_miss, self.metrics.bytes_scanned.value() | ||
| ); | ||
| } | ||
| } |
common_types/src/time.rs
Outdated
|
|
||
| /// Creates expression like: | ||
| /// start <= time && time < end | ||
| pub fn df_expr(&self, column_name: impl AsRef<str>) -> Expr { |
There was a problem hiding this comment.
Suggested change
| pub fn df_expr(&self, column_name: impl AsRef<str>) -> Expr { | |
| pub fn to_df_expr(&self, column_name: impl AsRef<str>) -> Expr { |
jiacai2050
force-pushed
the
new-parquet-reader
branch
3 times, most recently
from
c1f7c51
to
6d3851d
Compare
jiacai2050
force-pushed
the
new-parquet-reader
branch
from
6d3851d
to
1427794
Compare
|
In my local environment, the performance have regression when adopt this new reader, so further investigation is required before merge this. Tested sst file: 104,022,899 rows old: 11709ms(this exclude file read, since it's read out in advance) Related issue: apache/arrow-rs#2916 |
jiacai2050
changed the title
This was referenced Oct 31, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #291
Rationale for this change
As described in #291, this PR also fix object store cache isn't working.
After #14, parquet reader will read all bytes out, ignoring whether if it's already cached.
What changes are included in this PR?
Replace hand-rolled parquet reader with datafusion's ParquetExec, and add
CachableParquetFileReaderto implement row-group level cacheAre there any user-facing changes?
No
How does this change test
Using existing UT