GH-40592: [C++][Parquet] Implement SizeStatistics #40594
Conversation
github-actions
bot
added
Component: Parquet
Component: C++
awaiting review
github-actions
bot
added
awaiting changes
github-actions
bot
added
awaiting change review
github-actions
bot
added
awaiting changes
github-actions
bot
added
awaiting change review
github-actions
bot
added
awaiting changes
There was a problem hiding this comment.
LGTM, I'm OK with this as long as @pitrou is thank you for driving this.
|
@emkornfield @mapleFU Thanks for the feedback! I haven't addressed all comments from @pitrou yet. Will let you know once ready for review again. |
github-actions
bot
added
awaiting change review
wgtmac
force-pushed
the
size_stats
branch
2 times, most recently
from
0449426
to
a83ed41
Compare
|
@pitrou @emkornfield @mapleFU Gentle ping :) |
cpp/src/parquet/column_writer.cc
Outdated
| page_statistics_->Update(*referenced_dictionary, /*update_counts=*/false); | ||
| } | ||
| if (page_size_stats_builder_) { | ||
| page_size_stats_builder_->AddValues(*referenced_dictionary); |
There was a problem hiding this comment.
So we are dictionary-decoding the entire array just to run basic statistics? This seems incredibly wasteful.
There was a problem hiding this comment.
I dislike the approach as you did. It seems that it has been used for collecting page statistics already for a long time. Do you think it is better to fix it in a separate PR or just do it in this one altogether?
There was a problem hiding this comment.
Either would be fine with me, so whatever is more practical to you :-)
There was a problem hiding this comment.
I found the history for the current approach: https://issues.apache.org/jira/browse/ARROW-12513. It pays to collect precise min/max stats from a dictionary-encoded arrow array. This reason also applies to size stats.
@emkornfield may provide more context.
|
I have adopted your suggestion to let encoders take the responsibility of counting unencoded data bytes and simplified SizeStatistics interface to a great extent. Please take a look again. Thanks! @pitrou |
Rationale for this change
Parquet format 2.10.0 has introduced SizeStatistics. parquet-mr has also implemented this: apache/parquet-java#1177. Now it is time for parquet-cpp to pick the ball.
What changes are included in this PR?
Implement reading and writing size statistics for parquet-cpp.
Are these changes tested?
Yes, a bunch of test cases have been added.
Are there any user-facing changes?
Yes, now parquet users are able to read and write size statistics.