Core: Use scan API to read partition stats #14989
Conversation
| && Objects.equals(stats1.lastUpdatedSnapshotId(), stats2.lastUpdatedSnapshotId()); | ||
| } | ||
|
|
||
| @SuppressWarnings("checkstyle:CyclomaticComplexity") |
There was a problem hiding this comment.
This is temporarily needed because the tests write stats using PartitionStatistics, however the tests use still use PartitionStats on the read path. When comparing expectations with actual stats we need this function. Will be dropped with the follow-up PR
|
|
||
| private static final int STATS_COUNT = 13; | ||
|
|
||
| public BasePartitionStatistics(StructLike partition, int specId) { |
There was a problem hiding this comment.
Could this be package private?
| } | ||
|
|
||
| /** Used by internal readers to instantiate this class with a projection schema. */ | ||
| public BasePartitionStatistics(Types.StructType projection) { |
| value, | ||
| (existingEntry, newEntry) -> { | ||
| existingEntry.appendStats(newEntry); | ||
| ((BasePartitionStatistics) existingEntry).appendStats(newEntry); |
There was a problem hiding this comment.
I don't really like this. Why are we sure that this is an instance of BasePartitionStatistics?
There was a problem hiding this comment.
Refactored the code to move the liveEntry(), appendStats() etc. functions into PartitionStatsHandler and populate the stats there using the setter inherited from StructLike. This eliminated the need for the casts.
gaborkaszab
force-pushed
the
main_use_partition_stat_scan_api
branch
from
c2b4a14 to
f9beb4a
Compare
gaborkaszab
force-pushed
the
main_use_partition_stat_scan_api
branch
from
f9beb4a to
584f943
Compare
gaborkaszab
force-pushed
the
main_use_partition_stat_scan_api
branch
from
584f943 to
85124af
Compare
|
Merged to main. |
2 participants
#14640 introduced a new api to scan partition stats. The next step is to replace the usage of the old reader function to the new stat api. The work is divided into 2 steps:
PartitionStatsHandler.readPartitionStatsFile()in the prod code. See the tests still pass