[feat](Nereids) after partition prune, output rows of scan node only contains rows from selected partitions #36760
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thank you for your contribution to Apache Doris. Since 2024-03-18, the Document has been moved to doris-website. |
Contributor
Author
|
run buildall |
1 similar comment
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 40201 ms |
TPC-DS: Total hot run time: 173361 ms |
ClickBench: Total hot run time: 30.7 s |
Contributor
Author
|
run p0 |
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 39513 ms |
TPC-DS: Total hot run time: 172096 ms |
ClickBench: Total hot run time: 30.12 s |
Contributor
Author
|
run p0 |
Contributor
Author
|
run feut |
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 40097 ms |
TPC-DS: Total hot run time: 170427 ms |
ClickBench: Total hot run time: 30.82 s |
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 39906 ms |
TPC-DS: Total hot run time: 173587 ms |
ClickBench: Total hot run time: 30.72 s |
morrySnow
reviewed
Jun 26, 2024
| * if this is mv, return selectedIndexId, o.w -1 | ||
| * @return -1 or selectedIndexId | ||
| */ | ||
| long getSelectedIndexIdForMV(); |
Contributor
There was a problem hiding this comment.
only for sync mv? if true, rename to ForSyncMv
Comment on lines
+64
to
+65
| public Set<Map.Entry<Expression, ColumnStatistic>> getExpressionColumnStatsEntries() { | ||
| return expressionToColumnStats.entrySet(); |
Contributor
There was a problem hiding this comment.
Directly returning a map is more general.
Contributor
Author
There was a problem hiding this comment.
We want to guarantee that every entry is add by API StatististicBuilder.put(expr, columnStats). If the map is exposed, the guarantee may be broken.
| for (Slot slot : olapScan.getOutput()) { | ||
| if (isVisibleSlotReference(slot)) { | ||
| ColumnStatistic cache = getColumnStatistic(olapTable, slot.getName(), | ||
| olapScan.getSelectedIndexIdForMV()); |
Contributor
There was a problem hiding this comment.
getSelectedIndexIdForMV() i think this is only used in statistic, so impl it in StatsCalculator.java, not in OlapScan
Contributor
Author
There was a problem hiding this comment.
analyze module also need this function
| double rowCount = getSelectedPartitionRowCount(olapScan); | ||
| // if partition row count is not available, fallback to table stats | ||
| if (rowCount > 0) { | ||
| List<String> selectedPartitionNames = new ArrayList<>(olapScan.getSelectedPartitionIds().size()); |
Contributor
There was a problem hiding this comment.
use Lists.newArrayListWithExpectedSize() to avoid resize. because list will resize when filling rate more than a threshold
Contributor
Author
There was a problem hiding this comment.
this is fixed size, no need to resize
Contributor
Author
|
run external |
morrySnow
approved these changes
Jun 26, 2024
Contributor
|
PR approved by at least one committer and no changes requested. |
github-actions
bot
added
the
approved
Contributor
|
PR approved by anyone and no changes requested. |
starocean999
approved these changes
Jun 26, 2024
dataroaring
pushed a commit
that referenced
this pull request
Jun 28, 2024
…contains rows from selected partitions (#36760) 1. update rowcount if some partitions are pruned 2. refactor StatsCalcualtor for Scan
16 tasks
morrySnow
pushed a commit
that referenced
this pull request
Dec 8, 2025
### What problem does this PR solve? Related PR: #36760 #57850 Problem Summary: Fix stats unknown when calc sync mv plan statistics For SQLs that are related to statistics, we should not collect or compute statistics. Previously this was determined by the `isInternal` flag, but `isInternal` is too broad: it covers not only statistics-related SQL but also SQL used to generate materialized view plans. Materialized view plan generation requires statistics, so we introduce a new flag `isPlanWithUnKnownColumnStats` to indicate connections that are used for statistics-only operations (treat column statistics as unknown).
github-actions bot
pushed a commit
that referenced
this pull request
Dec 8, 2025
### What problem does this PR solve? Related PR: #36760 #57850 Problem Summary: Fix stats unknown when calc sync mv plan statistics For SQLs that are related to statistics, we should not collect or compute statistics. Previously this was determined by the `isInternal` flag, but `isInternal` is too broad: it covers not only statistics-related SQL but also SQL used to generate materialized view plans. Materialized view plan generation requires statistics, so we introduce a new flag `isPlanWithUnKnownColumnStats` to indicate connections that are used for statistics-only operations (treat column statistics as unknown).
yiguolei
pushed a commit
that referenced
this pull request
Dec 11, 2025
### What problem does this PR solve? Related PR: #36760 #57850 Problem Summary: Fix stats unknown when calc sync mv plan statistics For SQLs that are related to statistics, we should not collect or compute statistics. Previously this was determined by the `isInternal` flag, but `isInternal` is too broad: it covers not only statistics-related SQL but also SQL used to generate materialized view plans. Materialized view plan generation requires statistics, so we introduce a new flag `isPlanWithUnKnownColumnStats` to indicate connections that are used for statistics-only operations (treat column statistics as unknown).
nagisa-kunhah
pushed a commit
to nagisa-kunhah/doris
that referenced
this pull request
Dec 14, 2025
…#58426) ### What problem does this PR solve? Related PR: apache#36760 apache#57850 Problem Summary: Fix stats unknown when calc sync mv plan statistics For SQLs that are related to statistics, we should not collect or compute statistics. Previously this was determined by the `isInternal` flag, but `isInternal` is too broad: it covers not only statistics-related SQL but also SQL used to generate materialized view plans. Materialized view plan generation requires statistics, so we introduce a new flag `isPlanWithUnKnownColumnStats` to indicate connections that are used for statistics-only operations (treat column statistics as unknown).
yiguolei
pushed a commit
that referenced
this pull request
Dec 18, 2025
### What problem does this PR solve? Related PR: #36760 #57850 Problem Summary: Fix stats unknown when calc sync mv plan statistics For SQLs that are related to statistics, we should not collect or compute statistics. Previously this was determined by the `isInternal` flag, but `isInternal` is too broad: it covers not only statistics-related SQL but also SQL used to generate materialized view plans. Materialized view plan generation requires statistics, so we introduce a new flag `isPlanWithUnKnownColumnStats` to indicate connections that are used for statistics-only operations (treat column statistics as unknown).
seawinde
added a commit
to seawinde/doris
that referenced
this pull request
Dec 22, 2025
…#58426) ### What problem does this PR solve? Related PR: apache#36760 apache#57850 Problem Summary: Fix stats unknown when calc sync mv plan statistics For SQLs that are related to statistics, we should not collect or compute statistics. Previously this was determined by the `isInternal` flag, but `isInternal` is too broad: it covers not only statistics-related SQL but also SQL used to generate materialized view plans. Materialized view plan generation requires statistics, so we introduce a new flag `isPlanWithUnKnownColumnStats` to indicate connections that are used for statistics-only operations (treat column statistics as unknown).
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.