[feat] Virtual Slot Ref #52701
Merged
[feat] Virtual Slot Ref #52701
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
Contributor
FE UT Coverage ReportIncrement line coverage `` 🎉 |
Contributor
Author
|
run buildall |
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
Author
|
run buildall |
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
FE UT Coverage ReportIncrement line coverage `` 🎉 |
Contributor
Author
|
run buildall |
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
Author
|
run buildall |
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
FE UT Coverage ReportIncrement line coverage `` 🎉 |
Contributor
Author
|
run buildall |
Contributor
Author
|
run buildall |
Contributor
Author
|
run buildall |
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
TPC-H: Total hot run time: 33935 ms |
TPC-DS: Total hot run time: 184427 ms |
ClickBench: Total hot run time: 30.34 s |
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
yiguolei
reviewed
Jul 4, 2025
yiguolei
reviewed
Jul 4, 2025
yiguolei
reviewed
Jul 4, 2025
zhiqiang-hhhh
force-pushed
the
feat-virtual-column
branch
from
e61396d to
25ebe62
Compare
Contributor
Author
|
run buildall |
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
ClickBench: Total hot run time: 32.96 s |
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
github-actions
bot
added
the
approved
Contributor
|
PR approved by at least one committer and no changes requested. |
w41ter
pushed a commit
to w41ter/incubator-doris
that referenced
this pull request
Jul 30, 2025
### What problem does this PR solve?
**TL;DR:** Introduce virtual slot ref to eliminate redundant computation
of common sub-expressions
### Problem to solve
Consider the following queries:
```sql
select funcC(funcA(colA)), funcB(funcA(colA)) from table;
```
```sql
select funcA(colA) as sub from table where funcB(funcA(colA)) > 0;
```
```sql
select l2_distance(colA, [10]) as distance from table where l2_distance(colA, [10]) > 0
```
The common characteristic of these SQL statements is that certain
expressions appear multiple times in different places—whether in the
projection, in predicates, or during index computation (e.g., for ANN
index in Q3). These identical repeated expressions are currently
computed multiple times, but they could actually be computed just once.
We introduce **virtual slot ref** to address this issue.
In the storage layer, we implement a `VirtualColumnIterator`. The
behavior of `VirtualColumnIterator` is identical to other
`ColumnIterator`s, except it is not used to read any physical column.
Instead, it is dedicated to reading the result of expressions computed
from the index (for example, the distance returned by an ANN index, or
in the future, the relevance score from a full-text index). Once an
expression result is computed via the index, we use
`VirtualColumnIterator::prepare_materialization` to store the data
source. If a segment does not have the corresponding index, the data
source of the `VirtualColumnIterator` will be a special `ColumnNothing`
type (this is an important design trick that allows virtual slot ref to
elegantly handle the case where a segment does not yet have the index
built).
We also modify `SegmentIterator`. Before processing each block, we first
initialize the positions of the virtual slot ref in the block as
`ColumnNothing`. Before actually returning a block, we check whether the
virtual slot ref have been materialized; if not, we execute the
expressions corresponding to the virtual slot ref (e.g., `l2_distance`
or a `score` function) to generate the actual virtual slot ref, ensuring
that every virtual column in the block returned to the computation layer
has been materialized.
For expression evaluation, we introduce `VirtualSlotRef`, which is
essentially `SlotRef` + `FunctionCall`. When the expression tree
executes a node of this type, it automatically checks whether the
corresponding expression has been materialized: if it has,
`VirtualSlotRef` behaves like a `SlotRef`; if it hasn’t, it behaves like
a `FunctionCall`.
### Modification on planner
Here’s an example to better illustrate the execution of virtual slot
ref:
```sql
select func(colA) from table where func(colA) > 0;
```
For this SQL, our current `ScanNode` is:
```
ScanNode {
predicates: func(colA[#0]) > 0
final projection: func(colA[#0])
final projection tuple id: 1
tuple_id: 0
}
TupleDesc[id=0]{
SlotDesc{id=0, col=colA)}
}
TupleDesc[id=1] {
SlotDesc{id=1, col=null, ..., type=float64)
}
```
After this pr, our plan will become:
```
ScanNode {
predicate: function(colA)[apache#1] > 0
final projection: function(colA)[apache#1]
final projection tuple id: 1
tuple_id: 0
}
TupleDesc[id=0]{
SlotDesc{id=0, col="colA")},
SlotDesc{id=1, col=virtual_column_1, expr=function1(colA[#0])}
}
TupleDesc[id=1] {
SlotDesc{id=2, name=virtual_column_1[apache#1])
}
```
Note that we added a `VirtualSlot` in Tuple 0, and other places that
originally required computing the expression are transformed to
reference this `VirtualSlot`. In this way, redundant computation of
common expressions is eliminated.
### Benchmark
disable the plan rules
```sql
mysql> set disable_nereids_rules='PUSH_DOWN_VIRTUAL_COLUMNS_INTO_OLAP_SCAN';
Query OK, 0 rows affected (0.00 sec)
mysql> SELECT counterid, Count(*) AS hit_count, Count(DISTINCT userid) AS unique_users FROM hits WHERE ( Upper(Regexp_extract(referer, '^https?://([^/]+)', 1)) = 'GOOGLE.COM' OR Upper(Regexp_extract(referer, '^https?://([^/]+)', 1)) = 'GOOGLE.RU' OR Upper(Regexp_extract(referer, '^https?://([^/]+)', 1)) LIKE '%GOOGLE%' ) AND ( Length(Regexp_extract(referer, '^https?://([^/]+)', 1)) > 3 OR Regexp_extract(referer, '^https?://([^/]+)', 1) != ''
OR Regexp_extract(referer, '^https?://([^/]+)', 1) IS NOT NULL ) AND eventdate = '2013-07-15' GROUP BY counterid HAVING hit_count > 100 ORDER BY hit_count DESC LIMIT 20;
+-----------+-----------+--------------+
| counterid | hit_count | unique_users |
+-----------+-----------+--------------+
| 105857 | 1919075 | 1412926 |
| 117917 | 200018 | 50285 |
| 99062 | 114384 | 71408 |
| 1634 | 43839 | 14975 |
| 59 | 31328 | 6668 |
| 114157 | 28852 | 19729 |
| 62 | 22549 | 14130 |
| 1483 | 8425 | 5677 |
| 38 | 5436 | 1805 |
| 1060 | 4043 | 2948 |
| 76221 | 2060 | 1325 |
| 128858 | 1690 | 825 |
| 102847 | 1500 | 350 |
| 89761 | 1419 | 274 |
| 92040 | 1180 | 978 |
| 1089 | 1067 | 961 |
| 2004 | 880 | 698 |
| 1213 | 597 | 219 |
| 77729 | 448 | 108 |
| 71099 | 289 | 70 |
+-----------+-----------+--------------+
20 rows in set (1.50 sec)
```
reopen the rule:
```text
mysql> unset variable disable_nereids_rules;
--------------
unset variable disable_nereids_rules
--------------
Query OK, 0 rows affected (0.00 sec)
mysql> -- 查询 1: 分析从 Google 中获得最多点击的 20 个网站
mysql> SELECT counterid,
-> Count(*) AS hit_count,
-> Count(DISTINCT userid) AS unique_users
-> FROM hits
-> WHERE ( Upper(Regexp_extract(referer, '^https?://([^/]+)', 1)) = 'GOOGLE.COM'
-> OR Upper(Regexp_extract(referer, '^https?://([^/]+)', 1)) =
-> 'GOOGLE.RU'
-> OR Upper(Regexp_extract(referer, '^https?://([^/]+)', 1)) LIKE
-> '%GOOGLE%' )
-> AND ( Length(Regexp_extract(referer, '^https?://([^/]+)', 1)) > 3
-> OR Regexp_extract(referer, '^https?://([^/]+)', 1) != ''
-> OR Regexp_extract(referer, '^https?://([^/]+)', 1) IS NOT NULL )
-> AND eventdate = '2013-07-15'
-> GROUP BY counterid
-> HAVING hit_count > 100
-> ORDER BY hit_count DESC
-> LIMIT 20;
+-----------+-----------+--------------+
| counterid | hit_count | unique_users |
+-----------+-----------+--------------+
| 105857 | 1919075 | 1412926 |
| 117917 | 200018 | 50285 |
| 99062 | 114384 | 71408 |
| 1634 | 43839 | 14975 |
| 59 | 31328 | 6668 |
| 114157 | 28852 | 19729 |
| 62 | 22549 | 14130 |
| 1483 | 8425 | 5677 |
| 38 | 5436 | 1805 |
| 1060 | 4043 | 2948 |
| 76221 | 2060 | 1325 |
| 128858 | 1690 | 825 |
| 102847 | 1500 | 350 |
| 89761 | 1419 | 274 |
| 92040 | 1180 | 978 |
| 1089 | 1067 | 961 |
| 2004 | 880 | 698 |
| 1213 | 597 | 219 |
| 77729 | 448 | 108 |
| 71099 | 289 | 70 |
+-----------+-----------+--------------+
20 rows in set (0.57 sec)
```
About 300% optimization.
### TODO
In the future, we can leverage virtual slot ref to implement more
functionalities, including:
1. ANN Index
2. Relevance scoring based on full-text indexes
3. Generated Columns (of NOT ALWAYS type)
4. Index-Only Scan (which will require modifying SlotRef computation in
`SegmentIterator` to be pull-based)
5. CSE replace rule on FE is very basic, but enough to use for now. Need
a further modification on
fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/PushDownVirtualColumnsIntoOlapScanTest.java
---
Co-authored-by: morrySnow <zhangwenxin@selectdb.com>
16 tasks
zhiqiang-hhhh
added a commit
to zhiqiang-hhhh/doris
that referenced
this pull request
Aug 4, 2025
**TL;DR:** Introduce virtual slot ref to eliminate redundant computation
of common sub-expressions
Consider the following queries:
```sql
select funcC(funcA(colA)), funcB(funcA(colA)) from table;
```
```sql
select funcA(colA) as sub from table where funcB(funcA(colA)) > 0;
```
```sql
select l2_distance(colA, [10]) as distance from table where l2_distance(colA, [10]) > 0
```
The common characteristic of these SQL statements is that certain
expressions appear multiple times in different places—whether in the
projection, in predicates, or during index computation (e.g., for ANN
index in Q3). These identical repeated expressions are currently
computed multiple times, but they could actually be computed just once.
We introduce **virtual slot ref** to address this issue.
In the storage layer, we implement a `VirtualColumnIterator`. The
behavior of `VirtualColumnIterator` is identical to other
`ColumnIterator`s, except it is not used to read any physical column.
Instead, it is dedicated to reading the result of expressions computed
from the index (for example, the distance returned by an ANN index, or
in the future, the relevance score from a full-text index). Once an
expression result is computed via the index, we use
`VirtualColumnIterator::prepare_materialization` to store the data
source. If a segment does not have the corresponding index, the data
source of the `VirtualColumnIterator` will be a special `ColumnNothing`
type (this is an important design trick that allows virtual slot ref to
elegantly handle the case where a segment does not yet have the index
built).
We also modify `SegmentIterator`. Before processing each block, we first
initialize the positions of the virtual slot ref in the block as
`ColumnNothing`. Before actually returning a block, we check whether the
virtual slot ref have been materialized; if not, we execute the
expressions corresponding to the virtual slot ref (e.g., `l2_distance`
or a `score` function) to generate the actual virtual slot ref, ensuring
that every virtual column in the block returned to the computation layer
has been materialized.
For expression evaluation, we introduce `VirtualSlotRef`, which is
essentially `SlotRef` + `FunctionCall`. When the expression tree
executes a node of this type, it automatically checks whether the
corresponding expression has been materialized: if it has,
`VirtualSlotRef` behaves like a `SlotRef`; if it hasn’t, it behaves like
a `FunctionCall`.
Here’s an example to better illustrate the execution of virtual slot
ref:
```sql
select func(colA) from table where func(colA) > 0;
```
For this SQL, our current `ScanNode` is:
```
ScanNode {
predicates: func(colA[#0]) > 0
final projection: func(colA[#0])
final projection tuple id: 1
tuple_id: 0
}
TupleDesc[id=0]{
SlotDesc{id=0, col=colA)}
}
TupleDesc[id=1] {
SlotDesc{id=1, col=null, ..., type=float64)
}
```
After this pr, our plan will become:
```
ScanNode {
predicate: function(colA)[#1] > 0
final projection: function(colA)[#1]
final projection tuple id: 1
tuple_id: 0
}
TupleDesc[id=0]{
SlotDesc{id=0, col="colA")},
SlotDesc{id=1, col=virtual_column_1, expr=function1(colA[#0])}
}
TupleDesc[id=1] {
SlotDesc{id=2, name=virtual_column_1[#1])
}
```
Note that we added a `VirtualSlot` in Tuple 0, and other places that
originally required computing the expression are transformed to
reference this `VirtualSlot`. In this way, redundant computation of
common expressions is eliminated.
disable the plan rules
```sql
mysql> set disable_nereids_rules='PUSH_DOWN_VIRTUAL_COLUMNS_INTO_OLAP_SCAN';
Query OK, 0 rows affected (0.00 sec)
mysql> SELECT counterid, Count(*) AS hit_count, Count(DISTINCT userid) AS unique_users FROM hits WHERE ( Upper(Regexp_extract(referer, '^https?://([^/]+)', 1)) = 'GOOGLE.COM' OR Upper(Regexp_extract(referer, '^https?://([^/]+)', 1)) = 'GOOGLE.RU' OR Upper(Regexp_extract(referer, '^https?://([^/]+)', 1)) LIKE '%GOOGLE%' ) AND ( Length(Regexp_extract(referer, '^https?://([^/]+)', 1)) > 3 OR Regexp_extract(referer, '^https?://([^/]+)', 1) != ''
OR Regexp_extract(referer, '^https?://([^/]+)', 1) IS NOT NULL ) AND eventdate = '2013-07-15' GROUP BY counterid HAVING hit_count > 100 ORDER BY hit_count DESC LIMIT 20;
+-----------+-----------+--------------+
| counterid | hit_count | unique_users |
+-----------+-----------+--------------+
| 105857 | 1919075 | 1412926 |
| 117917 | 200018 | 50285 |
| 99062 | 114384 | 71408 |
| 1634 | 43839 | 14975 |
| 59 | 31328 | 6668 |
| 114157 | 28852 | 19729 |
| 62 | 22549 | 14130 |
| 1483 | 8425 | 5677 |
| 38 | 5436 | 1805 |
| 1060 | 4043 | 2948 |
| 76221 | 2060 | 1325 |
| 128858 | 1690 | 825 |
| 102847 | 1500 | 350 |
| 89761 | 1419 | 274 |
| 92040 | 1180 | 978 |
| 1089 | 1067 | 961 |
| 2004 | 880 | 698 |
| 1213 | 597 | 219 |
| 77729 | 448 | 108 |
| 71099 | 289 | 70 |
+-----------+-----------+--------------+
20 rows in set (1.50 sec)
```
reopen the rule:
```text
mysql> unset variable disable_nereids_rules;
--------------
unset variable disable_nereids_rules
--------------
Query OK, 0 rows affected (0.00 sec)
mysql> -- 查询 1: 分析从 Google 中获得最多点击的 20 个网站
mysql> SELECT counterid,
-> Count(*) AS hit_count,
-> Count(DISTINCT userid) AS unique_users
-> FROM hits
-> WHERE ( Upper(Regexp_extract(referer, '^https?://([^/]+)', 1)) = 'GOOGLE.COM'
-> OR Upper(Regexp_extract(referer, '^https?://([^/]+)', 1)) =
-> 'GOOGLE.RU'
-> OR Upper(Regexp_extract(referer, '^https?://([^/]+)', 1)) LIKE
-> '%GOOGLE%' )
-> AND ( Length(Regexp_extract(referer, '^https?://([^/]+)', 1)) > 3
-> OR Regexp_extract(referer, '^https?://([^/]+)', 1) != ''
-> OR Regexp_extract(referer, '^https?://([^/]+)', 1) IS NOT NULL )
-> AND eventdate = '2013-07-15'
-> GROUP BY counterid
-> HAVING hit_count > 100
-> ORDER BY hit_count DESC
-> LIMIT 20;
+-----------+-----------+--------------+
| counterid | hit_count | unique_users |
+-----------+-----------+--------------+
| 105857 | 1919075 | 1412926 |
| 117917 | 200018 | 50285 |
| 99062 | 114384 | 71408 |
| 1634 | 43839 | 14975 |
| 59 | 31328 | 6668 |
| 114157 | 28852 | 19729 |
| 62 | 22549 | 14130 |
| 1483 | 8425 | 5677 |
| 38 | 5436 | 1805 |
| 1060 | 4043 | 2948 |
| 76221 | 2060 | 1325 |
| 128858 | 1690 | 825 |
| 102847 | 1500 | 350 |
| 89761 | 1419 | 274 |
| 92040 | 1180 | 978 |
| 1089 | 1067 | 961 |
| 2004 | 880 | 698 |
| 1213 | 597 | 219 |
| 77729 | 448 | 108 |
| 71099 | 289 | 70 |
+-----------+-----------+--------------+
20 rows in set (0.57 sec)
```
About 300% optimization.
In the future, we can leverage virtual slot ref to implement more
functionalities, including:
1. ANN Index
2. Relevance scoring based on full-text indexes
3. Generated Columns (of NOT ALWAYS type)
4. Index-Only Scan (which will require modifying SlotRef computation in
`SegmentIterator` to be pull-based)
5. CSE replace rule on FE is very basic, but enough to use for now. Need
a further modification on
fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/PushDownVirtualColumnsIntoOlapScanTest.java
---
Co-authored-by: morrySnow <zhangwenxin@selectdb.com>
yiguolei
pushed a commit
that referenced
this pull request
Aug 4, 2025
#54223) ### What problem does this PR solve? Related PR: #52701 TimeV2 is a runtime type, it can not be used as VirtualSlotRef. Problem Summary: ### Release note None ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [ ] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [ ] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
This was referenced Aug 5, 2025
airborne12
pushed a commit
that referenced
this pull request
Aug 18, 2025
Tablet schema that contains virtual column should not be added into SchemaCache. Related PR: #52701
airborne12
pushed a commit
that referenced
this pull request
Aug 22, 2025
### What problem does this PR solve? Introducing Ann index to doris. This pull request introduces foundational support for ANN (Approximate Nearest Neighbor) vector index functionality in the storage engine, including new runtime structures, configuration options, and initial integration with the build system. The changes lay the groundwork for ANN-based search and statistics collection, and begin integrating ANN index support into various storage and query execution paths. The implementation of ann index is based on [faiss](https://github.com/facebookresearch/faiss). Faiss could return distance directly, so this pr using [virtual slot ref](#52701) to return result from index. Each data segment of doris will have a faiss index if user creates a table with Ann index, and new segment generated by compaction will have a faiss index automatically. Currently, create index and build index is not supported, index defination be added to ddl if you want it. **ANN Index Feature Integration:** * Added new runtime structures and parameters for ANN index operations, including `AnnIndexStats`, `AnnIndexParam`, `RangeSearchParams`, `RangeSearchResult`, and others in `ann_search_params.h`, as well as `RangeSearchRuntimeInfo` for managing ANN range search context. [[1]](diffhunk://#diff-088dbea44296fb3669fe0cd22005df6aff33f8b60b20d5a49a68c6bbd22c29d1R1-R109) [[2]](diffhunk://#diff-ec34e664611a5877cab8f157919c35fe9b697428533c702536c97fd4f05769bdR1-R97) [[3]](diffhunk://#diff-d41283d91ba2756db2b45cadf964a78ad1a5c3360e1b854cb1a3d1f60817c804R1-R44) * Extended `StorageReadOptions` and `RowsetReaderContext` to include `ann_topn_runtime` for passing ANN runtime information through the storage read path. [[1]](diffhunk://#diff-8dec3cee74c5e0821835a7125bede7e3358bd3b5067b2748262193cb4cf80d48R126) [[2]](diffhunk://#diff-19fb296aa338021a0806017aa78ddadbea1791de3f4545724e34f8d9683a6551R95) [[3]](diffhunk://#diff-66ec81528c724b2f69e242d61f35d8a54d7bb5286a7e334c486166a3cc946642R106) * Added new ANN-related statistics fields (timing and row counts) to `OlapReaderStatistics` for monitoring ANN index operations. **Build System and Dependency Updates:** * Added `doris-faiss` and `doris-openblas` as submodules for ANN/vector index support, and integrated the new `Vector` library into the build process and as a dependency for relevant targets. [[1]](diffhunk://#diff-fe7afb5c9c916e521401d3fcfb4277d5071798c3baf83baf11d6071742823584R24-R31) [[2]](diffhunk://#diff-3507aac2aff9b5fe5f66d28967f3aa848491d4ced2466f6bf201ab3a97531837R532) [[3]](diffhunk://#diff-3507aac2aff9b5fe5f66d28967f3aa848491d4ced2466f6bf201ab3a97531837R787-R788) [[4]](diffhunk://#diff-d67261040b7ca84a64e8aeef5f7e1a8bab5efcc20fcdd3402f24160f56c29959R26) **Index Handling and Schema Integration:** * Updated index file writer accessors and naming from "inverted_index" to more generic "index" to accommodate ANN and other index types. [[1]](diffhunk://#diff-0c1c144f791918ef5b05ded169a7efb22a0ae67565e641cc03c31d4c2872729eL747-R748) [[2]](diffhunk://#diff-60cd05e044b4218e4a4d774abe89636fa0f6e1290dd0ff7892231d30770cd2b1L193-R193) * Changed index creation logic in `SegmentFlusher` to use `has_extra_index()` (supporting both inverted and ANN indexes) instead of `has_inverted_index()`. [[1]](diffhunk://#diff-7e9f53b4ef59bdb00d10393a2941be9201ddd46c3aab957d1dae8bc5d8898ebeL139-R139) [[2]](diffhunk://#diff-7e9f53b4ef59bdb00d10393a2941be9201ddd46c3aab957d1dae8bc5d8898ebeL157-R157) [[3]](diffhunk://#diff-7e9f53b4ef59bdb00d10393a2941be9201ddd46c3aab957d1dae8bc5d8898ebeL176-R176) [[4]](diffhunk://#diff-7e9f53b4ef59bdb00d10393a2941be9201ddd46c3aab957d1dae8bc5d8898ebeL193-R193) **Configuration:** * Introduced a new configuration option `opm_threads_limit` to control the maximum number of OpenMP threads used per Doris thread, which is relevant for vectorized/ANN computation. [[1]](diffhunk://#diff-b626e6ab16bc72abf40db76bf5094fcc8ca3c37534c2eb83b63b7805e1b601ffR1578-R1580) [[2]](diffhunk://#diff-46e8c1ada0d43acf8c2965e46e90909089aada1f46531976c10605b837f8da3dR1634-R1635) These changes set up the infrastructure required for future development of ANN vector index features, including search, filtering, and statistics collection. Co-authored-by: chenlinzhong <490103404@qq.com> Co-authored-by: morrySnow <zhangwenxin@selectdb.com>
16 tasks
16 tasks
16 tasks
zhiqiang-hhhh
pushed a commit
to zhiqiang-hhhh/doris
that referenced
this pull request
Aug 29, 2025
…pache#54998) ### What problem does this PR solve? Related PR: apache#52701 Problem Summary: 1. do not push down WhenClause in case when 2. not generate virtual column which only used once: fix the pattern like `select func_a(x), func_b(func_a(x)), func_c(func_b(func_a(x)))`.
16 tasks
englefly
pushed a commit
that referenced
this pull request
Sep 5, 2025
### What problem does this PR solve? need adjust nullable for expression of the virtual slot bug introduced by: #52701 example as follow: SQL: ```sql SELECT t1.*, t2.* FROM tbl_adjust_virtual_slot_nullable_1 AS t1 LEFT JOIN tbl_adjust_virtual_slot_nullable_2 AS t2 ON t1.c_int = t2.c_int WHERE NOT ( day(t2.c_date) IN (1, 3) AND day(t2.c_date) IN (2, 3, 3) ); ``` throw exception: ``` java.sql.SQLException: errCode = 2, detailMessage = (127.0.0.1)[INTERNAL_ERROR]Could not find function dayofmonth, arg c_date return Nullable(TINYINT) ```
wenzhenghu
pushed a commit
to wenzhenghu/doris
that referenced
this pull request
Sep 8, 2025
…#55694) ### What problem does this PR solve? need adjust nullable for expression of the virtual slot bug introduced by: apache#52701 example as follow: SQL: ```sql SELECT t1.*, t2.* FROM tbl_adjust_virtual_slot_nullable_1 AS t1 LEFT JOIN tbl_adjust_virtual_slot_nullable_2 AS t2 ON t1.c_int = t2.c_int WHERE NOT ( day(t2.c_date) IN (1, 3) AND day(t2.c_date) IN (2, 3, 3) ); ``` throw exception: ``` java.sql.SQLException: errCode = 2, detailMessage = (127.0.0.1)[INTERNAL_ERROR]Could not find function dayofmonth, arg c_date return Nullable(TINYINT) ```
16 tasks
morrySnow
pushed a commit
that referenced
this pull request
Sep 12, 2025
### What problem does this PR solve? Related: #52701 1. Functions that processing comprehensive type is too complicated, and may have many unexpected problem, eg. return `array<null_type>`, so do not process them by using virtual slot. 2. lambda function can not be processed by virtual column. So stop removing common sub expression if we meet above cases.
16 tasks
englefly
pushed a commit
that referenced
this pull request
Sep 17, 2025
### What problem does this PR solve? Related PR: #52701 1. Do not optimize grouping scalar function. 2. Fix rule type of ann topn push down.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
9 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
TL;DR: Introduce virtual slot ref to eliminate redundant computation of common sub-expressions
Problem to solve
Consider the following queries:
The common characteristic of these SQL statements is that certain expressions appear multiple times in different places—whether in the projection, in predicates, or during index computation (e.g., for ANN index in Q3). These identical repeated expressions are currently computed multiple times, but they could actually be computed just once.
We introduce virtual slot ref to address this issue.
In the storage layer, we implement a
VirtualColumnIterator. The behavior ofVirtualColumnIteratoris identical to otherColumnIterators, except it is not used to read any physical column. Instead, it is dedicated to reading the result of expressions computed from the index (for example, the distance returned by an ANN index, or in the future, the relevance score from a full-text index). Once an expression result is computed via the index, we useVirtualColumnIterator::prepare_materializationto store the data source. If a segment does not have the corresponding index, the data source of theVirtualColumnIteratorwill be a specialColumnNothingtype (this is an important design trick that allows virtual slot ref to elegantly handle the case where a segment does not yet have the index built).We also modify
SegmentIterator. Before processing each block, we first initialize the positions of the virtual slot ref in the block asColumnNothing. Before actually returning a block, we check whether the virtual slot ref have been materialized; if not, we execute the expressions corresponding to the virtual slot ref (e.g.,l2_distanceor ascorefunction) to generate the actual virtual slot ref, ensuring that every virtual column in the block returned to the computation layer has been materialized.For expression evaluation, we introduce
VirtualSlotRef, which is essentiallySlotRef+FunctionCall. When the expression tree executes a node of this type, it automatically checks whether the corresponding expression has been materialized: if it has,VirtualSlotRefbehaves like aSlotRef; if it hasn’t, it behaves like aFunctionCall.Modification on planner
Here’s an example to better illustrate the execution of virtual slot ref:
For this SQL, our current
ScanNodeis:After this pr, our plan will become:
Note that we added a
VirtualSlotin Tuple 0, and other places that originally required computing the expression are transformed to reference thisVirtualSlot. In this way, redundant computation of common expressions is eliminated.Benchmark
disable the plan rules
reopen the rule:
About 300% optimization.
TODO
In the future, we can leverage virtual slot ref to implement more functionalities, including:
SegmentIteratorto be pull-based)Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)