feat: prune partitions by internal column predicates.

[RinChanNOWWW]

I hereby agree to the terms of the CLA available at: https://databend.rs/dev/policies/cla/

Summary

If there are internal columns in predicates (WHERE clause), we can prune partitions by this information.

mysql> explain select watchid from hits where  _segment_name = '1/150452/_sg/6d87fc741d4c4a5fb5e94786d4bc023d_v3.bincode';
+-------------------------------------------------------------------------------------------------------------------------------------------+
| explain                                                                                                                                   |
+-------------------------------------------------------------------------------------------------------------------------------------------+
| EvalScalar                                                                                                                                |
| ├── expressions: [hits.watchid (#0)]                                                                                                      |
| ├── estimated rows: 33332499.00                                                                                                           |
| └── Filter                                                                                                                                |
|     ├── filters: [_segment_name (#105) = '1/150452/_sg/6d87fc741d4c4a5fb5e94786d4bc023d_v3.bincode']                                      |
|     ├── estimated rows: 33332499.00                                                                                                       |
|     └── TableScan                                                                                                                         |
|         ├── table: default.default.hits                                                                                                   |
|         ├── read rows: 2457416                                                                                                            |
|         ├── read bytes: 19661141                                                                                                          |
|         ├── partitions total: 770                                                                                                         |
|         ├── partitions scanned: 19                                                                                                        |
|         ├── pruning stats: [segments: <range pruning: 1 to 1>, blocks: <range pruning: 19 to 19, bloom pruning: 19 to 19>]                |
|         ├── push downs: [filters: [_segment_name (#105) = '1/150452/_sg/6d87fc741d4c4a5fb5e94786d4bc023d_v3.bincode'], limit: NONE]       |
|         ├── output columns: [watchid, _segment_name]                                                                                      |
|         └── estimated rows: 99997497.00                                                                                                   |
+-------------------------------------------------------------------------------------------------------------------------------------------+

mysql> explain select watchid from hits where  _segment_name = '1/150452/_sg/6d87fc741d4c4a5fb5e94786d4bc023d_v3.bincode' and _block_name = '1/150452/_b/b3892fa880234074a17bb7d114c52df3_v2.parquet';
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| explain                                                                                                                                                                                                                                                                            |
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| EvalScalar                                                                                                                                                                                                                                                                         |
| ├── expressions: [hits.watchid (#0)]                                                                                                                                                                                                                                               |
| ├── estimated rows: 11110833.00                                                                                                                                                                                                                                                    |
| └── Filter                                                                                                                                                                                                                                                                         |
|     ├── filters: [_segment_name (#105) = '1/150452/_sg/6d87fc741d4c4a5fb5e94786d4bc023d_v3.bincode', _block_name (#106) = '1/150452/_b/b3892fa880234074a17bb7d114c52df3_v2.parquet']                                                                                               |
|     ├── estimated rows: 11110833.00                                                                                                                                                                                                                                                |
|     └── TableScan                                                                                                                                                                                                                                                                  |
|         ├── table: default.default.hits                                                                                                                                                                                                                                            |
|         ├── read rows: 136441                                                                                                                                                                                                                                                      |
|         ├── read bytes: 1091630                                                                                                                                                                                                                                                    |
|         ├── partitions total: 770                                                                                                                                                                                                                                                  |
|         ├── partitions scanned: 1                                                                                                                                                                                                                                                  |
|         ├── pruning stats: [segments: <range pruning: 1 to 1>, blocks: <range pruning: 1 to 1, bloom pruning: 1 to 1>]                                                                                                                                                             |
|         ├── push downs: [filters: [and_filters(CAST(_segment_name (#105) = '1/150452/_sg/6d87fc741d4c4a5fb5e94786d4bc023d_v3.bincode' AS Boolean NULL), CAST(_block_name (#106) = '1/150452/_b/b3892fa880234074a17bb7d114c52df3_v2.parquet' AS Boolean NULL))], limit: NONE]       |
|         ├── output columns: [watchid, _segment_name, _block_name]                                                                                                                                                                                                                  |
|         └── estimated rows: 99997497.00                                                                                                                                                                                                                                            |
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

mysql> explain select _segment_name from hits where substring(_segment_name, -7) = 'xxx' limit 10;
+------------------------------------------------------------------------------------------------------+
| explain                                                                                              |
+------------------------------------------------------------------------------------------------------+
| EvalScalar                                                                                           |
| ├── expressions: [_segment_name (#105)]                                                              |
| ├── estimated rows: 10.00                                                                            |
| └── Limit                                                                                            |
|     ├── limit: 10                                                                                    |
|     ├── offset: 0                                                                                    |
|     ├── estimated rows: 10.00                                                                        |
|     └── Filter                                                                                       |
|         ├── filters: [substr(_segment_name (#105), -7) = 'xxx']                                      |
|         ├── estimated rows: 33332499.00                                                              |
|         └── TableScan                                                                                |
|             ├── table: default.default.hits                                                          |
|             ├── read rows: 0                                                                         |
|             ├── read bytes: 0                                                                        |
|             ├── partitions total: 770                                                                |
|             ├── partitions scanned: 0                                                                |
|             ├── push downs: [filters: [substr(_segment_name (#105), -7) = 'xxx'], limit: NONE]       |
|             ├── output columns: [_segment_name]                                                      |
|             └── estimated rows: 99997497.00                                                          |
+------------------------------------------------------------------------------------------------------+

LIMITS:

• Only _segment_name and _block_name are supported.
• If one predicate (each part of the conjunction) does not only contain the internal column, it's invalid for pruning. For example,

select * from t where t.a = _block_name and substring(_segment_name, -10) = 'v3.bincode;'

, the first predicate is invalid while the second one is valid.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
databend	⬜️ Ignored (Inspect)	Visit Preview		Apr 28, 2023 1:25pm

[sundy-li]

_segment_name seems to be too long, how about making it just be file name without prefixes and suffixes, like 6d87fc741d4c4a5fb5e94786d4bc023d_v3