Add framework of mv selector #4014

EmmyMiao87 · 2020-07-03T12:53:52Z

This commit mainly supports creating bitmap_union, hll_union, and count materialized views.

The main changes are as follows:

When creating a materialized view, doris judge the semantic analysis of the newly supported aggregate function.
Only bitmap_union(to_bitmap(column)), hll_union(hll_hash(column)) and count(column) are supported.
Match the correct materialized view when querying.
After the user sends the query, if there is a possibility of matching the materialized view, the query will be rewritten firstly.
Such as:
Table: k1 int, k2 int
MV: k1 int, mv_bitmap_union_k2 bitmap mv_bitmap_union
mv_bitmap_union = to_bitmap(k2)
Query: select k1, count(distinct k2) from Table
Found that there is a match between the materialized view column and the query column, the query is rewritten as:
Rewritten query: select k1, bitmap_union_count(mv_bitmap_union_k2) from table

Then when the materialized view is matched, it can be matched to the query materialized view table.
Sometimes the rewritten query may not match any materialized view, which means that the rewriting failed. The query needs to be re-parsed and executed again.

fe/src/main/java/org/apache/doris/analysis/MVColumnHLLUnionPatten.java

fe/src/main/java/org/apache/doris/analysis/CreateMaterializedViewStmt.java

fe/src/main/java/org/apache/doris/analysis/QueryStmt.java

fe/src/main/java/org/apache/doris/planner/SingleNodePlanner.java

fe/src/main/java/org/apache/doris/qe/StmtExecutor.java

fe/src/main/java/org/apache/doris/rewrite/mvrewrite/CountFieldToSum.java

fe/src/main/java/org/apache/doris/rewrite/mvrewrite/HLLHashToSlotRefRule.java

fe/src/main/java/org/apache/doris/rewrite/mvrewrite/MVExprEquivalent.java

fe/src/main/java/org/apache/doris/rewrite/mvrewrite/SlotRefEqualRule.java

fe/src/main/java/org/apache/doris/rewrite/mvrewrite/ToBitmapToSlotRefRule.java

wutiangan · 2020-07-08T02:41:13Z

fe/src/main/java/org/apache/doris/analysis/MVColumnOneChildPattern.java

+            return true;
+        } else {
+            return false;
+        }


41-47 can rewrite:
SlotRef slotRef = functionCallExpr.getChild(0).unwrapSlotRef();
return slotRef != null;

wutiangan · 2020-07-08T03:15:16Z

fe/src/main/java/org/apache/doris/analysis/CreateMaterializedViewStmt.java

+        FN_NAME_TO_PATTERN.put("count", new MVColumnOneChildPattern("count"));
+        FN_NAME_TO_PATTERN.put("bitmap_union", new MVColumnBitmapUnionPattern());
+        FN_NAME_TO_PATTERN.put("hll_union", new MVColumnHLLUnionPattern());
+    }


Can you add comment which pattern is suitable for AggregateModel, which pattern is suitable for DuplicatedModel? if it is a need to add a type to distinguish the case.

The patterns in here do not distinguish Agg model and duplicate model. These patterns are compatible.

fe/src/main/java/org/apache/doris/analysis/MVColumnItem.java

HangyuanLiu

LGTM

morningman

LGTM

…a materialized view. This commit mainly supports creating bitmap_union, hll_union, and count materialized views. * The main changes are as follows: 1. When creating a materialized view, doris judge the semantic analysis of the newly supported aggregate function. Only bitmap_union(to_bitmap(column)), hll_union(hll_hash(column)) and count(column) are supported. 2. Match the correct materialized view when querying. After the user sends the query, if there is a possibility of matching the materialized view, the query will be rewritten firstly. Such as: Table: k1 int, k2 int MV: k1 int, mv_bitmap_union_k2 bitmap mv_bitmap_union mv_bitmap_union = to_bitmap(k2) Query: select k1, count(distinct k2) from Table Found that there is a match between the materialized view column and the query column, the query is rewritten as: Rewritten query: select k1, bitmap_union_count(mv_bitmap_union_k2) from table Then when the materialized view is matched, it can be matched to the query materialized view table. Sometimes the rewritten query may not match any materialized view, which means that the rewriting failed. The query needs to be re-parsed and executed again. Change-Id: I96a7522c4e6a58150b0df1886cf22c895c1c2bad

Change-Id: I4c8568d4b57384c342eaf5432856eaaa7393f49d

Change-Id: Ibdc44e527279308d6cad77aa7f1d230d27abd02e

Change-Id: I313b896c20a210ce5a2d9ccbc5ca6d8e78ade3e7

Change-Id: I897301af916b96377664df2d321aad3c4a2b24e1

Change-Id: Ibf277124144e012773de310a17cb5cd57cdc76de

Change-Id: I8feab7ac56aa17412bbf45b15b41754f3f3b7410

Change-Id: I73bd860091b14202acb731a0eab460d992237f03

Change-Id: Ib8ab5fa56b1caec597937f6bf3defa4acfe8aab1

Change-Id: If3e553ebb29e59b0e35ed4423094ad8caaa710ac

Change-Id: Id867098ded33667a0f80f67daf88ee068a1973a0

Change-Id: I3d5481fab5a09bf5f43b606d1fe0bda41eb2e17b

Change-Id: I38273603ae21245b38e0aa5dd2b266aceab06575

Change-Id: Id42b9102585901e647d56601e61c0a5d53a7208c

Change-Id: I6e16b29f8cdcd8fa4772d235fbeddae6b7ec9511

Change-Id: I6656a91208fa58bd870977a6a873bcd9cf9893c9

Change-Id: Ib034a61801fa2d9c0eb566fa5f67f20fc065aea2

Change-Id: Id9106590c48052ace64eea26831f7e283dddb0de

morningman

LGTM

Change-Id: I45a957598f0a4f8bbc3c8404530f86a614fc9890

morningman

LGTM

)

EmmyMiao87 added area/materialized-view Issues or PRs related to materialized view kind/feature Categorizes issue or PR as related to a new feature. labels Jul 3, 2020

EmmyMiao87 added this to the 0.13.0 milestone Jul 3, 2020

EmmyMiao87 linked an issue Jul 3, 2020 that may be closed by this pull request

[Proposal] Materialized View 2.0 #3344

Closed

HangyuanLiu reviewed Jul 6, 2020

View reviewed changes

fe/src/main/java/org/apache/doris/analysis/MVColumnHLLUnionPatten.java Outdated Show resolved Hide resolved

EmmyMiao87 force-pushed the mv_test branch 2 times, most recently from 2f2fba8 to 8a82ce5 Compare July 6, 2020 08:10

morningman reviewed Jul 6, 2020

View reviewed changes

wutiangan reviewed Jul 8, 2020

View reviewed changes

HangyuanLiu requested changes Jul 8, 2020

View reviewed changes

fe/src/main/java/org/apache/doris/analysis/MVColumnItem.java Show resolved Hide resolved

EmmyMiao87 force-pushed the mv_test branch 2 times, most recently from 73cf136 to e9ab81e Compare July 14, 2020 06:45

HangyuanLiu previously approved these changes Jul 16, 2020

View reviewed changes

EmmyMiao87 added the approved Indicates a PR has been approved by one committer. label Jul 16, 2020

morningman previously approved these changes Jul 19, 2020

View reviewed changes

EmmyMiao87 added 13 commits July 20, 2020 11:43

Add comment and change field name

dbf5707

Change-Id: I4c8568d4b57384c342eaf5432856eaaa7393f49d

Change code style

688f631

Change-Id: Ibdc44e527279308d6cad77aa7f1d230d27abd02e

Add unit test

83dbcdd

Change-Id: I313b896c20a210ce5a2d9ccbc5ca6d8e78ade3e7

Support bitmap_union and hll_union for agg model by Create MV Stmt

f1571f8

Change-Id: I897301af916b96377664df2d321aad3c4a2b24e1

Fix replay error

65196a3

Change-Id: Ibf277124144e012773de310a17cb5cd57cdc76de

Change display column name

3a44872

Change-Id: I8feab7ac56aa17412bbf45b15b41754f3f3b7410

Remove mv column when drop index

607aff3

Change-Id: I73bd860091b14202acb731a0eab460d992237f03

Reset the define expr

5d424d2

Change-Id: Ib8ab5fa56b1caec597937f6bf3defa4acfe8aab1

Analyze define expr when replay log

d6c885d

Change-Id: If3e553ebb29e59b0e35ed4423094ad8caaa710ac

Add cast when replay log

4bd80e3

Change-Id: Id867098ded33667a0f80f67daf88ee068a1973a0

Change display name

39d3854

Change-Id: I3d5481fab5a09bf5f43b606d1fe0bda41eb2e17b

Change the finalize

0337c0c

Change-Id: I38273603ae21245b38e0aa5dd2b266aceab06575

EmmyMiao87 added 5 commits July 20, 2020 11:45

Remove duplicate compute tablet

68746a2

Change-Id: Id42b9102585901e647d56601e61c0a5d53a7208c

Remove duplicate tablet

fe359d9

Change-Id: I6e16b29f8cdcd8fa4772d235fbeddae6b7ec9511

Fix compile error

bc30cfc

Change-Id: I6656a91208fa58bd870977a6a873bcd9cf9893c9

Fix ut

3bd6812

Change-Id: Ib034a61801fa2d9c0eb566fa5f67f20fc065aea2

Fix compile error

a28a07d

Change-Id: Id9106590c48052ace64eea26831f7e283dddb0de

EmmyMiao87 dismissed stale reviews from morningman and HangyuanLiu via a28a07d July 20, 2020 03:49

EmmyMiao87 force-pushed the mv_test branch from 1af6680 to a28a07d Compare July 20, 2020 03:49

morningman previously approved these changes Jul 20, 2020

View reviewed changes

Fix ut

3f5c9cb

Change-Id: I45a957598f0a4f8bbc3c8404530f86a614fc9890

EmmyMiao87 dismissed morningman’s stale review via 3f5c9cb July 20, 2020 06:42

morningman approved these changes Jul 20, 2020

View reviewed changes

morningman merged commit 2de4f24 into apache:master Jul 20, 2020

EmmyMiao87 mentioned this pull request Aug 17, 2020

Release Notes 0.13.0 #4370

Closed

seawinde mentioned this pull request Mar 3, 2025

[fix](mv) Fix fail when drop table column when sync mv which not use the dropped column exits on table #48565

Closed

16 tasks

csun5285 pushed a commit to csun5285/doris that referenced this pull request Sep 23, 2025

[opt](inverted index) Optimize prefix query in match_regexp (apache#4014

d71604e

)

Add framework of mv selector #4014

Add framework of mv selector #4014

Uh oh!

Conversation

EmmyMiao87 commented Jul 3, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

wutiangan Jul 8, 2020

Choose a reason for hiding this comment

Uh oh!

wutiangan Jul 8, 2020

Choose a reason for hiding this comment

Uh oh!

EmmyMiao87 Jul 13, 2020

Choose a reason for hiding this comment

Uh oh!

Uh oh!

HangyuanLiu left a comment

Choose a reason for hiding this comment

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

EmmyMiao87 commented Jul 3, 2020 •

edited

Loading