Skip to content

branch-3.0: [fix](mtmv) Fix filter position different but same causing rewritten by materialized view fail #44575 #44899

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morrySnow merged 1 commit into from
Dec 6, 2024
Merged

branch-3.0: [fix](mtmv) Fix filter position different but same causing rewritten by materialized view fail #44575 #44899

morrySnow merged 1 commit into from
Dec 6, 2024

Conversation

@github-actions
Copy link
Contributor

@github-actions github-actions bot commented Dec 3, 2024

Cherry-picked from #44575

@seawinde
[fix](mtmv) Fix filter position different but same causing rewritten …
bd1699c 
…by materialized view fail (#44575)

Such as mv def is as following and query is the same
this is the same filter `l_orderkey is null or l_orderkey <> 1` but they
are in the different position. this would cause rewrite fail, this pr
fix this.
```sql
select 
  o_custkey, 
  o_orderdate, 
  o_shippriority, 
  o_comment, 
  o_orderkey, 
  orders.public_col as col1, 
  l_orderkey, 
  l_partkey, 
  l_suppkey, 
  lineitem.public_col as col2, 
  ps_partkey, 
  ps_suppkey, 
  partsupp.public_col as col3, 
  partsupp.public_col * 2 as col4, 
  o_orderkey + l_orderkey + ps_partkey * 2, 
  sum(
    o_orderkey + l_orderkey + ps_partkey * 2
  ), 
  count() as count_all 
from 
  (
    select 
      o_custkey, 
      o_orderdate, 
      o_shippriority, 
      o_comment, 
      o_orderkey, 
      orders.public_col as public_col 
    from 
      orders
  ) orders 
  left join (
    select 
      l_orderkey, 
      l_partkey, 
      l_suppkey, 
      lineitem.public_col as public_col 
    from 
      lineitem 
    where 
      l_orderkey is null 
      or l_orderkey <> 1
  ) lineitem on l_orderkey = o_orderkey 
  inner join (
    select 
      ps_partkey, 
      ps_suppkey, 
      partsupp.public_col as public_col 
    from 
      partsupp
  ) partsupp on ps_partkey = o_orderkey 
  and ps_suppkey = o_custkey 
where 
  l_orderkey is null 
  or l_orderkey <> 1 
group by 
  1, 
  2, 
  3, 
  4, 
  5, 
  6, 
  7, 
  8, 
  9, 
  10, 
  11, 
  12, 
  13, 
  14;
```
Related PR: #34753 
Fix filter position different but same causing rewritten by materialized
view fail
@doris-robot
Copy link

doris-robot commented Dec 3, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring closed this Dec 3, 2024
@dataroaring dataroaring reopened this Dec 3, 2024
@doris-robot
Copy link

doris-robot commented Dec 3, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Dec 3, 2024

TPC-H: Total hot run time: 40619 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit bd1699c66d4cc92c62be8843e638ca21351b48b0, data reload: false

------ Round 1 ----------------------------------
q1	17579	7379	7308	7308
q2	2041	167	183	167
q3	10690	1085	1151	1085
q4	10550	717	762	717
q5	7740	2803	2825	2803
q6	238	147	149	147
q7	962	604	595	595
q8	9565	1921	1968	1921
q9	8038	6364	6370	6364
q10	7008	2297	2263	2263
q11	452	267	256	256
q12	405	214	208	208
q13	17758	3007	2929	2929
q14	245	211	219	211
q15	551	510	521	510
q16	691	601	597	597
q17	964	562	560	560
q18	7174	6620	6652	6620
q19	2887	1046	1032	1032
q20	455	203	207	203
q21	3869	3165	3264	3165
q22	1077	985	958	958
Total cold run time: 110939 ms
Total hot run time: 40619 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7352	7202	7283	7202
q2	316	224	219	219
q3	2997	2916	2859	2859
q4	2063	1791	1725	1725
q5	5601	5697	5710	5697
q6	220	142	138	138
q7	2200	1765	1750	1750
q8	3374	3455	3349	3349
q9	8842	8850	8737	8737
q10	3536	3465	3520	3465
q11	606	510	487	487
q12	818	657	594	594
q13	16677	3080	3083	3080
q14	300	275	282	275
q15	563	515	514	514
q16	699	667	681	667
q17	1812	1590	1582	1582
q18	8212	7757	7479	7479
q19	3764	1540	1536	1536
q20	2041	1858	1838	1838
q21	5293	5276	5211	5211
q22	1082	1007	1004	1004
Total cold run time: 78368 ms
Total hot run time: 59408 ms

@doris-robot
Copy link

doris-robot commented Dec 3, 2024

TPC-DS: Total hot run time: 195607 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit bd1699c66d4cc92c62be8843e638ca21351b48b0, data reload: false

query1	1227	932	931	931
query2	6331	2036	2030	2030
query3	10828	4001	4071	4001
query4	67074	28747	23336	23336
query5	5117	438	435	435
query6	421	174	179	174
query7	5680	309	317	309
query8	322	250	231	231
query9	8898	2683	2657	2657
query10	477	270	249	249
query11	17465	15126	15818	15126
query12	154	99	101	99
query13	1548	439	413	413
query14	10700	6974	7029	6974
query15	220	180	171	171
query16	7136	496	463	463
query17	1264	577	571	571
query18	1849	317	317	317
query19	218	157	157	157
query20	122	113	112	112
query21	212	106	105	105
query22	4649	4415	4623	4415
query23	34589	33861	33911	33861
query24	6118	2953	2885	2885
query25	530	423	421	421
query26	670	171	168	168
query27	1892	301	303	301
query28	4413	2561	2537	2537
query29	700	483	461	461
query30	235	166	160	160
query31	1023	802	854	802
query32	62	57	58	57
query33	410	298	285	285
query34	919	510	501	501
query35	860	716	737	716
query36	1060	950	950	950
query37	116	73	78	73
query38	4198	4008	3948	3948
query39	1557	1497	1480	1480
query40	203	101	95	95
query41	51	46	45	45
query42	112	96	95	95
query43	537	484	494	484
query44	1160	792	788	788
query45	185	166	166	166
query46	1151	732	727	727
query47	2004	1876	1902	1876
query48	448	364	374	364
query49	712	386	373	373
query50	831	404	405	404
query51	7305	7206	7189	7189
query52	102	90	86	86
query53	259	179	179	179
query54	557	448	441	441
query55	75	73	70	70
query56	245	238	235	235
query57	1157	1082	1064	1064
query58	203	205	199	199
query59	3108	3132	3248	3132
query60	270	254	244	244
query61	113	118	121	118
query62	765	640	658	640
query63	209	180	182	180
query64	1413	653	608	608
query65	3216	3160	3181	3160
query66	702	314	293	293
query67	15918	15553	15532	15532
query68	4428	549	547	547
query69	434	254	253	253
query70	1124	1060	1154	1060
query71	406	259	242	242
query72	6459	4109	4136	4109
query73	751	342	342	342
query74	10182	8844	8978	8844
query75	3322	2571	2578	2571
query76	2078	1053	1005	1005
query77	529	268	263	263
query78	10649	9693	9382	9382
query79	8708	591	589	589
query80	2107	408	410	408
query81	544	245	231	231
query82	1187	114	111	111
query83	266	138	138	138
query84	294	81	82	81
query85	1645	291	282	282
query86	478	300	303	300
query87	4427	4263	4193	4193
query88	5502	2358	2385	2358
query89	546	289	290	289
query90	2136	179	180	179
query91	177	144	143	143
query92	70	46	46	46
query93	6655	540	538	538
query94	873	289	289	289
query95	349	243	242	242
query96	621	279	283	279
query97	3351	3151	3133	3133
query98	212	194	195	194
query99	1877	1299	1297	1297
Total cold run time: 336662 ms
Total hot run time: 195607 ms

@morrySnow morrySnow merged commit e5217a2 into branch-3.0 Dec 6, 2024
12 checks passed
@morrySnow morrySnow deleted the auto-pick-44575-branch-3.0 branch December 6, 2024 02:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@doris-robot @dataroaring @morrySnow @seawinde