Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](mtmv)fix nested mtmv not refresh #40433

Merged
merged 5 commits into from
Sep 9, 2024

Conversation

zddr
Copy link
Contributor

@zddr zddr commented Sep 5, 2024

fix nested mtmv not refresh because the partition version remains unchanged after inserting overwrite for the underlying materialized view

we add partitionId in snapshot

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@zddr
Copy link
Contributor Author

zddr commented Sep 5, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 38652 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit cbad721a9416bb57b2f3bb84d6619c0863fc46b1, data reload: false

------ Round 1 ----------------------------------
q1	17751	4531	4370	4370
q2	2807	200	189	189
q3	11139	1148	1159	1148
q4	10251	819	789	789
q5	7762	2911	2795	2795
q6	234	142	142	142
q7	975	648	609	609
q8	9653	2119	2077	2077
q9	7248	6542	6519	6519
q10	7001	2192	2405	2192
q11	463	253	250	250
q12	425	236	230	230
q13	17992	3111	3057	3057
q14	289	252	249	249
q15	532	489	487	487
q16	590	509	539	509
q17	975	681	658	658
q18	7360	6875	6850	6850
q19	1392	1174	1012	1012
q20	690	348	337	337
q21	3987	3165	3156	3156
q22	1129	1029	1027	1027
Total cold run time: 110645 ms
Total hot run time: 38652 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4454	4309	4321	4309
q2	373	276	283	276
q3	2915	2746	2635	2635
q4	1957	1691	1757	1691
q5	5433	5384	5423	5384
q6	222	135	135	135
q7	2109	1735	1785	1735
q8	3252	3385	3401	3385
q9	8443	8514	8456	8456
q10	3524	3204	3169	3169
q11	627	509	521	509
q12	800	619	639	619
q13	11690	3090	3130	3090
q14	305	289	280	280
q15	524	481	486	481
q16	578	586	577	577
q17	1812	1508	1491	1491
q18	7803	7406	7685	7406
q19	1654	1669	1555	1555
q20	2058	1819	1822	1819
q21	5732	5361	5247	5247
q22	1139	1082	1031	1031
Total cold run time: 67404 ms
Total hot run time: 55280 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 188977 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit cbad721a9416bb57b2f3bb84d6619c0863fc46b1, data reload: false

query1	911	370	383	370
query2	6482	1869	1893	1869
query3	6658	217	218	217
query4	34216	23402	23140	23140
query5	4168	507	474	474
query6	256	186	178	178
query7	4592	311	296	296
query8	277	213	218	213
query9	8541	2515	2512	2512
query10	462	266	266	266
query11	17806	15080	15077	15077
query12	150	94	98	94
query13	1633	391	372	372
query14	9889	7200	7418	7200
query15	241	166	169	166
query16	8076	458	449	449
query17	1554	586	552	552
query18	2124	297	289	289
query19	213	143	138	138
query20	116	109	108	108
query21	209	106	103	103
query22	4494	4194	4234	4194
query23	34136	33765	33572	33572
query24	11221	2865	2874	2865
query25	615	383	388	383
query26	1126	161	156	156
query27	2433	275	277	275
query28	7182	2062	2057	2057
query29	822	417	443	417
query30	319	165	151	151
query31	983	776	801	776
query32	105	61	54	54
query33	745	278	288	278
query34	1005	488	499	488
query35	854	722	728	722
query36	1091	905	943	905
query37	155	91	94	91
query38	3973	3932	3906	3906
query39	1432	1413	1398	1398
query40	209	117	115	115
query41	49	45	49	45
query42	117	95	98	95
query43	496	445	462	445
query44	1276	764	750	750
query45	202	171	172	171
query46	1102	790	741	741
query47	1882	1858	1821	1821
query48	376	296	303	296
query49	1127	458	462	458
query50	834	412	412	412
query51	6903	6755	6910	6755
query52	101	89	87	87
query53	255	184	186	184
query54	913	467	469	467
query55	77	76	81	76
query56	300	269	273	269
query57	1175	1078	1066	1066
query58	254	234	257	234
query59	2988	2829	2700	2700
query60	323	279	282	279
query61	147	124	126	124
query62	838	658	674	658
query63	222	189	185	185
query64	4334	791	778	778
query65	3244	3184	3189	3184
query66	1441	355	353	353
query67	15594	15388	15325	15325
query68	4083	583	586	583
query69	415	284	290	284
query70	1174	1097	1149	1097
query71	340	288	277	277
query72	6579	4197	4277	4197
query73	741	331	328	328
query74	9178	8879	8913	8879
query75	3422	2737	2729	2729
query76	2202	1060	1088	1060
query77	476	321	312	312
query78	9674	9025	9642	9025
query79	2538	538	547	538
query80	1071	498	505	498
query81	601	233	241	233
query82	681	155	146	146
query83	232	155	149	149
query84	264	80	78	78
query85	1752	296	276	276
query86	310	276	302	276
query87	4478	4457	4328	4328
query88	3563	2389	2366	2366
query89	393	288	283	283
query90	1501	186	189	186
query91	126	106	99	99
query92	56	49	50	49
query93	2338	550	547	547
query94	546	333	290	290
query95	331	249	257	249
query96	606	271	269	269
query97	3228	3051	3072	3051
query98	226	200	196	196
query99	1734	1266	1255	1255
Total cold run time: 292903 ms
Total hot run time: 188977 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 33.09 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit cbad721a9416bb57b2f3bb84d6619c0863fc46b1, data reload: false

query1	0.05	0.04	0.05
query2	0.07	0.04	0.05
query3	0.23	0.06	0.06
query4	1.65	0.10	0.10
query5	0.51	0.50	0.52
query6	1.13	0.76	0.73
query7	0.02	0.01	0.02
query8	0.04	0.04	0.04
query9	0.55	0.49	0.49
query10	0.56	0.59	0.54
query11	0.15	0.12	0.11
query12	0.15	0.13	0.11
query13	0.61	0.59	0.58
query14	2.05	2.11	2.09
query15	0.85	0.82	0.82
query16	0.37	0.40	0.38
query17	1.03	1.04	1.03
query18	0.22	0.20	0.20
query19	1.96	1.77	1.85
query20	0.01	0.01	0.02
query21	15.41	0.68	0.66
query22	4.13	6.29	2.79
query23	18.32	1.49	1.21
query24	2.06	0.22	0.23
query25	0.15	0.08	0.09
query26	0.27	0.17	0.18
query27	0.08	0.08	0.08
query28	13.26	1.02	0.99
query29	12.64	3.33	3.34
query30	0.25	0.06	0.07
query31	2.88	0.40	0.39
query32	3.22	0.48	0.48
query33	3.05	3.00	3.10
query34	17.14	4.46	4.45
query35	4.44	4.46	4.48
query36	0.66	0.48	0.47
query37	0.18	0.16	0.16
query38	0.15	0.14	0.15
query39	0.06	0.03	0.04
query40	0.16	0.13	0.13
query41	0.09	0.05	0.05
query42	0.06	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 110.91 s
Total hot run time: 33.09 s

@zddr
Copy link
Contributor Author

zddr commented Sep 5, 2024

run cloud_p0

@zddr
Copy link
Contributor Author

zddr commented Sep 6, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 37974 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6b37022b36b99a9121362334ab38213aceb4e5c8, data reload: false

------ Round 1 ----------------------------------
q1	17616	4358	4323	4323
q2	2019	194	192	192
q3	11705	944	1074	944
q4	10519	699	696	696
q5	7748	2868	2802	2802
q6	224	139	137	137
q7	955	623	632	623
q8	9321	2077	2054	2054
q9	7133	6572	6565	6565
q10	7002	2197	2265	2197
q11	473	272	242	242
q12	392	222	226	222
q13	18755	3082	3020	3020
q14	276	241	243	241
q15	535	487	484	484
q16	526	446	430	430
q17	976	633	721	633
q18	7268	6797	6779	6779
q19	1402	1099	1063	1063
q20	688	328	330	328
q21	4126	3017	3082	3017
q22	1133	1026	982	982
Total cold run time: 110792 ms
Total hot run time: 37974 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4417	4284	4352	4284
q2	391	271	270	270
q3	2904	2646	2626	2626
q4	1939	1643	1671	1643
q5	5654	5695	5747	5695
q6	230	137	139	137
q7	2245	1833	1819	1819
q8	3319	3438	3435	3435
q9	8899	8867	8791	8791
q10	3557	3410	3364	3364
q11	610	512	520	512
q12	846	647	658	647
q13	13792	3447	3275	3275
q14	336	299	277	277
q15	551	484	498	484
q16	572	514	488	488
q17	1856	1542	1524	1524
q18	8139	7774	7786	7774
q19	1769	1642	1473	1473
q20	2140	1926	1933	1926
q21	5677	5447	5437	5437
q22	1181	1059	1035	1035
Total cold run time: 71024 ms
Total hot run time: 56916 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192734 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6b37022b36b99a9121362334ab38213aceb4e5c8, data reload: false

query1	1259	876	915	876
query2	6329	1914	1904	1904
query3	10618	4042	3892	3892
query4	59523	25692	23527	23527
query5	5394	509	492	492
query6	406	169	166	166
query7	5788	283	285	283
query8	317	219	211	211
query9	8998	2492	2492	2492
query10	487	269	261	261
query11	18312	15063	15350	15063
query12	156	103	105	103
query13	1547	419	395	395
query14	11240	7791	7295	7295
query15	218	179	168	168
query16	7149	464	516	464
query17	1102	585	583	583
query18	1390	301	290	290
query19	275	150	144	144
query20	124	110	106	106
query21	209	105	105	105
query22	4739	4409	4431	4409
query23	34312	34400	33389	33389
query24	5940	2820	2788	2788
query25	493	406	417	406
query26	656	151	155	151
query27	1743	277	274	274
query28	3590	2013	1987	1987
query29	634	401	404	401
query30	231	155	152	152
query31	946	763	779	763
query32	77	51	57	51
query33	437	287	276	276
query34	881	456	468	456
query35	856	722	747	722
query36	1079	955	941	941
query37	141	79	88	79
query38	3942	3807	3864	3807
query39	1482	1405	1419	1405
query40	198	110	110	110
query41	46	45	44	44
query42	115	96	95	95
query43	506	477	459	459
query44	1112	753	750	750
query45	196	166	167	166
query46	1094	768	723	723
query47	1885	1801	1822	1801
query48	388	294	289	289
query49	779	452	467	452
query50	822	410	415	410
query51	7136	6932	6888	6888
query52	97	86	89	86
query53	250	178	183	178
query54	570	458	462	458
query55	78	80	74	74
query56	295	260	273	260
query57	1182	1101	1080	1080
query58	227	239	248	239
query59	3057	2979	2801	2801
query60	307	273	289	273
query61	126	119	118	118
query62	782	662	669	662
query63	216	185	185	185
query64	2607	753	758	753
query65	3206	3131	3147	3131
query66	698	353	345	345
query67	15419	15355	15293	15293
query68	3213	598	585	585
query69	411	288	278	278
query70	1165	1016	1133	1016
query71	364	280	284	280
query72	6177	3925	3767	3767
query73	760	323	315	315
query74	9218	8737	8949	8737
query75	3374	2733	2672	2672
query76	1516	1033	975	975
query77	520	317	315	315
query78	9953	9123	9182	9123
query79	1111	536	531	531
query80	874	521	520	520
query81	455	233	228	228
query82	1208	140	140	140
query83	208	156	151	151
query84	258	75	74	74
query85	796	288	279	279
query86	308	303	293	293
query87	4408	4382	4361	4361
query88	3293	2284	2286	2284
query89	396	281	284	281
query90	1914	186	189	186
query91	120	98	97	97
query92	65	48	49	48
query93	1494	540	531	531
query94	743	292	272	272
query95	341	257	251	251
query96	584	259	267	259
query97	3184	3059	3079	3059
query98	262	194	203	194
query99	1646	1263	1232	1232
Total cold run time: 306753 ms
Total hot run time: 192734 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.15 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6b37022b36b99a9121362334ab38213aceb4e5c8, data reload: false

query1	0.04	0.04	0.04
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.68	0.07	0.07
query5	0.51	0.50	0.50
query6	1.14	0.73	0.74
query7	0.02	0.01	0.01
query8	0.05	0.05	0.05
query9	0.55	0.49	0.48
query10	0.54	0.56	0.52
query11	0.16	0.11	0.12
query12	0.15	0.12	0.13
query13	0.60	0.59	0.59
query14	1.39	1.35	1.43
query15	0.86	0.83	0.82
query16	0.36	0.38	0.38
query17	1.01	1.04	0.99
query18	0.22	0.21	0.21
query19	1.81	1.72	1.78
query20	0.01	0.01	0.01
query21	15.38	0.67	0.66
query22	4.55	6.72	1.65
query23	18.27	1.36	1.34
query24	2.09	0.22	0.23
query25	0.15	0.08	0.08
query26	0.28	0.18	0.19
query27	0.08	0.08	0.07
query28	13.25	1.02	1.01
query29	12.62	3.40	3.37
query30	0.24	0.06	0.06
query31	2.88	0.40	0.41
query32	3.24	0.48	0.48
query33	2.95	3.00	3.06
query34	16.95	4.35	4.42
query35	4.48	4.40	4.46
query36	0.67	0.48	0.50
query37	0.18	0.16	0.16
query38	0.16	0.16	0.15
query39	0.04	0.04	0.04
query40	0.16	0.12	0.14
query41	0.09	0.05	0.05
query42	0.06	0.05	0.06
query43	0.05	0.04	0.04
Total cold run time: 110.22 s
Total hot run time: 31.15 s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 9, 2024
Copy link
Contributor

github-actions bot commented Sep 9, 2024

PR approved by at least one committer and no changes requested.

Copy link
Contributor

github-actions bot commented Sep 9, 2024

PR approved by anyone and no changes requested.

@morrySnow morrySnow merged commit 5392cb1 into apache:master Sep 9, 2024
25 of 28 checks passed
zddr added a commit to zddr/incubator-doris that referenced this pull request Sep 9, 2024
fix nested mtmv not refresh because the partition version remains
unchanged after inserting overwrite for the underlying materialized view

we add partitionId in snapshot
yiguolei pushed a commit that referenced this pull request Sep 10, 2024
gavinchou pushed a commit that referenced this pull request Sep 12, 2024
fix nested mtmv not refresh because the partition version remains
unchanged after inserting overwrite for the underlying materialized view

we add partitionId in snapshot
@yiguolei yiguolei mentioned this pull request Nov 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.7-merged dev/3.0.2-merged p0_b p0_c reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants