[fix](mtmv)The refresh method for MTMV is commit. If the status is PAUSED, no more tasks should be generated #46020

zddr · 2024-12-26T07:39:41Z

What problem does this PR solve?

The refresh method for MTMV is commit. If the status is PAUSED, no more tasks should be generated

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:
The refresh method for MTMV is commit. If the status is PAUSED, no more tasks should be generated

Release note

The refresh method for MTMV is commit. If the status is PAUSED, no more tasks should be generated

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

Thearas · 2024-12-26T07:39:45Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

zddr · 2024-12-26T07:40:57Z

run buildall

doris-robot · 2024-12-26T08:22:29Z

TPC-H: Total hot run time: 32533 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 92f1242c3b8a4a9cc4ae639c1801ae39c2b609a6, data reload: false

------ Round 1 ----------------------------------
q1	17578	6140	6004	6004
q2	2050	304	165	165
q3	10442	1226	744	744
q4	10220	860	442	442
q5	7783	2152	1959	1959
q6	210	184	144	144
q7	894	760	592	592
q8	9239	1338	1106	1106
q9	5270	4996	4886	4886
q10	6766	2313	1874	1874
q11	492	288	259	259
q12	352	370	222	222
q13	18131	3582	3048	3048
q14	241	254	213	213
q15	561	520	496	496
q16	628	622	599	599
q17	581	872	336	336
q18	7027	6600	6428	6428
q19	3287	976	562	562
q20	308	316	191	191
q21	2926	2170	1957	1957
q22	350	341	306	306
Total cold run time: 105336 ms
Total hot run time: 32533 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6327	6240	6230	6230
q2	239	320	236	236
q3	2238	2639	2349	2349
q4	1384	1837	1351	1351
q5	4314	4750	4805	4750
q6	178	176	140	140
q7	2073	1938	1805	1805
q8	2637	2815	2675	2675
q9	7302	7249	7336	7249
q10	3051	3342	2781	2781
q11	561	497	494	494
q12	664	797	609	609
q13	3387	3673	3061	3061
q14	282	308	285	285
q15	573	505	521	505
q16	657	675	646	646
q17	1203	1731	1263	1263
q18	7733	7513	7474	7474
q19	821	1151	1084	1084
q20	2043	2050	1935	1935
q21	5663	5258	5019	5019
q22	605	657	610	610
Total cold run time: 53935 ms
Total hot run time: 52551 ms

fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVJobManager.java

doris-robot · 2024-12-26T08:34:54Z

TPC-DS: Total hot run time: 197468 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 92f1242c3b8a4a9cc4ae639c1801ae39c2b609a6, data reload: false

query1	1285	1006	906	906
query2	6483	2350	2329	2329
query3	11098	4934	4946	4934
query4	33108	24105	23883	23883
query5	4754	610	457	457
query6	295	202	182	182
query7	3989	485	310	310
query8	306	258	225	225
query9	9432	2702	2711	2702
query10	469	301	239	239
query11	19657	15379	15137	15137
query12	149	106	100	100
query13	1555	572	407	407
query14	9971	7206	8129	7206
query15	289	206	191	191
query16	8571	653	474	474
query17	1566	785	600	600
query18	2134	421	318	318
query19	200	202	162	162
query20	116	116	113	113
query21	206	129	112	112
query22	4505	4509	4619	4509
query23	34503	33799	34502	33799
query24	6505	2318	2328	2318
query25	492	473	428	428
query26	874	276	155	155
query27	2079	462	324	324
query28	5305	2499	2464	2464
query29	678	553	409	409
query30	212	178	148	148
query31	979	914	847	847
query32	88	58	60	58
query33	499	353	296	296
query34	800	865	522	522
query35	799	824	766	766
query36	980	1046	972	972
query37	114	100	70	70
query38	4328	4261	4086	4086
query39	1545	1509	1436	1436
query40	211	114	103	103
query41	46	56	43	43
query42	128	104	111	104
query43	509	538	477	477
query44	1360	838	827	827
query45	180	176	174	174
query46	899	1067	674	674
query47	2009	1994	1945	1945
query48	388	417	338	338
query49	698	517	413	413
query50	651	673	384	384
query51	7354	7437	7103	7103
query52	99	96	99	96
query53	231	262	187	187
query54	506	502	420	420
query55	84	77	78	77
query56	280	273	251	251
query57	1228	1213	1133	1133
query58	235	228	224	224
query59	3168	3230	3211	3211
query60	270	262	271	262
query61	113	104	104	104
query62	886	847	792	792
query63	221	198	202	198
query64	3273	1010	656	656
query65	3374	3270	3244	3244
query66	824	409	314	314
query67	16358	15724	15435	15435
query68	9709	761	526	526
query69	477	288	257	257
query70	1202	1101	1134	1101
query71	430	284	254	254
query72	6296	3926	3857	3857
query73	661	770	364	364
query74	10331	9114	8959	8959
query75	4602	3084	2682	2682
query76	4684	1191	774	774
query77	818	361	283	283
query78	10028	10218	9412	9412
query79	3222	927	617	617
query80	772	518	431	431
query81	474	264	233	233
query82	389	147	125	125
query83	189	165	149	149
query84	298	90	67	67
query85	744	367	352	352
query86	362	312	298	298
query87	4617	4477	4508	4477
query88	3146	2255	2203	2203
query89	436	332	290	290
query90	2034	189	185	185
query91	132	147	101	101
query92	71	56	50	50
query93	1875	893	546	546
query94	669	390	255	255
query95	340	272	252	252
query96	496	619	279	279
query97	2757	2879	2674	2674
query98	221	205	192	192
query99	1661	1577	1440	1440
Total cold run time: 300644 ms
Total hot run time: 197468 ms

doris-robot · 2024-12-26T08:40:13Z

ClickBench: Total hot run time: 31.04 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 92f1242c3b8a4a9cc4ae639c1801ae39c2b609a6, data reload: false

query1	0.03	0.02	0.03
query2	0.07	0.03	0.03
query3	0.23	0.07	0.07
query4	1.62	0.11	0.10
query5	0.41	0.39	0.41
query6	1.14	0.66	0.66
query7	0.03	0.02	0.01
query8	0.04	0.04	0.04
query9	0.59	0.50	0.50
query10	0.56	0.57	0.56
query11	0.14	0.10	0.10
query12	0.14	0.12	0.11
query13	0.61	0.61	0.60
query14	2.70	2.82	2.86
query15	0.89	0.82	0.82
query16	0.38	0.37	0.36
query17	1.00	1.06	1.05
query18	0.23	0.21	0.20
query19	1.96	1.88	2.00
query20	0.01	0.02	0.01
query21	15.35	0.93	0.59
query22	0.76	0.77	0.72
query23	15.30	1.44	0.53
query24	3.26	1.20	0.44
query25	0.24	0.23	0.06
query26	0.38	0.14	0.14
query27	0.05	0.05	0.05
query28	13.17	1.58	1.05
query29	12.57	4.00	3.35
query30	0.25	0.09	0.06
query31	2.84	0.57	0.38
query32	3.22	0.54	0.47
query33	3.14	3.05	3.12
query34	16.94	5.18	4.54
query35	4.54	4.57	4.50
query36	0.65	0.49	0.48
query37	0.09	0.06	0.06
query38	0.04	0.04	0.03
query39	0.03	0.02	0.02
query40	0.17	0.14	0.13
query41	0.08	0.02	0.03
query42	0.04	0.03	0.02
query43	0.04	0.02	0.02
Total cold run time: 105.93 s
Total hot run time: 31.04 s

github-actions · 2024-12-26T08:50:43Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-12-26T08:50:45Z

PR approved by anyone and no changes requested.

zddr · 2024-12-26T09:14:08Z

run buildall

doris-robot · 2024-12-26T09:52:32Z

TPC-H: Total hot run time: 32455 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 734461c9ab34ed8a5c63c7d653e330015bfbf9e9, data reload: false

------ Round 1 ----------------------------------
q1	17565	6080	6000	6000
q2	2047	290	158	158
q3	10443	1230	725	725
q4	10212	860	423	423
q5	7504	2190	1927	1927
q6	211	181	144	144
q7	898	760	590	590
q8	9235	1357	1152	1152
q9	5235	4992	4920	4920
q10	6775	2323	1847	1847
q11	486	283	259	259
q12	357	359	216	216
q13	17781	3622	2880	2880
q14	235	237	213	213
q15	570	509	494	494
q16	654	611	582	582
q17	561	848	323	323
q18	6911	6624	6626	6624
q19	1231	971	561	561
q20	319	327	191	191
q21	2827	2227	1921	1921
q22	360	334	305	305
Total cold run time: 102417 ms
Total hot run time: 32455 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6214	6225	6178	6178
q2	243	328	232	232
q3	2255	2621	2300	2300
q4	1469	1832	1355	1355
q5	4290	4694	4683	4683
q6	177	176	141	141
q7	2056	1900	1854	1854
q8	2548	2793	2671	2671
q9	7294	7257	7352	7257
q10	3059	3382	2902	2902
q11	589	519	499	499
q12	638	730	562	562
q13	3423	3829	3139	3139
q14	306	303	273	273
q15	586	530	512	512
q16	652	697	683	683
q17	1198	1713	1262	1262
q18	7633	7426	7269	7269
q19	835	1139	1114	1114
q20	1958	2012	1962	1962
q21	5648	5245	4876	4876
q22	601	617	559	559
Total cold run time: 53672 ms
Total hot run time: 52283 ms

doris-robot · 2024-12-26T10:04:51Z

TPC-DS: Total hot run time: 196705 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 734461c9ab34ed8a5c63c7d653e330015bfbf9e9, data reload: false

query1	1280	940	940	940
query2	6484	2304	2216	2216
query3	11053	4693	4798	4693
query4	32659	23988	23568	23568
query5	3772	616	460	460
query6	294	199	187	187
query7	3985	483	314	314
query8	297	241	224	224
query9	9417	2737	2740	2737
query10	458	307	244	244
query11	18008	15501	15169	15169
query12	155	106	101	101
query13	1583	532	401	401
query14	9479	7030	7815	7030
query15	231	218	198	198
query16	8534	643	467	467
query17	1605	787	563	563
query18	2079	410	311	311
query19	201	178	155	155
query20	129	120	116	116
query21	203	137	106	106
query22	4586	4690	4672	4672
query23	35108	33714	33627	33627
query24	6457	2269	2361	2269
query25	460	461	395	395
query26	875	271	153	153
query27	2185	472	342	342
query28	5250	2529	2479	2479
query29	628	567	434	434
query30	207	192	154	154
query31	953	910	864	864
query32	69	60	59	59
query33	489	350	300	300
query34	740	864	495	495
query35	813	838	764	764
query36	1026	1075	974	974
query37	112	102	75	75
query38	4365	4467	4104	4104
query39	1561	1452	1494	1452
query40	210	115	115	115
query41	44	44	48	44
query42	123	102	103	102
query43	503	521	491	491
query44	1329	842	835	835
query45	194	187	179	179
query46	898	1070	657	657
query47	1962	1980	1939	1939
query48	393	403	330	330
query49	741	495	397	397
query50	625	668	386	386
query51	7148	7318	7043	7043
query52	101	106	92	92
query53	223	252	192	192
query54	470	492	418	418
query55	74	77	78	77
query56	256	270	234	234
query57	1241	1258	1183	1183
query58	237	221	225	221
query59	3177	3339	3074	3074
query60	280	271	263	263
query61	137	130	152	130
query62	899	820	762	762
query63	235	198	196	196
query64	2964	1069	686	686
query65	3329	3293	3260	3260
query66	788	432	318	318
query67	16371	15950	15568	15568
query68	9026	781	520	520
query69	483	285	252	252
query70	1168	1168	1157	1157
query71	420	276	262	262
query72	6316	3898	3805	3805
query73	665	771	360	360
query74	10133	9040	9266	9040
query75	4457	3244	2628	2628
query76	4633	1329	780	780
query77	829	371	284	284
query78	10198	10176	9405	9405
query79	3058	877	592	592
query80	734	519	438	438
query81	484	269	230	230
query82	702	142	127	127
query83	199	163	145	145
query84	279	100	69	69
query85	774	351	315	315
query86	338	325	319	319
query87	4410	4550	4422	4422
query88	3479	2290	2221	2221
query89	427	335	286	286
query90	1882	187	191	187
query91	135	137	106	106
query92	66	55	50	50
query93	1362	885	543	543
query94	659	384	276	276
query95	333	262	256	256
query96	481	595	281	281
query97	2766	2785	2726	2726
query98	218	200	200	200
query99	1735	1553	1419	1419
Total cold run time: 295591 ms
Total hot run time: 196705 ms

doris-robot · 2024-12-26T10:10:10Z

ClickBench: Total hot run time: 31.44 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 734461c9ab34ed8a5c63c7d653e330015bfbf9e9, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.03	0.03
query3	0.24	0.07	0.07
query4	1.65	0.11	0.11
query5	0.42	0.42	0.41
query6	1.15	0.66	0.66
query7	0.02	0.02	0.02
query8	0.05	0.04	0.03
query9	0.59	0.51	0.52
query10	0.53	0.58	0.55
query11	0.14	0.10	0.10
query12	0.15	0.11	0.12
query13	0.60	0.61	0.59
query14	2.69	2.86	2.71
query15	0.89	0.82	0.82
query16	0.39	0.38	0.37
query17	1.01	0.99	0.97
query18	0.22	0.22	0.21
query19	1.80	1.73	1.94
query20	0.01	0.01	0.02
query21	15.36	0.98	0.59
query22	0.75	0.88	0.74
query23	15.12	1.46	0.57
query24	2.57	1.56	1.01
query25	0.23	0.13	0.17
query26	0.28	0.16	0.14
query27	0.05	0.05	0.04
query28	13.74	1.61	1.04
query29	12.57	4.03	3.31
query30	0.25	0.09	0.07
query31	2.83	0.62	0.38
query32	3.23	0.56	0.45
query33	3.09	3.14	3.09
query34	16.69	5.18	4.54
query35	4.51	4.53	4.54
query36	0.65	0.50	0.47
query37	0.09	0.06	0.06
query38	0.04	0.05	0.03
query39	0.03	0.02	0.03
query40	0.17	0.14	0.12
query41	0.08	0.02	0.02
query42	0.04	0.03	0.02
query43	0.03	0.03	0.04
Total cold run time: 105.06 s
Total hot run time: 31.44 s

github-actions · 2025-01-02T02:51:04Z

PR approved by at least one committer and no changes requested.

…USED, no more tasks should be generated (#46020) ### What problem does this PR solve? The refresh method for MTMV is commit. If the status is PAUSED, no more tasks should be generated Issue Number: close #xxx Related PR: #xxx Problem Summary: The refresh method for MTMV is commit. If the status is PAUSED, no more tasks should be generated

…status is PAUSED, no more tasks should be generated #46020 (#46265) Cherry-picked from #46020 Co-authored-by: zhangdong <zhangdong@selectdb.com>

…status is PAUSED, no more tasks should be generated #46020 (#46264) Cherry-picked from #46020 Co-authored-by: zhangdong <zhangdong@selectdb.com>

zddr added 3 commits December 26, 2024 15:14

1

8cdccb4

1

0c056c8

1

92f1242

morrySnow added usercase Important user case type label p0_b dev/2.1.x dev/3.0.x labels Dec 26, 2024

cambyzju reviewed Dec 26, 2024

View reviewed changes

fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVJobManager.java Outdated Show resolved Hide resolved

morrySnow previously approved these changes Dec 26, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 26, 2024

github-actions bot added the reviewed label Dec 26, 2024

1

734461c

zddr dismissed morrySnow’s stale review via 734461c December 26, 2024 09:13

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Dec 26, 2024

zddr requested review from cambyzju and morrySnow December 27, 2024 03:44

morrySnow approved these changes Jan 2, 2025

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 2, 2025

yiguolei approved these changes Jan 2, 2025

View reviewed changes

yiguolei merged commit 685f87c into apache:master Jan 2, 2025
27 of 28 checks passed

github-actions bot mentioned this pull request Jan 2, 2025

branch-3.0: [fix](mtmv)The refresh method for MTMV is commit. If the status is PAUSED, no more tasks should be generated #46020 #46264

Merged

github-actions bot mentioned this pull request Jan 2, 2025

branch-2.1: [fix](mtmv)The refresh method for MTMV is commit. If the status is PAUSED, no more tasks should be generated #46020 #46265

Merged

yiguolei added dev/2.1.8-merged and removed dev/2.1.x labels Jan 2, 2025

morningman added dev/3.0.4-merged and removed dev/3.0.x labels Jan 6, 2025

gavinchou mentioned this pull request Feb 18, 2025

Release Notes 3.0.4 #48013

Closed

[fix](mtmv)The refresh method for MTMV is commit. If the status is PAUSED, no more tasks should be generated #46020

[fix](mtmv)The refresh method for MTMV is commit. If the status is PAUSED, no more tasks should be generated #46020

Uh oh!

Conversation

zddr commented Dec 26, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

Thearas commented Dec 26, 2024

Uh oh!

zddr commented Dec 26, 2024

Uh oh!

doris-robot commented Dec 26, 2024

Uh oh!

Uh oh!

doris-robot commented Dec 26, 2024

Uh oh!

doris-robot commented Dec 26, 2024

Uh oh!

github-actions bot commented Dec 26, 2024

Uh oh!

github-actions bot commented Dec 26, 2024

Uh oh!

zddr commented Dec 26, 2024

Uh oh!

doris-robot commented Dec 26, 2024

Uh oh!

doris-robot commented Dec 26, 2024

Uh oh!

doris-robot commented Dec 26, 2024

Uh oh!

github-actions bot commented Jan 2, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

zddr commented Dec 26, 2024 •

edited

Loading