Skip to content

[fix](export) remove task from map finally #51171

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
May 26, 2025
Merged

[fix](export) remove task from map finally #51171

merged 1 commit into from
May 26, 2025

Conversation

@morningman
Copy link
Contributor

@morningman morningman commented May 22, 2025 •
edited
Loading

What problem does this PR solve?

Related PR: #47974

Problem Summary:

When cancel task, the task should be removed from taskExecutorMap,
otherwise, if tryPublishTask() failed when addMemoryTask, the task will
be leaked in taskExecutorMap

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented May 22, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@morningman
Copy link
Contributor Author

morningman commented May 23, 2025

run buildall

@doris-robot
Copy link

doris-robot commented May 23, 2025

TPC-H: Total hot run time: 33899 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6f7e01fae2c95bc3f7d40b6bce131d5d98f82f9e, data reload: false

------ Round 1 ----------------------------------
q1	25929	5544	5027	5027
q2	2082	273	196	196
q3	10381	1225	705	705
q4	10233	996	535	535
q5	7524	2302	2391	2302
q6	178	163	130	130
q7	899	720	605	605
q8	9308	1235	1108	1108
q9	6871	5110	5015	5015
q10	7101	2317	1888	1888
q11	495	282	284	282
q12	341	348	211	211
q13	17821	3633	3126	3126
q14	235	218	214	214
q15	533	479	495	479
q16	423	437	373	373
q17	614	908	379	379
q18	7566	7207	7075	7075
q19	1544	968	591	591
q20	342	343	239	239
q21	4173	3182	2433	2433
q22	1051	1022	986	986
Total cold run time: 115644 ms
Total hot run time: 33899 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5161	5064	5054	5054
q2	240	336	233	233
q3	2156	2657	2263	2263
q4	1345	1808	1364	1364
q5	4417	4405	4382	4382
q6	220	176	134	134
q7	2045	1932	1773	1773
q8	2606	2468	2456	2456
q9	7253	7160	6812	6812
q10	3030	3212	2790	2790
q11	583	522	497	497
q12	656	762	596	596
q13	3459	3959	3269	3269
q14	284	309	289	289
q15	526	500	480	480
q16	459	490	441	441
q17	1151	1561	1361	1361
q18	7662	7422	7405	7405
q19	857	870	970	870
q20	2004	2032	1869	1869
q21	4680	4340	4417	4340
q22	1094	1056	1020	1020
Total cold run time: 51888 ms
Total hot run time: 49698 ms

@doris-robot
Copy link

doris-robot commented May 23, 2025

TPC-DS: Total hot run time: 193171 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6f7e01fae2c95bc3f7d40b6bce131d5d98f82f9e, data reload: false

query1	1420	1084	1058	1058
query2	6300	1824	1809	1809
query3	11039	4425	4529	4425
query4	55670	25587	23871	23871
query5	4925	576	489	489
query6	355	217	212	212
query7	4891	513	282	282
query8	302	243	222	222
query9	5212	2681	2683	2681
query10	433	319	280	280
query11	14956	15050	14933	14933
query12	163	110	107	107
query13	1047	536	411	411
query14	10183	6302	6309	6302
query15	204	208	188	188
query16	7145	641	484	484
query17	1094	758	613	613
query18	1594	418	327	327
query19	208	213	182	182
query20	127	123	117	117
query21	220	130	108	108
query22	4561	4610	4461	4461
query23	34229	33548	33665	33548
query24	6513	2403	2443	2403
query25	453	547	408	408
query26	719	280	158	158
query27	2503	531	333	333
query28	3129	2156	2198	2156
query29	572	580	437	437
query30	289	229	199	199
query31	857	902	786	786
query32	73	74	65	65
query33	447	365	309	309
query34	783	876	523	523
query35	834	819	766	766
query36	946	991	914	914
query37	112	102	76	76
query38	4213	4343	4242	4242
query39	1528	1467	1473	1467
query40	215	116	108	108
query41	54	60	53	53
query42	133	111	110	110
query43	515	542	493	493
query44	1342	872	833	833
query45	178	176	166	166
query46	843	1045	654	654
query47	1889	1891	1860	1860
query48	398	434	320	320
query49	678	516	427	427
query50	666	703	415	415
query51	4307	4267	4160	4160
query52	115	105	105	105
query53	229	258	196	196
query54	576	590	508	508
query55	88	84	86	84
query56	303	298	307	298
query57	1196	1220	1110	1110
query58	279	260	255	255
query59	2614	2730	2634	2634
query60	337	331	307	307
query61	135	127	129	127
query62	706	728	675	675
query63	238	191	195	191
query64	1858	1044	729	729
query65	4307	4290	4252	4252
query66	758	400	303	303
query67	16042	15670	15333	15333
query68	7819	886	524	524
query69	560	291	264	264
query70	1206	1070	1101	1070
query71	501	321	289	289
query72	5984	4859	4907	4859
query73	1523	677	360	360
query74	9013	9181	8683	8683
query75	3998	3183	2661	2661
query76	4239	1203	733	733
query77	676	361	287	287
query78	9918	10038	9327	9327
query79	5675	772	552	552
query80	689	505	436	436
query81	481	256	219	219
query82	687	128	96	96
query83	318	254	229	229
query84	293	102	85	85
query85	765	365	313	313
query86	418	293	284	284
query87	4444	4475	4351	4351
query88	2947	2290	2325	2290
query89	441	325	290	290
query90	2108	203	208	203
query91	139	141	110	110
query92	70	64	55	55
query93	3615	924	572	572
query94	689	407	286	286
query95	362	298	288	288
query96	495	570	279	279
query97	2713	2786	2662	2662
query98	230	203	204	203
query99	1408	1376	1253	1253
Total cold run time: 305672 ms
Total hot run time: 193171 ms

@doris-robot
Copy link

doris-robot commented May 23, 2025

ClickBench: Total hot run time: 28.67 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6f7e01fae2c95bc3f7d40b6bce131d5d98f82f9e, data reload: false

query1	0.04	0.04	0.03
query2	0.12	0.11	0.12
query3	0.25	0.20	0.20
query4	1.60	0.20	0.19
query5	0.45	0.43	0.44
query6	1.16	0.67	0.67
query7	0.03	0.02	0.02
query8	0.04	0.03	0.03
query9	0.59	0.53	0.53
query10	0.56	0.57	0.58
query11	0.16	0.11	0.11
query12	0.15	0.11	0.12
query13	0.61	0.59	0.60
query14	0.79	0.80	0.81
query15	0.88	0.86	0.86
query16	0.38	0.39	0.38
query17	1.03	1.05	1.01
query18	0.22	0.22	0.21
query19	1.93	1.85	1.78
query20	0.01	0.01	0.01
query21	15.43	0.92	0.53
query22	0.77	1.17	0.71
query23	14.86	1.37	0.62
query24	7.75	0.69	0.37
query25	0.53	0.20	0.16
query26	0.64	0.17	0.14
query27	0.05	0.05	0.06
query28	8.92	0.97	0.44
query29	12.65	3.95	3.30
query30	0.24	0.09	0.07
query31	2.83	0.61	0.39
query32	3.23	0.55	0.46
query33	3.06	3.09	3.01
query34	15.73	5.08	4.44
query35	4.57	4.55	4.54
query36	0.66	0.49	0.48
query37	0.09	0.07	0.07
query38	0.05	0.03	0.03
query39	0.03	0.02	0.03
query40	0.17	0.14	0.13
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.04	0.04	0.03
Total cold run time: 103.42 s
Total hot run time: 28.67 s

@morningman
Copy link
Contributor Author

morningman commented May 24, 2025

run buildall

This was referenced May 24, 2025
Merged
Merged
@doris-robot
Copy link

doris-robot commented May 24, 2025

TPC-H: Total hot run time: 34708 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4e353f81eebf089b65a54cea553d3f5927a27aae, data reload: false

------ Round 1 ----------------------------------
q1	26479	5060	5042	5042
q2	2100	298	196	196
q3	10357	1236	719	719
q4	10230	1013	533	533
q5	7519	2400	2306	2306
q6	182	160	133	133
q7	907	741	612	612
q8	9302	1238	1083	1083
q9	6962	5119	5061	5061
q10	6933	2320	1925	1925
q11	506	307	299	299
q12	380	381	230	230
q13	17788	3709	3107	3107
q14	237	231	215	215
q15	532	470	495	470
q16	433	443	380	380
q17	632	857	399	399
q18	7546	7194	7222	7194
q19	1670	950	558	558
q20	344	361	233	233
q21	4154	3507	3028	3028
q22	1076	1042	985	985
Total cold run time: 116269 ms
Total hot run time: 34708 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5161	5054	5194	5054
q2	248	325	243	243
q3	2145	2644	2257	2257
q4	1411	1827	1458	1458
q5	4468	4396	4419	4396
q6	223	173	135	135
q7	2007	1944	1749	1749
q8	2587	2575	2514	2514
q9	7219	7121	7126	7121
q10	2988	3177	2757	2757
q11	575	512	498	498
q12	707	765	626	626
q13	3522	3906	3340	3340
q14	325	307	278	278
q15	523	482	472	472
q16	452	500	446	446
q17	1185	1537	1421	1421
q18	7647	7590	7461	7461
q19	882	848	907	848
q20	2061	1970	1873	1873
q21	4846	4540	4370	4370
q22	1141	1069	1006	1006
Total cold run time: 52323 ms
Total hot run time: 50323 ms

@doris-robot
Copy link

doris-robot commented May 24, 2025

TPC-DS: Total hot run time: 193931 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4e353f81eebf089b65a54cea553d3f5927a27aae, data reload: false

query1	1406	1100	1101	1100
query2	6419	1817	1864	1817
query3	10994	4526	4435	4435
query4	52358	26260	23390	23390
query5	5135	556	457	457
query6	330	208	201	201
query7	4872	525	297	297
query8	292	245	228	228
query9	5309	2630	2608	2608
query10	433	340	267	267
query11	15044	15200	15054	15054
query12	162	113	110	110
query13	1037	590	428	428
query14	10288	6744	6584	6584
query15	210	190	184	184
query16	7111	645	483	483
query17	1085	768	587	587
query18	1531	412	313	313
query19	208	197	172	172
query20	139	159	125	125
query21	209	130	109	109
query22	4469	4544	4285	4285
query23	34284	33504	33613	33504
query24	6583	2433	2474	2433
query25	469	471	420	420
query26	695	278	155	155
query27	2298	533	380	380
query28	2987	2152	2164	2152
query29	619	591	470	470
query30	282	227	196	196
query31	867	884	811	811
query32	88	77	76	76
query33	483	402	336	336
query34	815	878	554	554
query35	798	868	770	770
query36	967	1002	914	914
query37	115	103	81	81
query38	4256	4377	4208	4208
query39	1566	1485	1469	1469
query40	224	130	114	114
query41	55	52	53	52
query42	122	110	116	110
query43	513	503	475	475
query44	1389	869	850	850
query45	197	184	173	173
query46	906	1028	674	674
query47	1903	1869	1780	1780
query48	418	440	318	318
query49	711	534	438	438
query50	709	723	422	422
query51	4297	4227	4351	4227
query52	114	112	106	106
query53	239	265	193	193
query54	599	596	514	514
query55	88	91	99	91
query56	327	318	289	289
query57	1189	1203	1149	1149
query58	283	268	305	268
query59	2736	2855	2705	2705
query60	356	357	332	332
query61	134	123	123	123
query62	738	733	673	673
query63	233	190	201	190
query64	1742	1043	719	719
query65	4314	4248	4253	4248
query66	757	402	310	310
query67	16170	15741	15577	15577
query68	7174	911	543	543
query69	553	304	278	278
query70	1265	1132	1167	1132
query71	567	332	303	303
query72	5921	4858	4987	4858
query73	1563	743	376	376
query74	9305	8982	8672	8672
query75	3842	3221	2686	2686
query76	4198	1262	775	775
query77	674	389	373	373
query78	10281	10261	9390	9390
query79	2205	836	589	589
query80	613	517	445	445
query81	483	269	234	234
query82	438	126	102	102
query83	252	254	247	247
query84	294	114	91	91
query85	817	357	332	332
query86	378	317	296	296
query87	4463	4557	4311	4311
query88	3959	2323	2312	2312
query89	413	321	281	281
query90	1806	211	212	211
query91	143	153	112	112
query92	75	60	62	60
query93	1775	932	582	582
query94	662	399	312	312
query95	383	302	288	288
query96	509	576	283	283
query97	2730	2781	2685	2685
query98	241	210	201	201
query99	1460	1387	1288	1288
Total cold run time: 298118 ms
Total hot run time: 193931 ms

@doris-robot
Copy link

doris-robot commented May 24, 2025

ClickBench: Total hot run time: 29.56 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4e353f81eebf089b65a54cea553d3f5927a27aae, data reload: false

query1	0.04	0.04	0.04
query2	0.12	0.11	0.11
query3	0.26	0.19	0.20
query4	1.59	0.20	0.11
query5	0.43	0.40	0.43
query6	1.17	0.66	0.66
query7	0.03	0.02	0.02
query8	0.04	0.04	0.03
query9	0.59	0.52	0.53
query10	0.57	0.58	0.57
query11	0.16	0.11	0.11
query12	0.15	0.11	0.11
query13	0.62	0.60	0.61
query14	0.78	0.81	0.82
query15	0.86	0.87	0.87
query16	0.38	0.38	0.39
query17	1.07	1.05	1.04
query18	0.23	0.20	0.20
query19	1.87	1.77	1.84
query20	0.02	0.02	0.01
query21	15.40	0.90	0.52
query22	0.77	1.16	0.65
query23	14.96	1.41	0.66
query24	6.71	1.67	1.13
query25	0.51	0.13	0.07
query26	0.61	0.16	0.16
query27	0.06	0.05	0.04
query28	9.68	0.94	0.45
query29	12.60	4.17	3.49
query30	0.26	0.09	0.07
query31	2.81	0.60	0.40
query32	3.23	0.55	0.47
query33	3.13	3.12	3.11
query34	15.76	5.05	4.46
query35	4.54	4.52	4.54
query36	0.68	0.50	0.49
query37	0.09	0.06	0.06
query38	0.06	0.04	0.04
query39	0.03	0.03	0.02
query40	0.17	0.14	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.03
query43	0.04	0.04	0.03
Total cold run time: 103.19 s
Total hot run time: 29.56 s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label May 24, 2025
@github-actions
Copy link
Contributor

github-actions bot commented May 24, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented May 24, 2025

PR approved by anyone and no changes requested.

yiguolei pushed a commit that referenced this pull request May 24, 2025
@morningman
[fix](export) remove task from map finally (#51171) (#51207)
701fc5f 
bp #51171
@dataroaring dataroaring merged commit 2f795bc into apache:master May 26, 2025
28 of 29 checks passed
dataroaring pushed a commit that referenced this pull request May 26, 2025
@morningman
[fix](export) remove task from map finally(#51171) (#51206)
857a0eb 
bp #51171
koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
@morningman @koarz
[fix](export) remove task from map finally (apache#51171)
0ce8329 
### What problem does this PR solve?

Related PR: apache#47974

Problem Summary:

When cancel task, the task should be removed from `taskExecutorMap`,
otherwise, if `tryPublishTask()` failed when `addMemoryTask`, the task
will
be leaked in `taskExecutorMap`
@gavinchou gavinchou mentioned this pull request Jun 11, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@CalvinKirs CalvinKirs CalvinKirs approved these changes

@wuwenchi wuwenchi wuwenchi approved these changes

@zy-kkk zy-kkk zy-kkk approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.11-merged dev/3.0.x-conflict dev/3.0.6-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

8 participants

@morningman @Thearas @doris-robot @CalvinKirs @wuwenchi @zy-kkk @yiguolei @dataroaring