Skip to content

[chore](dep)upgrade netty and hadoop #57226

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
CalvinKirs merged 6 commits into from
Nov 3, 2025
Merged

[chore](dep)upgrade netty and hadoop #57226

CalvinKirs merged 6 commits into from
Nov 3, 2025

Conversation

@CalvinKirs
Copy link
Member

@CalvinKirs CalvinKirs commented Oct 22, 2025

No description provided.

@Thearas
Copy link
Contributor

Thearas commented Oct 22, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@CalvinKirs
Copy link
Member Author

CalvinKirs commented Oct 22, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Oct 22, 2025

TPC-H: Total hot run time: 16495 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 089c782c39a4ffdee5a75fcb2a78092c045c162b, data reload: false

------ Round 1 ----------------------------------
nation	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:27:24	NULL	utf-8	NULL	NULL	
customer	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:27:43	NULL	utf-8	NULL	NULL	
lineitem	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:38:59	NULL	utf-8	NULL	NULL	
supplier	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:27:25	NULL	utf-8	NULL	NULL	
region	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:27:24	NULL	utf-8	NULL	NULL	
orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	17632	5310	5115	5115
q2	2032	350	228	228
q3	10211	1327	774	774
q4	10238	876	381	381
q5	7493	2390	2398	2390
q6	188	169	141	141
q7	951	789	646	646
q8	9374	1391	1181	1181
q9	q10	11164	q11	q12	400	382	243	243
q13	24627	q14	243	241	227	227
q15	608	515	524	515
q16	q17	718	895	423	423
q18	7678	22533	q19	1296	977	602	602
q20	q21	13400	7388	2557	2557
q22	1096	1123	1072	1072
Total cold run time: 75880 ms
Total hot run time: 16495 ms

----- Round 2, with runtime_filter_mode=off -----
revenue0	View	NULL	NULL	-1	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:24	NULL	NULL	utf-8	NULL	NULL	
partsupp	Doris	NULL	NULL	80000000	56	4534120086	NULL	44625495	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:44:20	NULL	utf-8	NULL	NULL	
part	Doris	NULL	NULL	20000000	37	748811035	NULL	1935627	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:27:57	NULL	utf-8	NULL	NULL	
nation	Doris	NULL	NULL	25	138	3473	NULL	366	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:27:24	NULL	utf-8	NULL	NULL	
customer	Doris	NULL	NULL	15000000	92	1381653732	NULL	4374759	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:27:43	NULL	utf-8	NULL	NULL	
lineitem	Doris	NULL	NULL	600037902	33	19843441616	NULL	61784740	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:38:59	NULL	utf-8	NULL	NULL	
supplier	Doris	NULL	NULL	1000000	87	87519212	NULL	194931	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:27:25	NULL	utf-8	NULL	NULL	
region	Doris	NULL	NULL	5	240	1201	NULL	147	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:27:24	NULL	utf-8	NULL	NULL	
orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	5253	5272	q2	726	232	q3	4897	2355	2386	2355
q4	1453	1812	1384	1384
q5	4274	4177	4143	4143
q6	209	169	130	130
q7	1939	1890	1714	1714
q8	2538	2522	2629	2522
q9	q10	q11	q12	681	745	630	630
q13	q14	288	289	268	268
q15	562	493	495	493
q16	q17	1193	1563	1335	1335
q18	14613	q19	911	925	960	925
q20	q21	4874	q22	1146	1074	1009	1009
Total cold run time: 20091 ms
Total hot run time: 16908 ms

@doris-robot
Copy link

doris-robot commented Oct 22, 2025

TPC-DS: Total hot run time: 106778 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 089c782c39a4ffdee5a75fcb2a78092c045c162b, data reload: false

item	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 22:43:58	2023-12-26 22:44:05	NULL	utf-8	NULL	NULL	
date_dim	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 22:43:56	2023-12-26 22:44:01	NULL	utf-8	NULL	NULL	
call_center	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 22:43:57	2023-12-26 22:44:01	NULL	utf-8	NULL	NULL	
web_sales	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 22:43:58	2023-12-26 22:53:25	NULL	utf-8	NULL	NULL	
inventory	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 22:43:57	2023-12-26 22:50:32	NULL	utf-8	NULL	NULL	
store	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 22:43:59	2023-12-26 22:44:01	NULL	utf-8	NULL	NULL	
catalog_returns	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 22:43:57	2023-12-26 22:46:25	NULL	utf-8	NULL	NULL	
time_dim	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 22:43:59	2023-12-26 22:44:03	NULL	utf-8	NULL	NULL	
household_demographics	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 22:43:58	2023-12-26 22:44:01	NULL	utf-8	NULL	NULL	
store_sales	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 22:43:59	2023-12-26 22:59:20	NULL	utf-8	NULL	NULL	
ship_mode	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 22:44:00	2023-12-26 22:44:05	NULL	utf-8	NULL	NULL	
customer_address	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 22:43:58	2023-12-26 22:44:41	NULL	utf-8	NULL	NULL	
customer	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 22:44:00	2023-12-26 22:44:53	NULL	utf-8	NULL	NULL	
web_returns	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 22:43:58	2023-12-26 22:45:43	NULL	utf-8	NULL	NULL	
warehouse	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 22:43:56	2023-12-26 22:44:01	NULL	utf-8	NULL	NULL	
web_site	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 22:43:58	2023-12-26 22:44:01	NULL	utf-8	NULL	NULL	
customer_demographics	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 22:43:56	2023-12-26 22:44:34	NULL	utf-8	NULL	NULL	
promotion	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 22:43:58	2023-12-26 22:44:01	NULL	utf-8	NULL	NULL	
web_page	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 22:43:59	2023-12-26 22:44:01	NULL	utf-8	NULL	NULL	
store_returns	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 22:43:59	2023-12-26 22:47:09	NULL	utf-8	NULL	NULL	
income_band	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 22:43:58	2023-12-26 22:44:01	NULL	utf-8	NULL	NULL	
dbgen_version	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 22:44:00	2023-12-26 22:44:01	NULL	utf-8	NULL	NULL	
catalog_page	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 22:43:58	2023-12-26 22:44:01	NULL	utf-8	NULL	NULL	
reason	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 22:43:56	2023-12-26 22:44:01	NULL	utf-8	NULL	NULL	
============================================
query1	1120	421	417	417
query2	6611	1859	1828	1828
query3	6755	254	244	244
query4	26606	24334	query5	4565	703	524	524
query6	query7	4662	559	333	333
query8	351	261	279	261
query9	8745	2879	2851	2851
query10	516	419	310	310
query11	16135	query12	186	127	116	116
query13	1680	624	472	472
query14	query15	405	query16	7699	707	533	533
query17	query18	2415	391	358	358
query19	216	231	599	231
query20	140	331	225	225
query21	403	120	125	120
query22	16710	4184	4106	4106
query23	34762	83176	query24	8434	6106	query25	query26	1249	570	query27	query28	4537	2519	2483	2483
query29	853	686	query30	897	582	query31	query32	85	84	85	84
query33	1522	424	659	424
query34	query35	858	982	877	877
query36	1077	1080	1913	1080
query37	145	126	97	97
query38	3570	3629	3522	3522
query39	1515	query40	243	235	142	142
query41	72	69	69	69
query42	135	124	118	118
query43	548	548	514	514
query44	1357	832	834	832
query45	206	191	187	187
query46	971	1054	696	696
query47	1818	3497	1793	1793
query48	426	470	355	355
query49	790	547	473	473
query50	731	720	446	446
query51	3923	4002	3894	3894
query52	131	124	115	115
query53	272	288	215	215
query54	676	666	612	612
query55	127	94	97	94
query56	385	349	333	333
query57	1213	1218	1134	1134
query58	322	313	321	313
query59	2975	3068	3022	3022
query60	403	373	370	370
query61	184	157	150	150
query62	819	2731	693	693
query63	252	214	213	213
query64	4451	1246	883	883
query65	8039	4038	4017	4017
query66	query67	15491	15454	15400	15400
query68	8370	2298	686	686
query69	519	602	305	305
query70	query71	query72	13669	11665	9959	9959
query73	833	query74	8975	9179	8977	8977
query75	4224	3365	8433	3365
query76	4784	803	930	803
query77	816	443	649	443
query78	10038	9786	9059	9059
query79	3268	890	634	634
query80	1766	564	1041	564
query81	516	274	239	239
query82	query83	1041	278	275	275
query84	311	135	255	135
query85	query86	query87	3849	3769	3652	3652
query88	4010	2436	2441	2436
query89	415	342	318	318
query90	1888	244	237	237
query91	174	183	query92	93	72	79	72
query93	2816	1144	750	750
query94	1260	487	383	383
query95	489	394	386	386
query96	591	668	309	309
query97	3011	3097	2905	2905
query98	query99	1445	1438	1307	1307
Total cold run time: 199692 ms
Total hot run time: 106778 ms

@CalvinKirs
Copy link
Member Author

CalvinKirs commented Oct 31, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Oct 31, 2025

ClickBench: Total hot run time: 28.01 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 841f70068c4d2c65a7ae58c3354fee30ab9121cd, data reload: false

query1	0.06	0.05	0.05
query2	0.10	0.06	0.06
query3	0.25	0.08	0.09
query4	1.61	0.12	0.12
query5	0.28	0.27	0.27
query6	1.17	0.67	0.65
query7	0.03	0.03	0.03
query8	0.05	0.04	0.05
query9	0.62	0.53	0.53
query10	0.59	0.58	0.58
query11	0.17	0.11	0.12
query12	0.16	0.13	0.12
query13	0.62	0.61	0.61
query14	1.01	1.01	1.01
query15	0.85	0.83	0.86
query16	0.39	0.40	0.39
query17	1.04	1.05	1.05
query18	0.21	0.20	0.20
query19	1.93	1.79	1.87
query20	0.02	0.01	0.02
query21	15.46	0.20	0.12
query22	4.95	0.07	0.05
query23	15.66	0.26	0.11
query24	2.77	0.91	0.88
query25	0.10	0.05	0.05
query26	0.15	0.14	0.13
query27	0.07	0.06	0.06
query28	5.08	1.14	0.92
query29	12.60	3.96	3.23
query30	0.28	0.13	0.12
query31	2.82	0.59	0.39
query32	3.24	0.54	0.48
query33	3.09	3.03	3.00
query34	15.86	5.14	4.49
query35	4.56	4.58	4.68
query36	0.69	0.50	0.50
query37	0.10	0.07	0.07
query38	0.06	0.05	0.04
query39	0.04	0.03	0.04
query40	0.19	0.14	0.15
query41	0.09	0.03	0.03
query42	0.04	0.03	0.03
query43	0.04	0.03	0.04
Total cold run time: 99.1 s
Total hot run time: 28.01 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Oct 31, 2025

FE UT Coverage Report

Increment line coverage `` 🎉
Increment coverage report
Complete coverage report

@CalvinKirs
Copy link
Member Author

CalvinKirs commented Oct 31, 2025

run buildall

morningman
morningman previously approved these changes Oct 31, 2025
View reviewed changes
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Oct 31, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Oct 31, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Oct 31, 2025

PR approved by anyone and no changes requested.

@CalvinKirs
Copy link
Member Author

CalvinKirs commented Oct 31, 2025

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Oct 31, 2025
@doris-robot
Copy link

doris-robot commented Oct 31, 2025

TPC-DS: Total hot run time: 189690 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit fb3f587c7af63a940bfa42b7b431abcbc6eeae2c, data reload: false

query1	1034	405	411	405
query2	6565	1671	1728	1671
query3	6758	223	220	220
query4	26379	23386	23457	23386
query5	5863	616	477	477
query6	340	253	257	253
query7	4655	499	306	306
query8	324	259	259	259
query9	8754	2590	2585	2585
query10	545	333	289	289
query11	15333	15273	14797	14797
query12	186	122	118	118
query13	1690	575	439	439
query14	11662	9308	9356	9308
query15	224	192	178	178
query16	7750	701	571	571
query17	1800	824	672	672
query18	2078	462	365	365
query19	263	228	193	193
query20	146	141	148	141
query21	342	146	117	117
query22	4701	4554	4543	4543
query23	34971	33954	33790	33790
query24	8449	2576	2542	2542
query25	657	531	474	474
query26	1301	290	171	171
query27	2693	538	351	351
query28	4407	2240	2222	2222
query29	799	632	547	547
query30	305	256	214	214
query31	944	842	817	817
query32	89	70	70	70
query33	563	380	319	319
query34	802	845	513	513
query35	785	843	761	761
query36	947	995	893	893
query37	126	111	87	87
query38	3534	3529	3437	3437
query39	1478	1406	1425	1406
query40	237	127	115	115
query41	58	54	55	54
query42	117	111	107	107
query43	496	499	465	465
query44	1223	740	728	728
query45	183	179	167	167
query46	881	993	633	633
query47	1737	1794	1731	1731
query48	389	435	330	330
query49	786	523	392	392
query50	636	694	403	403
query51	3905	4008	3890	3890
query52	108	105	99	99
query53	234	267	190	190
query54	315	287	274	274
query55	84	84	83	83
query56	316	331	304	304
query57	1169	1181	1105	1105
query58	288	282	273	273
query59	2551	2683	2487	2487
query60	340	350	329	329
query61	157	152	160	152
query62	769	735	659	659
query63	231	190	203	190
query64	4443	1175	857	857
query65	4015	3968	3924	3924
query66	1025	428	379	379
query67	15383	15146	15088	15088
query68	8649	896	594	594
query69	504	324	290	290
query70	1392	1225	1269	1225
query71	523	356	311	311
query72	5968	4945	4852	4852
query73	698	565	354	354
query74	9269	9064	8735	8735
query75	4171	3358	2810	2810
query76	3862	1164	741	741
query77	811	427	311	311
query78	9572	9752	8978	8978
query79	2113	838	601	601
query80	684	585	495	495
query81	495	261	225	225
query82	432	163	133	133
query83	289	272	248	248
query84	297	105	90	90
query85	890	488	433	433
query86	364	318	299	299
query87	3800	3769	3686	3686
query88	3225	2258	2267	2258
query89	403	324	299	299
query90	2011	216	219	216
query91	165	169	137	137
query92	75	73	67	67
query93	1258	975	644	644
query94	682	460	339	339
query95	403	331	316	316
query96	498	577	283	283
query97	2937	2952	2900	2900
query98	253	217	209	209
query99	1459	1407	1296	1296
Total cold run time: 280470 ms
Total hot run time: 189690 ms

@doris-robot
Copy link

doris-robot commented Oct 31, 2025

ClickBench: Total hot run time: 27.85 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit fb3f587c7af63a940bfa42b7b431abcbc6eeae2c, data reload: false

query1	0.05	0.05	0.05
query2	0.09	0.05	0.05
query3	0.26	0.09	0.09
query4	1.62	0.11	0.12
query5	0.28	0.25	0.25
query6	1.17	0.64	0.64
query7	0.03	0.03	0.03
query8	0.05	0.05	0.04
query9	0.60	0.52	0.52
query10	0.58	0.58	0.58
query11	0.17	0.11	0.11
query12	0.15	0.13	0.12
query13	0.62	0.60	0.60
query14	1.01	1.01	1.02
query15	0.85	0.84	0.84
query16	0.40	0.40	0.39
query17	1.03	1.04	1.00
query18	0.21	0.20	0.19
query19	1.91	1.90	1.90
query20	0.01	0.02	0.02
query21	15.45	0.21	0.13
query22	5.04	0.07	0.05
query23	15.71	0.27	0.10
query24	3.22	0.56	0.49
query25	0.08	0.07	0.06
query26	0.14	0.12	0.14
query27	0.08	0.05	0.04
query28	5.02	1.16	0.94
query29	12.57	3.97	3.30
query30	0.28	0.13	0.14
query31	2.82	0.63	0.38
query32	3.23	0.55	0.48
query33	2.97	3.07	3.09
query34	15.97	5.21	4.61
query35	4.56	4.57	4.57
query36	0.68	0.50	0.49
query37	0.10	0.06	0.07
query38	0.06	0.04	0.04
query39	0.04	0.03	0.03
query40	0.17	0.14	0.13
query41	0.08	0.03	0.04
query42	0.04	0.03	0.03
query43	0.04	0.03	0.04
Total cold run time: 99.44 s
Total hot run time: 27.85 s

@CalvinKirs
Copy link
Member Author

CalvinKirs commented Oct 31, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Oct 31, 2025

TPC-DS: Total hot run time: 190007 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 70f9a857b6a061c67b660a0c7b2a76cb277b2fc1, data reload: false

query1	1085	411	397	397
query2	6565	1741	1686	1686
query3	6754	229	224	224
query4	26465	23887	23447	23447
query5	5849	624	480	480
query6	351	256	231	231
query7	4661	525	301	301
query8	311	260	251	251
query9	8756	2612	2576	2576
query10	548	347	291	291
query11	15343	15144	14775	14775
query12	176	121	113	113
query13	1677	583	445	445
query14	11746	9159	9166	9159
query15	231	194	173	173
query16	7796	737	575	575
query17	1918	850	693	693
query18	2103	459	378	378
query19	335	222	204	204
query20	128	125	125	125
query21	216	143	118	118
query22	4768	4771	4641	4641
query23	34801	33926	34018	33926
query24	8519	2492	2514	2492
query25	655	533	469	469
query26	1596	273	164	164
query27	3277	590	374	374
query28	4890	2317	2248	2248
query29	800	638	507	507
query30	318	234	205	205
query31	992	885	784	784
query32	92	80	74	74
query33	597	405	328	328
query34	853	858	518	518
query35	787	812	779	779
query36	939	997	914	914
query37	117	103	84	84
query38	3519	3530	3476	3476
query39	1467	1425	1420	1420
query40	220	124	115	115
query41	60	60	56	56
query42	120	109	106	106
query43	488	513	464	464
query44	1190	738	738	738
query45	181	177	173	173
query46	887	1001	628	628
query47	1774	1851	1715	1715
query48	392	422	306	306
query49	771	515	402	402
query50	627	682	404	404
query51	4001	3956	3843	3843
query52	108	110	96	96
query53	227	264	203	203
query54	301	278	270	270
query55	88	85	81	81
query56	309	310	304	304
query57	1168	1188	1129	1129
query58	289	280	267	267
query59	2541	2701	2621	2621
query60	341	342	319	319
query61	155	147	153	147
query62	826	739	682	682
query63	232	188	190	188
query64	4406	1280	966	966
query65	4032	3977	3987	3977
query66	1049	453	362	362
query67	15407	15342	15010	15010
query68	8596	903	600	600
query69	536	322	292	292
query70	1330	1320	1304	1304
query71	529	342	331	331
query72	5977	4875	4877	4875
query73	654	570	355	355
query74	8956	8829	8934	8829
query75	4138	3361	2873	2873
query76	3905	1155	717	717
query77	819	415	301	301
query78	9498	9831	8920	8920
query79	2015	838	598	598
query80	638	564	489	489
query81	492	257	223	223
query82	436	166	132	132
query83	262	269	247	247
query84	253	120	87	87
query85	883	486	432	432
query86	335	334	288	288
query87	3718	3774	3588	3588
query88	3576	2247	2232	2232
query89	375	328	292	292
query90	2020	214	222	214
query91	164	167	135	135
query92	83	66	60	60
query93	1660	983	648	648
query94	668	441	318	318
query95	395	325	313	313
query96	497	583	278	278
query97	2964	2986	2876	2876
query98	243	210	210	210
query99	1430	1406	1333	1333
Total cold run time: 282236 ms
Total hot run time: 190007 ms

@doris-robot
Copy link

doris-robot commented Oct 31, 2025

ClickBench: Total hot run time: 27.84 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 70f9a857b6a061c67b660a0c7b2a76cb277b2fc1, data reload: false

query1	0.06	0.05	0.05
query2	0.09	0.05	0.05
query3	0.26	0.09	0.09
query4	1.61	0.11	0.11
query5	0.27	0.27	0.26
query6	1.14	0.65	0.63
query7	0.03	0.03	0.02
query8	0.06	0.04	0.04
query9	0.60	0.52	0.52
query10	0.58	0.58	0.58
query11	0.16	0.11	0.12
query12	0.15	0.12	0.12
query13	0.62	0.61	0.61
query14	1.01	1.01	1.00
query15	0.87	0.84	0.85
query16	0.39	0.39	0.40
query17	1.03	1.01	1.04
query18	0.21	0.19	0.20
query19	1.88	1.77	1.80
query20	0.02	0.01	0.01
query21	15.43	0.19	0.14
query22	5.06	0.07	0.04
query23	15.69	0.25	0.10
query24	2.80	0.54	0.80
query25	0.07	0.06	0.06
query26	0.14	0.13	0.12
query27	0.06	0.05	0.06
query28	5.01	1.14	0.93
query29	12.57	3.97	3.33
query30	0.28	0.13	0.12
query31	2.82	0.59	0.38
query32	3.23	0.56	0.48
query33	3.07	3.15	3.09
query34	15.92	5.17	4.59
query35	4.53	4.61	4.62
query36	0.67	0.51	0.50
query37	0.10	0.07	0.07
query38	0.07	0.04	0.04
query39	0.04	0.03	0.03
query40	0.18	0.14	0.13
query41	0.09	0.04	0.02
query42	0.04	0.03	0.03
query43	0.05	0.04	0.04
Total cold run time: 98.96 s
Total hot run time: 27.84 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Oct 31, 2025

FE UT Coverage Report

Increment line coverage `` 🎉
Increment coverage report
Complete coverage report

@doris-robot
Copy link

doris-robot commented Oct 31, 2025

ClickBench: Total hot run time: 27.73 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f1fef412ea74c2f6dca97a029b106e5ae6b2bc68, data reload: false

query1	0.05	0.05	0.05
query2	0.09	0.05	0.04
query3	0.25	0.08	0.09
query4	1.60	0.12	0.12
query5	0.27	0.27	0.25
query6	1.17	0.65	0.65
query7	0.03	0.03	0.02
query8	0.05	0.04	0.04
query9	0.61	0.53	0.53
query10	0.59	0.58	0.58
query11	0.17	0.11	0.12
query12	0.16	0.12	0.12
query13	0.62	0.61	0.60
query14	1.00	1.00	1.00
query15	0.85	0.84	0.85
query16	0.40	0.38	0.45
query17	1.05	0.99	1.08
query18	0.22	0.21	0.20
query19	1.90	1.85	1.80
query20	0.02	0.01	0.02
query21	15.43	0.18	0.13
query22	5.09	0.07	0.05
query23	15.64	0.26	0.10
query24	2.19	1.31	0.47
query25	0.09	0.07	0.06
query26	0.14	0.14	0.13
query27	0.07	0.06	0.06
query28	4.42	1.14	0.94
query29	12.65	4.01	3.35
query30	0.28	0.13	0.12
query31	2.82	0.61	0.39
query32	3.23	0.55	0.47
query33	3.03	3.03	3.17
query34	15.82	5.11	4.62
query35	4.58	4.57	4.55
query36	0.69	0.51	0.49
query37	0.09	0.07	0.07
query38	0.07	0.04	0.03
query39	0.03	0.03	0.03
query40	0.17	0.14	0.14
query41	0.09	0.04	0.03
query42	0.04	0.03	0.03
query43	0.04	0.04	0.03
Total cold run time: 97.8 s
Total hot run time: 27.73 s

@CalvinKirs
Copy link
Member Author

CalvinKirs commented Nov 1, 2025

run buildall

1 similar comment
@CalvinKirs
Copy link
Member Author

CalvinKirs commented Nov 1, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Nov 1, 2025

ClickBench: Total hot run time: 27.94 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 68f56c1fb8d9f87b9848626005e13c1b4c17641d, data reload: false

query1	0.05	0.05	0.05
query2	0.09	0.05	0.05
query3	0.26	0.09	0.08
query4	1.61	0.12	0.11
query5	0.27	0.26	0.25
query6	1.16	0.64	0.65
query7	0.03	0.03	0.02
query8	0.06	0.04	0.04
query9	0.60	0.53	0.53
query10	0.58	0.59	0.57
query11	0.17	0.11	0.12
query12	0.16	0.12	0.12
query13	0.62	0.60	0.61
query14	1.01	1.00	1.02
query15	0.86	0.85	0.84
query16	0.39	0.38	0.42
query17	1.03	1.01	1.03
query18	0.22	0.20	0.21
query19	2.00	1.79	1.84
query20	0.01	0.02	0.01
query21	15.65	0.21	0.13
query22	4.78	0.07	0.05
query23	15.67	0.26	0.10
query24	2.80	0.59	0.79
query25	0.07	0.06	0.06
query26	0.14	0.12	0.14
query27	0.06	0.06	0.05
query28	5.50	1.15	0.93
query29	12.54	3.97	3.38
query30	0.28	0.13	0.12
query31	2.81	0.58	0.39
query32	3.23	0.56	0.47
query33	3.15	3.10	3.12
query34	15.92	5.15	4.60
query35	4.56	4.64	4.60
query36	0.66	0.50	0.49
query37	0.10	0.07	0.07
query38	0.07	0.04	0.04
query39	0.04	0.03	0.03
query40	0.18	0.15	0.13
query41	0.09	0.04	0.03
query42	0.04	0.04	0.03
query43	0.04	0.03	0.04
Total cold run time: 99.56 s
Total hot run time: 27.94 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Nov 1, 2025

FE UT Coverage Report

Increment line coverage `` 🎉
Increment coverage report
Complete coverage report

@github-actions
Copy link
Contributor

github-actions bot commented Nov 1, 2025

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 1, 2025
@morningman morningman changed the title [chore](dep)upgrade netty [chore](dep)upgrade netty and hadoop Nov 3, 2025
@CalvinKirs CalvinKirs merged commit 781f428 into apache:master Nov 3, 2025
27 of 29 checks passed
github-actions bot pushed a commit that referenced this pull request Nov 3, 2025
@CalvinKirs
[chore](dep)upgrade netty and hadoop (#57226)
7b1ec30 
upgrade netty and hadoop
@github-actions github-actions bot mentioned this pull request Nov 3, 2025
Merged
github-actions bot pushed a commit that referenced this pull request Nov 3, 2025
@CalvinKirs
[chore](dep)upgrade netty and hadoop (#57226)
22d4aae 
upgrade netty and hadoop
@github-actions github-actions bot mentioned this pull request Nov 3, 2025
Merged
CalvinKirs added a commit that referenced this pull request Nov 5, 2025
@github-actions @CalvinKirs
branch-3.1: [chore](dep)upgrade netty and hadoop #57226 (#57608)
fb43dc5 
Cherry-picked from #57226

Co-authored-by: Calvin Kirs <guoqiang@selectdb.com>
yiguolei pushed a commit that referenced this pull request Dec 2, 2025
@github-actions @CalvinKirs
branch-4.0: [chore](dep)upgrade netty and hadoop #57226 (#57609)
0c1283a 
Cherry-picked from #57226

Co-authored-by: Calvin Kirs <guoqiang@selectdb.com>
@CalvinKirs CalvinKirs mentioned this pull request Dec 2, 2025
Merged
CalvinKirs added a commit that referenced this pull request Dec 2, 2025
@CalvinKirs
[fix](S3)Downgrade the AWS SDK version to avoid impacting other S3-co…
21d56a5 
…mpatible services. (#58566)

…

This PR addresses an issue related to AWS SDK for Java v2 S3 client
behavior described in:
aws/aws-sdk-java-v2#5805

Starting from newer versions of the SDK, the S3 client introduces
unexpected behavior that causes our application to fail during normal S3
operations. Due to this problem, we were forced to downgrade aws-s3 from
the latest version to 2.29.x, which restored correct functionality. The
regression affects stability in our production environment and prevents
us from upgrading the SDK.

#57226
github-actions bot pushed a commit that referenced this pull request Dec 2, 2025
@CalvinKirs
[fix](S3)Downgrade the AWS SDK version to avoid impacting other S3-co…
0fc8db7 
…mpatible services. (#58566)

…

This PR addresses an issue related to AWS SDK for Java v2 S3 client
behavior described in:
aws/aws-sdk-java-v2#5805

Starting from newer versions of the SDK, the S3 client introduces
unexpected behavior that causes our application to fail during normal S3
operations. Due to this problem, we were forced to downgrade aws-s3 from
the latest version to 2.29.x, which restored correct functionality. The
regression affects stability in our production environment and prevents
us from upgrading the SDK.

#57226
github-actions bot pushed a commit that referenced this pull request Dec 2, 2025
@CalvinKirs
[fix](S3)Downgrade the AWS SDK version to avoid impacting other S3-co…
05ee88b 
…mpatible services. (#58566)

…

This PR addresses an issue related to AWS SDK for Java v2 S3 client
behavior described in:
aws/aws-sdk-java-v2#5805

Starting from newer versions of the SDK, the S3 client introduces
unexpected behavior that causes our application to fail during normal S3
operations. Due to this problem, we were forced to downgrade aws-s3 from
the latest version to 2.29.x, which restored correct functionality. The
regression affects stability in our production environment and prevents
us from upgrading the SDK.

#57226
nagisa-kunhah pushed a commit to nagisa-kunhah/doris that referenced this pull request Dec 14, 2025
@CalvinKirs @nagisa-kunhah
[fix](S3)Downgrade the AWS SDK version to avoid impacting other S3-co…
97d7a96 
…mpatible services. (apache#58566)

…

This PR addresses an issue related to AWS SDK for Java v2 S3 client
behavior described in:
aws/aws-sdk-java-v2#5805

Starting from newer versions of the SDK, the S3 client introduces
unexpected behavior that causes our application to fail during normal S3
operations. Due to this problem, we were forced to downgrade aws-s3 from
the latest version to 2.29.x, which restored correct functionality. The
regression affects stability in our production environment and prevents
us from upgrading the SDK.

apache#57226
@CalvinKirs CalvinKirs mentioned this pull request Jan 7, 2026
Merged
16 tasks
CalvinKirs added a commit that referenced this pull request Jan 7, 2026
@CalvinKirs
[Fix](catalog)Resources should be closed when dropping a Catalog. (#5…
40191ed 
…9512)

### What problem does this PR solve?
When creating an AWS SDK V2 async client (e.g. S3AsyncClient), the SDK
requires a thread pool to manage asynchronous task scheduling, timeouts,
and retries (e.g. ScheduledExecutorService or async executor).


If no executor is explicitly provided, the AWS SDK creates its own
internal thread pool, which is expected to be shut down when
client.close() is invoked.

### Issue

In Doris, when using Paimon Catalog, some catalog implementations
provide an empty close() method. As a result, when a user executes DROP
CATALOG:

starting with Hadoop 3.4, the AWS SDK was upgraded to v2. Since #57226
upgraded HDFS to 3.4.2, the catalog runs into this issue.

Some Catalog instance is discarded,AWS SDK client.close() is never
called,The internally created thread pool cannot be shut down.

This leads to a thread leak

FYI aws/aws-sdk-java-v2#3746

### Problem Analysis

The lifecycle of Catalog creation and destruction is complex and managed
internally by Paimon

Doris cannot reliably intervene in the call chain to enforce AWS SDK
client.close()

If each Catalog creates its own AWS SDK client with an internally
managed thread pool, threads will continue to leak as Catalogs are
repeatedly created and dropped

 ### Solution

This PR resolves the issue by introducing a shared executor strategy:

A Doris-managed shared thread pool is explicitly passed when creating
AWS SDK V2 clients


This prevents the AWS SDK from implicitly creating per-client internal
thread pools

The lifecycle of the executor is fully controlled by Doris and no longer
depends on Paimon Catalog’s close() implementation


With this approach, even if a Paimon Catalog’s close() method is a
no-op, the system will no longer leak threads.
None
zzzxl1993 pushed a commit to zzzxl1993/doris that referenced this pull request Jan 13, 2026
@CalvinKirs @zzzxl1993
[Fix](catalog)Resources should be closed when dropping a Catalog. (ap…
7e66892 
…ache#59512)

### What problem does this PR solve?
When creating an AWS SDK V2 async client (e.g. S3AsyncClient), the SDK
requires a thread pool to manage asynchronous task scheduling, timeouts,
and retries (e.g. ScheduledExecutorService or async executor).


If no executor is explicitly provided, the AWS SDK creates its own
internal thread pool, which is expected to be shut down when
client.close() is invoked.

### Issue

In Doris, when using Paimon Catalog, some catalog implementations
provide an empty close() method. As a result, when a user executes DROP
CATALOG:

starting with Hadoop 3.4, the AWS SDK was upgraded to v2. Since apache#57226
upgraded HDFS to 3.4.2, the catalog runs into this issue.

Some Catalog instance is discarded,AWS SDK client.close() is never
called,The internally created thread pool cannot be shut down.

This leads to a thread leak

FYI aws/aws-sdk-java-v2#3746

### Problem Analysis

The lifecycle of Catalog creation and destruction is complex and managed
internally by Paimon

Doris cannot reliably intervene in the call chain to enforce AWS SDK
client.close()

If each Catalog creates its own AWS SDK client with an internally
managed thread pool, threads will continue to leak as Catalogs are
repeatedly created and dropped

 ### Solution

This PR resolves the issue by introducing a shared executor strategy:

A Doris-managed shared thread pool is explicitly passed when creating
AWS SDK V2 clients


This prevents the AWS SDK from implicitly creating per-client internal
thread pools

The lifecycle of the executor is fully controlled by Doris and no longer
depends on Paimon Catalog’s close() implementation


With this approach, even if a Paimon Catalog’s close() method is a
no-op, the system will no longer leak threads.
None
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morningman morningman morningman approved these changes

+1 more reviewer

@hubgeter hubgeter hubgeter approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.1.3-merged dev/4.0.2-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@CalvinKirs @Thearas @doris-robot @hello-stephen @morningman @hubgeter @yiguolei