Skip to content

Conversation

@github-actions
Copy link
Contributor

@github-actions github-actions bot commented Dec 5, 2024

Cherry-picked from #44907

…pool for metadata fetching (#44907)

In production, we encountered an issue where the librdkafka consumer
stucked during destruction, causing the heavy work pool to become
saturated, which in turn made all heavy work pool-dependent
functionalities, such as querying, unusable. To mitigate this impact, we
replaced the heavy work pool with routine load threads for metadata
fetching.
@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring closed this Dec 5, 2024
@dataroaring dataroaring reopened this Dec 5, 2024
@doris-robot
Copy link

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40441 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8db6ea3a018600da17efefe4aecec65a8af7f702, data reload: false

------ Round 1 ----------------------------------
q1	17563	7329	7288	7288
q2	2056	164	179	164
q3	10687	1051	1195	1051
q4	10542	726	731	726
q5	7742	2824	2814	2814
q6	236	144	146	144
q7	948	611	605	605
q8	9573	1939	1925	1925
q9	7797	6341	6393	6341
q10	7034	2275	2291	2275
q11	456	263	262	262
q12	400	212	207	207
q13	17772	2960	2967	2960
q14	251	216	222	216
q15	557	507	527	507
q16	663	609	607	607
q17	950	641	550	550
q18	7080	6591	6560	6560
q19	1382	997	1065	997
q20	474	212	192	192
q21	3874	3086	3115	3086
q22	1098	983	964	964
Total cold run time: 109135 ms
Total hot run time: 40441 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7265	7238	7221	7221
q2	327	243	227	227
q3	2797	2872	2905	2872
q4	1971	1809	1788	1788
q5	5671	5728	5652	5652
q6	220	140	146	140
q7	2222	1798	1777	1777
q8	3302	3527	3447	3447
q9	8806	8807	8833	8807
q10	3555	3523	3464	3464
q11	594	495	491	491
q12	801	629	613	613
q13	15374	3144	3137	3137
q14	314	270	287	270
q15	568	532	534	532
q16	720	682	669	669
q17	1820	1624	1570	1570
q18	8210	7716	7445	7445
q19	8381	1621	1574	1574
q20	2074	1844	1824	1824
q21	5367	5068	5211	5068
q22	1130	1024	1018	1018
Total cold run time: 81489 ms
Total hot run time: 59606 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 195418 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8db6ea3a018600da17efefe4aecec65a8af7f702, data reload: false

query1	1243	915	918	915
query2	6262	2039	2014	2014
query3	10861	4000	4067	4000
query4	66455	29634	23506	23506
query5	5404	445	434	434
query6	435	175	181	175
query7	5657	306	307	306
query8	298	216	222	216
query9	9212	2677	2672	2672
query10	509	263	255	255
query11	17899	15204	15563	15204
query12	160	101	101	101
query13	1550	439	422	422
query14	11076	7200	7389	7200
query15	223	183	184	183
query16	6833	515	494	494
query17	1205	581	567	567
query18	1751	333	316	316
query19	255	162	157	157
query20	122	112	118	112
query21	215	107	102	102
query22	4836	4589	4458	4458
query23	34495	34149	34051	34051
query24	6148	2896	2864	2864
query25	556	428	407	407
query26	669	183	168	168
query27	1824	299	301	299
query28	4330	2566	2534	2534
query29	701	480	482	480
query30	233	166	170	166
query31	1011	813	804	804
query32	64	52	53	52
query33	411	267	268	267
query34	935	481	516	481
query35	859	721	738	721
query36	1080	920	937	920
query37	118	73	73	73
query38	4074	3970	4014	3970
query39	1486	1457	1518	1457
query40	200	94	99	94
query41	48	44	44	44
query42	109	93	100	93
query43	498	486	484	484
query44	1151	819	813	813
query45	183	167	162	162
query46	1132	709	727	709
query47	2009	1950	1922	1922
query48	460	367	369	367
query49	735	361	374	361
query50	848	409	403	403
query51	7355	7125	7103	7103
query52	108	89	84	84
query53	248	181	176	176
query54	556	438	433	433
query55	73	72	73	72
query56	241	226	232	226
query57	1215	1143	1077	1077
query58	198	199	194	194
query59	3111	2845	3022	2845
query60	269	243	241	241
query61	107	105	101	101
query62	756	646	645	645
query63	210	181	191	181
query64	1727	661	607	607
query65	3261	3159	3152	3152
query66	717	290	292	290
query67	15849	15567	15244	15244
query68	4567	559	544	544
query69	420	255	240	240
query70	1161	1139	1091	1091
query71	407	251	249	249
query72	6388	3977	3947	3947
query73	750	335	333	333
query74	10092	8865	8964	8865
query75	3273	2625	2630	2625
query76	2060	982	1046	982
query77	487	262	267	262
query78	10810	9703	9368	9368
query79	8498	583	589	583
query80	2243	410	407	407
query81	558	254	234	234
query82	1205	115	111	111
query83	312	138	142	138
query84	289	82	71	71
query85	1599	293	283	283
query86	504	284	298	284
query87	4436	4218	4235	4218
query88	5674	2362	2364	2362
query89	541	288	294	288
query90	2077	177	178	177
query91	178	142	144	142
query92	67	45	47	45
query93	6858	534	537	534
query94	914	279	298	279
query95	352	246	243	243
query96	621	279	279	279
query97	3303	3108	3211	3108
query98	212	204	195	195
query99	1749	1298	1285	1285
Total cold run time: 337434 ms
Total hot run time: 195418 ms

@dataroaring dataroaring merged commit fffe19a into branch-3.0 Dec 5, 2024
11 of 12 checks passed
@github-actions github-actions bot deleted the auto-pick-44907-branch-3.0 branch December 5, 2024 11:15
@gavinchou gavinchou mentioned this pull request Feb 18, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants