Skip to content

branch-3.1: [opt](multi-catalog) Optimize remote scan concurrency #51415 #52518

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morrySnow merged 1 commit into from
Jul 1, 2025
Merged

branch-3.1: [opt](multi-catalog) Optimize remote scan concurrency #51415 #52518

morrySnow merged 1 commit into from
Jul 1, 2025

Conversation

@kaka11chen
Copy link
Contributor

@kaka11chen kaka11chen commented Jun 30, 2025

What problem does this PR solve?

Problem Summary:

Release note

Cherry-pick #51415

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@kaka11chen kaka11chen requested a review from morrySnow as a code owner June 30, 2025 08:49
@hello-stephen
Copy link
Contributor

hello-stephen commented Jun 30, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@kaka11chen
Copy link
Contributor Author

kaka11chen commented Jun 30, 2025

run buildall

@kaka11chen
[opt](multi-catalog) Optimize remote scan concurrency. (apache#51415)
8053750 
Problem Summary:

[opt] (multi-catalog) Optimize remote scan concurrency.
1. Use `ScannerScheduler::get_remote_scan_thread_num()` to replace
`config::doris_scanner_thread_pool_thread_num` when calculate max
scanners in the external table case.
2. Remove `parallel_scan_max_scanners_count` calculation logic.
@kaka11chen kaka11chen force-pushed the cherry-pick-51415_3.1 branch from aca5e3e to 8053750 Compare June 30, 2025 08:59
@kaka11chen
Copy link
Contributor Author

kaka11chen commented Jun 30, 2025

run buidall

@doris-robot
Copy link

doris-robot commented Jun 30, 2025

BE UT Coverage Report

Increment line coverage 72.73% (8/11) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 45.02% (12353/27440)
Line Coverage 36.03% (109414/303646)
Region Coverage 35.08% (56694/161620)
Branch Coverage 32.20% (30754/95508)

@morningman morningman closed this Jul 1, 2025
@morningman morningman reopened this Jul 1, 2025
@morningman
Copy link
Contributor

morningman commented Jul 1, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jul 1, 2025

TPC-H: Total hot run time: 39682 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8053750773b61cdadaccc8afaa6b083ee18f1891, data reload: false

------ Round 1 ----------------------------------
q1	17604	6755	6603	6603
q2	2058	194	197	194
q3	10494	1107	1212	1107
q4	10312	757	818	757
q5	7703	2846	2714	2714
q6	212	132	134	132
q7	982	624	603	603
q8	9359	1876	1977	1876
q9	6622	6355	6379	6355
q10	7063	2238	2269	2238
q11	458	262	259	259
q12	407	214	207	207
q13	17786	2953	2971	2953
q14	243	210	203	203
q15	504	457	467	457
q16	477	375	376	375
q17	970	548	568	548
q18	7187	6613	6644	6613
q19	1307	989	976	976
q20	471	201	199	199
q21	3868	3310	3348	3310
q22	1099	1003	1007	1003
Total cold run time: 107186 ms
Total hot run time: 39682 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6595	6611	6590	6590
q2	324	233	233	233
q3	2898	2877	2866	2866
q4	2084	1797	1830	1797
q5	5706	5676	5695	5676
q6	202	128	132	128
q7	2172	1839	1746	1746
q8	3372	3514	3445	3445
q9	8934	8891	8938	8891
q10	3585	3509	3516	3509
q11	614	493	481	481
q12	839	643	599	599
q13	6071	3192	3163	3163
q14	315	279	289	279
q15	516	461	469	461
q16	487	430	431	430
q17	1828	1610	1631	1610
q18	8246	7860	7751	7751
q19	1648	1463	1553	1463
q20	2055	1897	1883	1883
q21	5129	5078	4955	4955
q22	1145	1039	991	991
Total cold run time: 64765 ms
Total hot run time: 58947 ms

@doris-robot
Copy link

doris-robot commented Jul 1, 2025

BE UT Coverage Report

Increment line coverage 72.73% (8/11) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 44.98% (12351/27460)
Line Coverage 36.01% (109398/303833)
Region Coverage 35.06% (56680/161673)
Branch Coverage 32.19% (30751/95532)

@doris-robot
Copy link

doris-robot commented Jul 1, 2025

TPC-DS: Total hot run time: 196331 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8053750773b61cdadaccc8afaa6b083ee18f1891, data reload: false

query1	1302	911	914	911
query2	6336	1952	1888	1888
query3	10922	4362	4469	4362
query4	33084	23523	23660	23523
query5	3730	468	449	449
query6	273	172	184	172
query7	4012	318	324	318
query8	286	238	237	237
query9	9472	2583	2561	2561
query10	473	274	264	264
query11	17919	15306	15278	15278
query12	160	101	110	101
query13	1551	420	421	420
query14	8846	6686	7385	6686
query15	277	184	188	184
query16	8053	473	505	473
query17	1626	609	621	609
query18	2172	345	326	326
query19	252	171	185	171
query20	127	116	114	114
query21	204	113	106	106
query22	4577	4348	4272	4272
query23	34703	33999	34441	33999
query24	11863	3058	3005	3005
query25	726	429	425	425
query26	1884	183	178	178
query27	2806	358	361	358
query28	8106	2164	2122	2122
query29	1097	479	469	469
query30	279	165	160	160
query31	1054	837	857	837
query32	102	61	59	59
query33	809	329	321	321
query34	981	518	521	518
query35	858	753	762	753
query36	1120	963	947	947
query37	196	68	71	68
query38	4085	3981	3946	3946
query39	1485	1466	1479	1466
query40	270	102	102	102
query41	49	46	57	46
query42	127	111	107	107
query43	533	476	475	475
query44	1260	804	821	804
query45	191	169	180	169
query46	1170	752	738	738
query47	1979	1926	1885	1885
query48	437	342	340	340
query49	1025	406	421	406
query50	835	425	442	425
query51	7525	7191	7242	7191
query52	101	93	90	90
query53	266	188	186	186
query54	1389	478	470	470
query55	79	81	87	81
query56	279	280	254	254
query57	1315	1217	1232	1217
query58	237	215	226	215
query59	3172	3082	3075	3075
query60	298	264	261	261
query61	127	176	122	122
query62	871	686	697	686
query63	219	196	196	196
query64	4965	684	645	645
query65	3371	3378	3268	3268
query66	1373	298	303	298
query67	15807	15648	15308	15308
query68	5368	570	586	570
query69	424	270	270	270
query70	1188	1125	1113	1113
query71	358	276	274	274
query72	6251	3986	4081	3986
query73	757	351	360	351
query74	9784	9314	8947	8947
query75	3359	2659	2642	2642
query76	3016	1004	922	922
query77	397	288	274	274
query78	10534	9518	9513	9513
query79	2229	615	618	615
query80	1151	427	434	427
query81	550	237	218	218
query82	927	90	92	90
query83	233	149	144	144
query84	237	74	79	74
query85	1420	309	298	298
query86	465	287	288	287
query87	4353	4184	4228	4184
query88	4093	2401	2387	2387
query89	416	298	291	291
query90	2031	189	186	186
query91	139	109	111	109
query92	68	49	51	49
query93	2263	554	557	554
query94	962	294	311	294
query95	354	254	258	254
query96	612	279	282	279
query97	3252	3126	3154	3126
query98	225	203	199	199
query99	1509	1344	1288	1288
Total cold run time: 306003 ms
Total hot run time: 196331 ms

@doris-robot
Copy link

doris-robot commented Jul 1, 2025

ClickBench: Total hot run time: 29.99 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8053750773b61cdadaccc8afaa6b083ee18f1891, data reload: false

query1	0.03	0.03	0.03
query2	0.06	0.04	0.03
query3	0.23	0.07	0.07
query4	1.65	0.11	0.10
query5	0.50	0.51	0.52
query6	1.14	0.73	0.73
query7	0.02	0.01	0.02
query8	0.05	0.03	0.03
query9	0.58	0.50	0.49
query10	0.55	0.55	0.56
query11	0.15	0.11	0.11
query12	0.14	0.11	0.11
query13	0.60	0.61	0.59
query14	0.78	0.79	0.80
query15	0.84	0.82	0.83
query16	0.39	0.38	0.37
query17	1.08	1.05	1.06
query18	0.23	0.20	0.21
query19	1.92	1.82	1.76
query20	0.02	0.01	0.01
query21	15.40	0.59	0.58
query22	2.12	1.98	2.55
query23	16.78	1.33	0.80
query24	3.01	0.56	1.82
query25	0.24	0.05	0.04
query26	0.41	0.13	0.14
query27	0.05	0.05	0.05
query28	10.22	0.48	0.54
query29	12.59	3.27	3.24
query30	0.24	0.06	0.06
query31	2.85	0.38	0.38
query32	3.23	0.46	0.46
query33	2.94	3.03	3.04
query34	16.95	4.49	4.52
query35	4.56	4.56	4.57
query36	0.68	0.49	0.48
query37	0.09	0.06	0.06
query38	0.05	0.04	0.03
query39	0.04	0.02	0.02
query40	0.17	0.12	0.12
query41	0.09	0.02	0.02
query42	0.03	0.03	0.03
query43	0.04	0.03	0.03
Total cold run time: 103.74 s
Total hot run time: 29.99 s

@morrySnow morrySnow changed the title [opt](multi-catalog) Optimize remote scan concurrency. branch-3.1: [opt](multi-catalog) Optimize remote scan concurrency #51415 Jul 1, 2025
@morrySnow morrySnow merged commit 8c430e3 into apache:branch-3.1 Jul 1, 2025
31 of 34 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@kaka11chen @hello-stephen @doris-robot @morningman @morrySnow