Skip to content

[opt](load) Increase the default value of webserver_num_workers #46304

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
liaoxin01 merged 1 commit into from
Jan 8, 2025
Merged

[opt](load) Increase the default value of webserver_num_workers #46304

liaoxin01 merged 1 commit into from
Jan 8, 2025

Conversation

@liaoxin01
Copy link
Contributor

@liaoxin01 liaoxin01 commented Jan 2, 2025 •
edited
Loading

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

  1. In high-concurrency stream load scenarios, there aren't enough webserver threads available.
  2. During stream load, after data reception and while waiting for execution, the synchronous wait occupies HTTP threads.

To address the issue of insufficient HTTP threads during high-concurrency stream loads causing slow data reception, we increased the number of threads.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

hello-stephen commented Jan 2, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@liaoxin01
Copy link
Contributor Author

liaoxin01 commented Jan 2, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jan 2, 2025

TPC-H: Total hot run time: 32735 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5307ca404676deb8db5c69dbd2c2721b3b23635a, data reload: false

------ Round 1 ----------------------------------
q1	17758	6750	6020	6020
q2	2048	314	190	190
q3	10385	1255	743	743
q4	10222	893	451	451
q5	7502	2203	2003	2003
q6	206	187	155	155
q7	908	767	610	610
q8	9244	1378	1196	1196
q9	5263	4828	4926	4828
q10	6774	2290	1854	1854
q11	486	279	269	269
q12	353	366	225	225
q13	17761	3626	3151	3151
q14	244	242	212	212
q15	564	514	492	492
q16	640	612	595	595
q17	587	853	352	352
q18	7055	6458	6270	6270
q19	1546	965	586	586
q20	334	338	204	204
q21	2879	2187	2012	2012
q22	364	330	317	317
Total cold run time: 103123 ms
Total hot run time: 32735 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6271	6212	6200	6200
q2	238	331	233	233
q3	2175	2638	2323	2323
q4	1384	1810	1319	1319
q5	4383	4770	4829	4770
q6	184	174	142	142
q7	2070	2020	1917	1917
q8	2620	2750	2636	2636
q9	7290	7234	7289	7234
q10	3060	3360	2806	2806
q11	589	506	507	506
q12	648	773	585	585
q13	3471	3901	3203	3203
q14	279	290	287	287
q15	579	504	502	502
q16	640	693	650	650
q17	1217	1720	1236	1236
q18	7697	7491	7203	7203
q19	835	1068	1098	1068
q20	1950	1988	1843	1843
q21	5330	4983	4838	4838
q22	610	589	570	570
Total cold run time: 53520 ms
Total hot run time: 52071 ms

@doris-robot
Copy link

doris-robot commented Jan 2, 2025

TPC-DS: Total hot run time: 190969 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5307ca404676deb8db5c69dbd2c2721b3b23635a, data reload: false

query1	984	392	380	380
query2	6530	2404	2460	2404
query3	6705	218	214	214
query4	33598	23740	23693	23693
query5	4375	651	472	472
query6	287	211	184	184
query7	4649	515	319	319
query8	307	259	235	235
query9	9641	2633	2638	2633
query10	460	320	285	285
query11	18253	15541	15255	15255
query12	170	108	109	108
query13	1641	529	404	404
query14	9744	6779	7036	6779
query15	237	207	184	184
query16	8216	633	443	443
query17	1574	759	556	556
query18	2122	406	298	298
query19	202	177	151	151
query20	120	126	113	113
query21	210	126	107	107
query22	4406	4363	4331	4331
query23	34411	33413	33313	33313
query24	6889	2305	2302	2302
query25	504	470	385	385
query26	1196	287	156	156
query27	2306	465	330	330
query28	5249	2442	2419	2419
query29	755	561	413	413
query30	232	189	156	156
query31	998	916	811	811
query32	99	59	61	59
query33	524	360	288	288
query34	780	876	501	501
query35	828	830	719	719
query36	1005	1036	929	929
query37	118	103	114	103
query38	4146	4279	4162	4162
query39	1463	1450	1417	1417
query40	203	117	104	104
query41	47	44	44	44
query42	124	103	104	103
query43	535	565	505	505
query44	1366	795	781	781
query45	197	176	164	164
query46	917	1089	640	640
query47	1938	1909	1822	1822
query48	383	413	322	322
query49	787	485	406	406
query50	690	677	390	390
query51	7157	7096	6959	6959
query52	107	103	99	99
query53	237	268	199	199
query54	508	496	422	422
query55	82	86	88	86
query56	277	286	265	265
query57	1217	1184	1126	1126
query58	244	233	243	233
query59	3154	3312	3201	3201
query60	306	281	273	273
query61	140	132	139	132
query62	894	808	748	748
query63	260	188	193	188
query64	4661	1015	678	678
query65	3274	3202	3260	3202
query66	1079	431	306	306
query67	15959	15818	15493	15493
query68	8443	796	510	510
query69	470	293	261	261
query70	1213	1183	1144	1144
query71	440	305	278	278
query72	6522	3948	3862	3862
query73	662	796	363	363
query74	9866	9030	8976	8976
query75	3904	3166	2688	2688
query76	3608	1280	819	819
query77	855	373	278	278
query78	10132	10042	9483	9483
query79	3046	871	593	593
query80	751	536	438	438
query81	478	283	228	228
query82	628	164	129	129
query83	201	169	149	149
query84	288	102	84	84
query85	791	400	305	305
query86	371	328	315	315
query87	4402	4462	4693	4462
query88	4230	2164	2150	2150
query89	441	335	303	303
query90	1899	187	198	187
query91	139	140	106	106
query92	69	60	55	55
query93	1381	869	538	538
query94	657	409	305	305
query95	336	263	256	256
query96	484	670	285	285
query97	2905	3004	2875	2875
query98	231	207	204	204
query99	1748	1556	1438	1438
Total cold run time: 294268 ms
Total hot run time: 190969 ms

@doris-robot
Copy link

doris-robot commented Jan 2, 2025

ClickBench: Total hot run time: 30.79 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5307ca404676deb8db5c69dbd2c2721b3b23635a, data reload: false

query1	0.04	0.03	0.02
query2	0.07	0.04	0.03
query3	0.24	0.07	0.06
query4	1.62	0.10	0.12
query5	0.43	0.41	0.39
query6	1.15	0.66	0.64
query7	0.02	0.02	0.01
query8	0.04	0.03	0.04
query9	0.58	0.50	0.53
query10	0.56	0.56	0.54
query11	0.14	0.10	0.11
query12	0.14	0.11	0.11
query13	0.61	0.61	0.59
query14	2.83	2.83	2.71
query15	0.88	0.83	0.83
query16	0.38	0.39	0.40
query17	1.05	1.04	1.07
query18	0.22	0.21	0.21
query19	1.97	1.86	2.02
query20	0.01	0.01	0.01
query21	15.36	0.94	0.60
query22	0.75	0.73	0.62
query23	15.42	1.42	0.61
query24	3.47	0.41	1.18
query25	0.18	0.21	0.17
query26	0.32	0.15	0.13
query27	0.05	0.07	0.08
query28	13.37	1.55	1.05
query29	12.58	3.93	3.26
query30	0.26	0.09	0.06
query31	2.83	0.61	0.38
query32	3.29	0.54	0.46
query33	3.13	3.14	3.07
query34	16.71	5.14	4.49
query35	4.49	4.44	4.50
query36	0.64	0.48	0.49
query37	0.10	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.03
query40	0.17	0.14	0.13
query41	0.07	0.03	0.02
query42	0.04	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.33 s
Total hot run time: 30.79 s

@doris-robot
Copy link

doris-robot commented Jan 2, 2025

TeamCity be ut coverage result:
Function Coverage: 38.89% (10127/26039)
Line Coverage: 29.91% (85633/286309)
Region Coverage: 29.02% (43709/150622)
Branch Coverage: 25.56% (22312/87280)
Coverage Report: http://coverage.selectdb-in.cc/coverage/5307ca404676deb8db5c69dbd2c2721b3b23635a_5307ca404676deb8db5c69dbd2c2721b3b23635a/report/index.html

dataroaring
dataroaring approved these changes Jan 7, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 7, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Jan 7, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Jan 7, 2025

PR approved by anyone and no changes requested.

sollhui
sollhui approved these changes Jan 8, 2025
View reviewed changes
Copy link
Contributor

@sollhui sollhui left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@liaoxin01 liaoxin01 merged commit 8322683 into apache:master Jan 8, 2025
29 of 32 checks passed
@liaoxin01 liaoxin01 deleted the http_thread branch January 8, 2025 03:54
github-actions bot pushed a commit that referenced this pull request Jan 8, 2025
@liaoxin01
[opt](load) Increase the default value of webserver_num_workers (#46304)
2d7e08a 
1. In high-concurrency stream load scenarios, there aren't enough
webserver threads available.
2. During stream load, after data reception and while waiting for
execution, the synchronous wait occupies HTTP threads.

To address the issue of insufficient HTTP threads during
high-concurrency stream loads causing slow data reception, we increased
the number of threads.
@github-actions github-actions bot mentioned this pull request Jan 8, 2025
Merged
github-actions bot pushed a commit that referenced this pull request Jan 8, 2025
@liaoxin01
[opt](load) Increase the default value of webserver_num_workers (#46304)
fffdbf1 
1. In high-concurrency stream load scenarios, there aren't enough
webserver threads available.
2. During stream load, after data reception and while waiting for
execution, the synchronous wait occupies HTTP threads.

To address the issue of insufficient HTTP threads during
high-concurrency stream loads causing slow data reception, we increased
the number of threads.
@github-actions github-actions bot mentioned this pull request Jan 8, 2025
Closed
yiguolei pushed a commit that referenced this pull request Jan 8, 2025
@liaoxin01 @yiguolei
[opt](load) Increase the default value of webserver_num_workers (#46304)
d57122d 
1. In high-concurrency stream load scenarios, there aren't enough
webserver threads available.
2. During stream load, after data reception and while waiting for
execution, the synchronous wait occupies HTTP threads.

To address the issue of insufficient HTTP threads during
high-concurrency stream loads causing slow data reception, we increased
the number of threads.
liaoxin01 added a commit that referenced this pull request Jan 9, 2025
@github-actions @liaoxin01
branch-3.0: [opt](load) Increase the default value of webserver_num_w…
ade6209 
…orkers #46304 (#46593)

Cherry-picked from #46304

Co-authored-by: Xin Liao <liaoxin@selectdb.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaring dataroaring dataroaring approved these changes

+1 more reviewer

@sollhui sollhui sollhui approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.0.4-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@liaoxin01 @hello-stephen @doris-robot @sollhui @dataroaring