Skip to content

branch-3.1: [fix](compaction) use rowset meta FS in segcompaction and add RPC client ready check #55951 #55988

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morrySnow merged 1 commit into from
Sep 13, 2025
Merged

branch-3.1: [fix](compaction) use rowset meta FS in segcompaction and add RPC client ready check #55951 #55988

morrySnow merged 1 commit into from
Sep 13, 2025

Conversation

@github-actions
Copy link
Contributor

@github-actions github-actions bot commented Sep 12, 2025

Cherry-picked from #55951

@luwei16
[fix](compaction) use rowset meta FS in segcompaction and add RPC cli…
1ab794d 
…ent ready check (#55951)

1 Segcompaction uses the FS from rowset meta, leveraging the
functionality provided by the FS abstraction layer.
2 Add an interface in thrift_rpc_helper to check whether the RPC client
is ready, avoiding core dumps when BE has just started but the RPC
client is not yet initialized.
@github-actions github-actions bot requested a review from morrySnow as a code owner September 12, 2025 13:54
@hello-stephen
Copy link
Contributor

hello-stephen commented Sep 12, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring reopened this Sep 12, 2025
@hello-stephen
Copy link
Contributor

hello-stephen commented Sep 12, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Sep 12, 2025

TPC-H: Total hot run time: 32562 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1ab794d5a3c77a1edec28cc911dd16ade0bb950c, data reload: false

------ Round 1 ----------------------------------
q1	17580	5461	5419	5419
q2	2022	402	274	274
q3	11961	1247	735	735
q4	10520	872	444	444
q5	9571	2410	2154	2154
q6	190	161	128	128
q7	885	759	610	610
q8	9321	1422	1125	1125
q9	5269	5003	4912	4912
q10	6823	2285	1836	1836
q11	468	289	272	272
q12	345	358	220	220
q13	17752	3589	2997	2997
q14	233	230	211	211
q15	535	473	470	470
q16	425	428	371	371
q17	604	853	356	356
q18	6787	6307	6321	6307
q19	1223	962	561	561
q20	327	339	207	207
q21	2796	2203	1953	1953
q22	1054	1047	1000	1000
Total cold run time: 106691 ms
Total hot run time: 32562 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5510	5483	5467	5467
q2	236	322	249	249
q3	2263	2565	2321	2321
q4	1313	1799	1377	1377
q5	4397	4955	4975	4955
q6	173	164	129	129
q7	2089	1954	1810	1810
q8	2707	2915	2767	2767
q9	7436	7489	7303	7303
q10	3047	3279	2794	2794
q11	591	528	509	509
q12	701	807	620	620
q13	3543	3839	3218	3218
q14	298	302	273	273
q15	526	469	470	469
q16	449	490	438	438
q17	1237	1715	1274	1274
q18	7745	7528	7189	7189
q19	828	1193	1088	1088
q20	2006	2044	1944	1944
q21	5292	5003	4622	4622
q22	1052	1070	1008	1008
Total cold run time: 53439 ms
Total hot run time: 51824 ms

@doris-robot
Copy link

doris-robot commented Sep 12, 2025

TPC-DS: Total hot run time: 192227 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1ab794d5a3c77a1edec28cc911dd16ade0bb950c, data reload: false

query1	961	406	405	405
query2	6115	1911	1867	1867
query3	8689	210	208	208
query4	33412	23844	23422	23422
query5	3721	610	477	477
query6	291	187	177	177
query7	4206	488	329	329
query8	330	238	240	238
query9	9386	2585	2571	2571
query10	471	350	249	249
query11	18067	15452	15237	15237
query12	167	106	108	106
query13	1555	541	428	428
query14	9370	6649	6731	6649
query15	284	190	180	180
query16	8069	634	547	547
query17	1548	751	615	615
query18	2159	431	324	324
query19	226	190	167	167
query20	126	125	121	121
query21	210	130	105	105
query22	4613	4646	4301	4301
query23	35040	34158	33942	33942
query24	7833	2775	2752	2752
query25	534	496	439	439
query26	893	294	177	177
query27	2991	467	348	348
query28	5496	2225	2187	2187
query29	745	600	475	475
query30	255	195	159	159
query31	957	942	826	826
query32	103	62	60	60
query33	497	407	309	309
query34	760	867	542	542
query35	798	810	748	748
query36	1013	1063	965	965
query37	101	91	70	70
query38	4069	4150	4014	4014
query39	1529	1489	1502	1489
query40	209	121	109	109
query41	50	49	48	48
query42	114	107	106	106
query43	500	501	480	480
query44	1313	854	848	848
query45	186	187	184	184
query46	882	1066	677	677
query47	1960	1975	1915	1915
query48	407	434	358	358
query49	770	503	393	393
query50	686	690	445	445
query51	7356	7282	7267	7267
query52	104	97	88	88
query53	231	269	204	204
query54	548	537	478	478
query55	82	79	80	79
query56	275	289	271	271
query57	1276	1276	1232	1232
query58	246	218	219	218
query59	3075	3117	3002	3002
query60	317	277	268	268
query61	136	116	127	116
query62	800	776	696	696
query63	229	192	196	192
query64	4042	1031	662	662
query65	3428	3306	3310	3306
query66	1007	410	318	318
query67	16431	15723	15647	15647
query68	7916	831	549	549
query69	493	312	302	302
query70	1154	1080	1131	1080
query71	434	304	261	261
query72	5208	3763	3802	3763
query73	633	749	356	356
query74	10478	9421	9199	9199
query75	3916	3166	2690	2690
query76	3452	1181	754	754
query77	766	356	289	289
query78	10297	10789	9584	9584
query79	3248	868	601	601
query80	667	517	438	438
query81	505	262	217	217
query82	477	118	91	91
query83	165	163	146	146
query84	285	93	78	78
query85	790	363	302	302
query86	386	320	290	290
query87	4318	4381	4252	4252
query88	4946	2425	2411	2411
query89	401	334	296	296
query90	1723	191	189	189
query91	134	135	118	118
query92	69	55	52	52
query93	1785	887	552	552
query94	637	409	310	310
query95	337	277	265	265
query96	498	606	290	290
query97	3191	3258	3161	3161
query98	226	204	200	200
query99	1525	1416	1309	1309
Total cold run time: 295310 ms
Total hot run time: 192227 ms

@doris-robot
Copy link

doris-robot commented Sep 12, 2025

ClickBench: Total hot run time: 28.09 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1ab794d5a3c77a1edec28cc911dd16ade0bb950c, data reload: false

query1	0.04	0.03	0.02
query2	0.07	0.03	0.03
query3	0.23	0.06	0.06
query4	1.62	0.10	0.11
query5	0.50	0.51	0.50
query6	1.14	0.74	0.72
query7	0.02	0.02	0.02
query8	0.05	0.04	0.03
query9	0.56	0.50	0.48
query10	0.56	0.54	0.54
query11	0.14	0.10	0.10
query12	0.13	0.10	0.11
query13	0.62	0.60	0.58
query14	0.79	0.81	0.80
query15	0.85	0.83	0.81
query16	0.42	0.39	0.39
query17	1.06	0.99	1.08
query18	0.24	0.23	0.22
query19	1.89	1.82	1.78
query20	0.01	0.01	0.01
query21	15.39	0.94	0.58
query22	0.74	0.83	0.64
query23	15.08	1.45	0.61
query24	3.23	1.25	0.30
query25	0.10	0.08	0.05
query26	0.27	0.16	0.13
query27	0.05	0.05	0.05
query28	12.11	1.05	0.43
query29	12.58	3.86	3.24
query30	0.25	0.09	0.06
query31	2.81	0.60	0.38
query32	3.22	0.53	0.46
query33	2.98	3.02	3.06
query34	16.68	5.21	4.50
query35	4.56	4.56	4.59
query36	0.64	0.50	0.48
query37	0.10	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.02
query40	0.16	0.14	0.13
query41	0.08	0.03	0.03
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 102.12 s
Total hot run time: 28.09 s

@doris-robot
Copy link

doris-robot commented Sep 12, 2025

BE UT Coverage Report

Increment line coverage 50.00% (1/2) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 45.53% (12760/28024)
Line Coverage 36.36% (113751/312827)
Region Coverage 33.99% (65057/191382)
Branch Coverage 31.03% (34144/110050)

@hello-stephen
Copy link
Contributor

hello-stephen commented Sep 12, 2025

BE Regression && UT Coverage Report

Increment line coverage 50.00% (1/2) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 66.16% (18234/27562)
Line Coverage 57.81% (180284/311856)
Region Coverage 55.52% (106691/192159)
Branch Coverage 49.87% (55134/110564)

@luwei16
Copy link
Contributor

luwei16 commented Sep 13, 2025

run cloud_p0

@hello-stephen
Copy link
Contributor

hello-stephen commented Sep 13, 2025

BE Regression && UT Coverage Report

Increment line coverage 50.00% (1/2) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 66.16% (18234/27562)
Line Coverage 57.81% (180284/311856)
Region Coverage 55.52% (106691/192159)
Branch Coverage 49.87% (55134/110564)

dataroaring
dataroaring approved these changes Sep 13, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@morrySnow morrySnow merged commit 1415893 into branch-3.1 Sep 13, 2025
21 of 23 checks passed
@morrySnow morrySnow deleted the auto-pick-55951-branch-3.1 branch September 13, 2025 16:24
@morrySnow morrySnow mentioned this pull request Sep 22, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaring dataroaring dataroaring approved these changes

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@hello-stephen @doris-robot @luwei16 @dataroaring @morrySnow