Skip to content

[fix](compaction) use rowset meta FS in segcompaction and add RPC client ready check #55951

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 2 commits into from
Sep 12, 2025
Merged

[fix](compaction) use rowset meta FS in segcompaction and add RPC client ready check #55951

dataroaring merged 2 commits into from
Sep 12, 2025

Conversation

@luwei16
Copy link
Contributor

@luwei16 luwei16 commented Sep 12, 2025

1 Segcompaction uses the FS from rowset meta, leveraging the functionality provided by the FS abstraction layer.
2 Add an interface in thrift_rpc_helper to check whether the RPC client is ready, avoiding core dumps when BE has just started but the RPC client is not yet initialized.

@hello-stephen
Copy link
Contributor

hello-stephen commented Sep 12, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@luwei16
Copy link
Contributor Author

luwei16 commented Sep 12, 2025

run buildall

@luwei16
Copy link
Contributor Author

luwei16 commented Sep 12, 2025

run buildall

dataroaring
dataroaring approved these changes Sep 12, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 12, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Sep 12, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Sep 12, 2025

PR approved by anyone and no changes requested.

@doris-robot
Copy link

doris-robot commented Sep 12, 2025

TPC-H: Total hot run time: 35243 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6209a5cb38594eed3bee3a0ece4a71ab29ed495d, data reload: false

------ Round 1 ----------------------------------
q1	17682	5259	5181	5181
q2	2027	332	214	214
q3	10250	1313	723	723
q4	10232	1032	550	550
q5	7530	2406	2409	2406
q6	193	179	138	138
q7	925	796	639	639
q8	9365	1380	1084	1084
q9	7058	5221	5260	5221
q10	7016	2413	1978	1978
q11	503	322	289	289
q12	379	373	244	244
q13	17790	3711	3097	3097
q14	252	241	211	211
q15	596	491	495	491
q16	1001	1002	964	964
q17	617	888	380	380
q18	8031	7218	7199	7199
q19	1163	983	590	590
q20	356	344	243	243
q21	3891	3291	2419	2419
q22	1090	1053	982	982
Total cold run time: 107947 ms
Total hot run time: 35243 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5218	5157	5151	5151
q2	255	334	230	230
q3	2207	2692	2293	2293
q4	1393	1781	1378	1378
q5	4518	4535	4536	4535
q6	237	203	141	141
q7	2072	1960	1903	1903
q8	2710	2715	2687	2687
q9	7414	7346	7280	7280
q10	3123	3337	2899	2899
q11	594	529	512	512
q12	728	791	658	658
q13	3630	3994	3302	3302
q14	311	329	283	283
q15	539	499	488	488
q16	1059	1115	1070	1070
q17	1252	1739	1407	1407
q18	7991	7627	7197	7197
q19	835	869	1012	869
q20	1927	1962	1833	1833
q21	4748	4426	4217	4217
q22	1109	1055	978	978
Total cold run time: 53870 ms
Total hot run time: 51311 ms

@doris-robot
Copy link

doris-robot commented Sep 12, 2025

TPC-DS: Total hot run time: 189112 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6209a5cb38594eed3bee3a0ece4a71ab29ed495d, data reload: false

query1	1090	420	414	414
query2	6562	1670	1640	1640
query3	6765	228	213	213
query4	26744	23710	23136	23136
query5	4359	595	487	487
query6	337	231	234	231
query7	4649	498	311	311
query8	313	258	272	258
query9	8658	2640	2646	2640
query10	480	340	286	286
query11	15319	14957	14816	14816
query12	163	115	112	112
query13	1697	606	452	452
query14	10652	9244	9242	9242
query15	236	202	184	184
query16	7460	662	499	499
query17	1237	753	621	621
query18	2014	426	340	340
query19	203	235	174	174
query20	133	128	119	119
query21	215	137	122	122
query22	4028	4187	4172	4172
query23	33927	33162	33022	33022
query24	8518	2451	2467	2451
query25	585	525	437	437
query26	1229	276	164	164
query27	2744	506	355	355
query28	4367	2245	2213	2213
query29	808	621	498	498
query30	295	231	208	208
query31	915	827	749	749
query32	88	85	71	71
query33	595	384	352	352
query34	809	878	536	536
query35	835	829	747	747
query36	989	1007	920	920
query37	132	120	99	99
query38	3470	3554	3530	3530
query39	1511	1428	1427	1427
query40	230	141	127	127
query41	73	72	66	66
query42	129	120	124	120
query43	526	525	489	489
query44	1372	882	866	866
query45	193	182	176	176
query46	885	1029	669	669
query47	1757	1859	1716	1716
query48	411	440	317	317
query49	788	498	413	413
query50	671	698	402	402
query51	3917	3894	3889	3889
query52	116	114	105	105
query53	242	272	192	192
query54	609	590	544	544
query55	94	88	86	86
query56	319	320	302	302
query57	1207	1212	1126	1126
query58	280	269	267	267
query59	2620	2712	2598	2598
query60	346	333	320	320
query61	214	168	167	167
query62	888	739	662	662
query63	229	195	194	194
query64	4489	1129	801	801
query65	4034	3949	3992	3949
query66	1172	459	336	336
query67	15531	15088	15203	15088
query68	7912	954	588	588
query69	501	327	289	289
query70	1365	1361	1364	1361
query71	569	345	318	318
query72	5967	5087	5208	5087
query73	726	714	364	364
query74	8912	9199	8966	8966
query75	3854	3339	2817	2817
query76	3474	1319	734	734
query77	813	404	312	312
query78	9611	9477	8908	8908
query79	2587	842	598	598
query80	686	560	520	520
query81	500	265	226	226
query82	481	161	128	128
query83	270	260	246	246
query84	256	116	91	91
query85	883	473	416	416
query86	395	307	311	307
query87	3782	3779	3700	3700
query88	3643	2268	2242	2242
query89	414	328	301	301
query90	1837	222	221	221
query91	173	170	136	136
query92	83	69	65	65
query93	2162	991	654	654
query94	702	433	333	333
query95	403	318	305	305
query96	495	573	280	280
query97	2932	2967	2874	2874
query98	257	217	210	210
query99	1449	1421	1280	1280
Total cold run time: 276567 ms
Total hot run time: 189112 ms

@doris-robot
Copy link

doris-robot commented Sep 12, 2025

ClickBench: Total hot run time: 29.89 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6209a5cb38594eed3bee3a0ece4a71ab29ed495d, data reload: false

query1	0.06	0.05	0.05
query2	0.10	0.05	0.06
query3	0.25	0.09	0.08
query4	1.61	0.12	0.12
query5	0.28	0.28	0.27
query6	1.17	0.66	0.65
query7	0.04	0.03	0.03
query8	0.06	0.05	0.05
query9	0.63	0.52	0.52
query10	0.58	0.58	0.59
query11	0.16	0.11	0.11
query12	0.16	0.13	0.13
query13	0.62	0.64	0.62
query14	1.02	1.04	1.01
query15	0.87	0.86	0.86
query16	0.42	0.43	0.39
query17	1.11	1.07	1.05
query18	0.22	0.20	0.21
query19	1.96	1.88	1.87
query20	0.01	0.02	0.01
query21	15.43	0.95	0.58
query22	0.76	1.19	0.70
query23	14.91	1.43	0.66
query24	6.80	1.17	0.53
query25	0.48	0.23	0.07
query26	0.56	0.16	0.14
query27	0.06	0.05	0.07
query28	9.22	0.86	0.45
query29	12.57	3.92	3.32
query30	0.30	0.13	0.12
query31	2.84	0.59	0.39
query32	3.25	0.55	0.49
query33	3.09	3.13	3.07
query34	16.13	5.55	4.87
query35	4.93	4.95	4.90
query36	0.69	0.51	0.51
query37	0.12	0.08	0.07
query38	0.06	0.05	0.04
query39	0.03	0.04	0.03
query40	0.18	0.16	0.15
query41	0.09	0.03	0.03
query42	0.04	0.03	0.03
query43	0.05	0.04	0.04
Total cold run time: 103.92 s
Total hot run time: 29.89 s

@luwei16 luwei16 mentioned this pull request Sep 12, 2025
Closed
@doris-robot
Copy link

doris-robot commented Sep 12, 2025

BE UT Coverage Report

Increment line coverage 50.00% (1/2) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 52.20% (17427/33383)
Line Coverage 37.44% (158397/423090)
Region Coverage 32.03% (120878/377398)
Branch Coverage 33.38% (53004/158780)

@hello-stephen
Copy link
Contributor

hello-stephen commented Sep 12, 2025

BE Regression && UT Coverage Report

Increment line coverage 50.00% (1/2) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 70.83% (23121/32644)
Line Coverage 57.27% (241740/422078)
Region Coverage 52.80% (201901/382391)
Branch Coverage 54.33% (86654/159481)

@dataroaring dataroaring merged commit 7908019 into apache:master Sep 12, 2025
27 of 30 checks passed
github-actions bot pushed a commit that referenced this pull request Sep 12, 2025
@luwei16
[fix](compaction) use rowset meta FS in segcompaction and add RPC cli…
1ab794d 
…ent ready check (#55951)

1 Segcompaction uses the FS from rowset meta, leveraging the
functionality provided by the FS abstraction layer.
2 Add an interface in thrift_rpc_helper to check whether the RPC client
is ready, avoiding core dumps when BE has just started but the RPC
client is not yet initialized.
@github-actions github-actions bot mentioned this pull request Sep 12, 2025
Merged
morrySnow pushed a commit that referenced this pull request Sep 13, 2025
@github-actions @luwei16
branch-3.1: [fix](compaction) use rowset meta FS in segcompaction and…
1415893 
… add RPC client ready check #55951 (#55988)

Cherry-picked from #55951

Co-authored-by: Luwei <luwei@selectdb.com>
@morrySnow morrySnow mentioned this pull request Sep 22, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gavinchou gavinchou gavinchou approved these changes

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.1.1-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@luwei16 @hello-stephen @doris-robot @gavinchou @dataroaring @morrySnow