Skip to content

[fix](delete) Fix missing prepare_rowset for delete job #59089

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 1 commit into from
Dec 16, 2025
Merged

[fix](delete) Fix missing prepare_rowset for delete job #59089

dataroaring merged 1 commit into from
Dec 16, 2025
+7 −1

Conversation

@wyxxxcat
Copy link
Contributor

@wyxxxcat wyxxxcat commented Dec 16, 2025 •
edited by gavinchou
Loading

load, compaction, schema change
use workflow with prepare_rowset -> commit_rowset
delete job needs to be consistent even though the delete can work without prepare rowset

@hello-stephen
Copy link
Contributor

hello-stephen commented Dec 16, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

gavinchou
gavinchou reviewed Dec 16, 2025
View reviewed changes
@gavinchou gavinchou changed the title [feat](delete) Add prepare_rowset for delete job [fix](delete) Fix missing prepare_rowset for delete job Dec 16, 2025
@wyxxxcat wyxxxcat force-pushed the delete_job_prepare_rowset branch from 8e06b21 to 9cdfbca Compare December 16, 2025 09:53
@wyxxxcat
Copy link
Contributor Author

wyxxxcat commented Dec 16, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Dec 16, 2025

TPC-H: Total hot run time: 35506 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9cdfbcaacecf042806593a3955bde63a6e133be8, data reload: false

------ Round 1 ----------------------------------
q1	17641	4205	4102	4102
q2	2043	356	238	238
q3	10159	1336	759	759
q4	10219	863	318	318
q5	7516	2143	1956	1956
q6	189	174	141	141
q7	1014	868	720	720
q8	9350	1501	1151	1151
q9	7060	5359	5339	5339
q10	6789	2407	1960	1960
q11	524	320	298	298
q12	663	723	613	613
q13	17779	3700	3074	3074
q14	289	294	281	281
q15	595	519	513	513
q16	719	682	632	632
q17	688	779	573	573
q18	7757	7103	7052	7052
q19	939	953	614	614
q20	401	360	251	251
q21	4285	3932	3937	3932
q22	1045	995	989	989
Total cold run time: 107664 ms
Total hot run time: 35506 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4231	4071	4098	4071
q2	342	396	311	311
q3	2165	2638	2265	2265
q4	1298	1746	1289	1289
q5	4254	4610	4788	4610
q6	218	170	130	130
q7	2051	2000	1859	1859
q8	2712	2654	2603	2603
q9	7776	7593	7496	7496
q10	3135	3336	2840	2840
q11	595	512	485	485
q12	702	790	639	639
q13	3669	3958	3470	3470
q14	282	297	290	290
q15	581	536	499	499
q16	639	702	648	648
q17	1198	1439	1418	1418
q18	8105	7726	7750	7726
q19	911	908	906	906
q20	2054	2117	1939	1939
q21	4841	4292	4165	4165
q22	1077	1034	977	977
Total cold run time: 52836 ms
Total hot run time: 50636 ms

@doris-robot
Copy link

doris-robot commented Dec 16, 2025

TPC-DS: Total hot run time: 179602 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9cdfbcaacecf042806593a3955bde63a6e133be8, data reload: false

query5	4753	623	479	479
query6	349	252	232	232
query7	4236	467	277	277
query8	326	291	251	251
query9	8787	2554	2583	2554
query10	484	405	330	330
query11	15322	14777	14632	14632
query12	176	121	121	121
query13	1276	515	439	439
query14	6305	3021	2780	2780
query14_1	2673	2682	2708	2682
query15	212	201	185	185
query16	921	481	461	461
query17	1151	721	611	611
query18	2550	447	351	351
query19	233	234	221	221
query20	126	117	118	117
query21	224	144	116	116
query22	3980	4007	3963	3963
query23	16584	16192	16156	16156
query23_1	16149	16123	16284	16123
query24	7361	1683	1253	1253
query24_1	1270	1281	1243	1243
query25	594	505	441	441
query26	1256	277	171	171
query27	2730	461	315	315
query28	4417	2165	2156	2156
query29	845	575	474	474
query30	331	249	222	222
query31	819	724	617	617
query32	81	83	69	69
query33	558	348	338	338
query34	898	893	538	538
query35	765	825	721	721
query36	873	919	834	834
query37	120	98	79	79
query38	2879	2839	2813	2813
query39	750	745	700	700
query39_1	712	870	693	693
query40	224	139	121	121
query41	67	66	62	62
query42	107	104	109	104
query43	439	432	408	408
query44	1362	784	740	740
query45	194	199	184	184
query46	888	986	616	616
query47	1661	1742	1616	1616
query48	334	340	245	245
query49	645	435	347	347
query50	679	303	215	215
query51	3855	3839	3776	3776
query52	105	109	97	97
query53	321	354	297	297
query54	279	265	283	265
query55	83	75	72	72
query56	296	305	299	299
query57	1178	1169	1073	1073
query58	280	258	253	253
query59	2404	2504	2435	2435
query60	317	313	291	291
query61	165	159	163	159
query62	720	660	622	622
query63	330	307	300	300
query64	5041	1328	1004	1004
query65	4021	3952	3982	3952
query66	1430	436	331	331
query67	15222	14991	15025	14991
query68	8345	997	729	729
query69	485	349	313	313
query70	1090	992	1003	992
query71	369	310	278	278
query72	6127	4926	5036	4926
query73	671	605	309	309
query74	8512	8815	8822	8815
query75	3238	3176	2819	2819
query76	4018	1153	741	741
query77	609	396	302	302
query78	9426	9563	8913	8913
query79	1614	929	629	629
query80	735	666	549	549
query81	492	264	233	233
query82	205	137	111	111
query83	265	262	245	245
query84	265	130	111	111
query85	900	517	463	463
query86	386	296	261	261
query87	3026	3115	2971	2971
query88	3803	2305	2304	2304
query89	499	438	402	402
query90	2201	161	153	153
query91	175	177	145	145
query92	78	67	69	67
query93	2191	909	556	556
query94	480	302	279	279
query95	588	391	311	311
query96	589	492	214	214
query97	2229	2304	2262	2262
query98	212	194	191	191
query99	1283	1277	1247	1247
Total cold run time: 261868 ms
Total hot run time: 179602 ms

@doris-robot
Copy link

doris-robot commented Dec 16, 2025

ClickBench: Total hot run time: 27.53 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9cdfbcaacecf042806593a3955bde63a6e133be8, data reload: false

query1	0.05	0.05	0.06
query2	0.10	0.05	0.05
query3	0.25	0.09	0.09
query4	1.61	0.12	0.11
query5	0.27	0.28	0.26
query6	1.16	0.63	0.63
query7	0.02	0.03	0.02
query8	0.06	0.04	0.04
query9	0.56	0.52	0.52
query10	0.55	0.55	0.55
query11	0.15	0.10	0.11
query12	0.16	0.11	0.12
query13	0.61	0.60	0.61
query14	0.98	0.99	0.98
query15	0.81	0.80	0.80
query16	0.40	0.40	0.40
query17	1.02	1.02	1.03
query18	0.25	0.22	0.22
query19	1.93	1.88	1.83
query20	0.01	0.01	0.01
query21	15.45	0.30	0.15
query22	4.67	0.05	0.05
query23	15.97	0.28	0.10
query24	1.74	0.32	0.28
query25	0.08	0.05	0.05
query26	0.14	0.14	0.14
query27	0.09	0.06	0.06
query28	3.95	1.23	1.03
query29	12.62	3.93	3.32
query30	0.28	0.13	0.12
query31	2.82	0.63	0.40
query32	3.23	0.54	0.47
query33	2.96	3.04	3.10
query34	16.91	5.25	4.53
query35	4.55	4.58	4.57
query36	0.67	0.50	0.49
query37	0.10	0.06	0.06
query38	0.07	0.04	0.04
query39	0.05	0.03	0.02
query40	0.17	0.14	0.14
query41	0.10	0.03	0.02
query42	0.05	0.03	0.03
query43	0.04	0.03	0.04
Total cold run time: 97.66 s
Total hot run time: 27.53 s

@doris-robot
Copy link

doris-robot commented Dec 16, 2025

BE UT Coverage Report

Increment line coverage 0.00% (0/6) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 53.45% (18850/35267)
Line Coverage 39.24% (174600/444956)
Region Coverage 33.77% (135005/399799)
Branch Coverage 34.71% (58192/167675)

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 16, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Dec 16, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Dec 16, 2025

PR approved by anyone and no changes requested.

@hello-stephen
Copy link
Contributor

hello-stephen commented Dec 16, 2025

BE Regression && UT Coverage Report

Increment line coverage 50.00% (3/6) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 72.27% (24975/34560)
Line Coverage 58.99% (262124/444341)
Region Coverage 53.86% (217835/404426)
Branch Coverage 55.34% (93237/168486)

liaoxin01
liaoxin01 approved these changes Dec 16, 2025
View reviewed changes
Copy link
Contributor

@liaoxin01 liaoxin01 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@hello-stephen
Copy link
Contributor

hello-stephen commented Dec 16, 2025

BE Regression && UT Coverage Report

Increment line coverage 50.00% (3/6) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 72.30% (24987/34560)
Line Coverage 59.04% (262320/444341)
Region Coverage 53.84% (217732/404426)
Branch Coverage 55.38% (93307/168486)

@dataroaring dataroaring merged commit 99d14a8 into apache:master Dec 16, 2025
33 of 35 checks passed
github-actions bot pushed a commit that referenced this pull request Dec 16, 2025
@wyxxxcat
[fix](delete) Fix missing prepare_rowset for delete job (#59089)
b06d0ec 
load, compaction, schema change
use workflow with `prepare_rowset` -> `commit_rowset`
delete job needs to be consistent even though the `delete` can work
without prepare rowset
@github-actions github-actions bot mentioned this pull request Dec 16, 2025
Merged
github-actions bot pushed a commit that referenced this pull request Dec 16, 2025
@wyxxxcat
[fix](delete) Fix missing prepare_rowset for delete job (#59089)
bd00e56 
load, compaction, schema change
use workflow with `prepare_rowset` -> `commit_rowset`
delete job needs to be consistent even though the `delete` can work
without prepare rowset
@github-actions github-actions bot mentioned this pull request Dec 16, 2025
Merged
yiguolei pushed a commit that referenced this pull request Dec 17, 2025
@github-actions @wyxxxcat
branch-4.0: [fix](delete) Fix missing prepare_rowset for delete job #…
b20bb56 
…59089 (#59097)

Cherry-picked from #59089

Co-authored-by: Yixuan Wang <wangyixuan@selectdb.com>
morrySnow pushed a commit that referenced this pull request Dec 18, 2025
@github-actions @wyxxxcat
branch-3.1: [fix](delete) Fix missing prepare_rowset for delete job #…
d035d34 
…59089 (#59096)

Cherry-picked from #59089

Co-authored-by: Yixuan Wang <wangyixuan@selectdb.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gavinchou gavinchou gavinchou approved these changes

@liaoxin01 liaoxin01 liaoxin01 approved these changes

@dataroaring dataroaring Awaiting requested review from dataroaring dataroaring is a code owner

+1 more reviewer

@Hastyshell Hastyshell Hastyshell approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.1.4-merged dev/4.0.3-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

9 participants

@wyxxxcat @hello-stephen @doris-robot @gavinchou @liaoxin01 @Hastyshell @yiguolei @dataroaring @morrySnow