Skip to content

branch-3.1: [fix](sc) Skip empty rowset version hole filling for versions <= alter_version #56209 #56212

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morrySnow merged 1 commit into from
Sep 19, 2025
Merged

branch-3.1: [fix](sc) Skip empty rowset version hole filling for versions <= alter_version #56209 #56212

morrySnow merged 1 commit into from
Sep 19, 2025

Conversation

@liaoxin01
Copy link
Contributor

@liaoxin01 liaoxin01 commented Sep 19, 2025

cherry pick from #56209

@liaoxin01
[fix](sc) Skip empty rowset version hole filling for versions <= alte…
6b0692e 
…r_version (apache#56209)

**Schema change and version hole handling:**

* Improved the logic in `CloudMetaMgr::fill_version_holes` to
selectively skip filling version holes for schema change tablets
(`TABLET_NOTREADY` state) when the version is less than or equal to
`alter_version`. This prevents abnormal compaction scores and unexpected
errors during schema changes, allowing the process to proceed smoothly.
* In `CloudSchemaChangeJob::_process_delete_bitmap`, modified the
handling of rowsets during schema change jobs to retain only the [0-1]
version rowset and set the `alter_version` for the temporary tablet,
ensuring that hole rowsets are only filled for versions greater than
`alter_version`.
@liaoxin01
Copy link
Contributor Author

liaoxin01 commented Sep 19, 2025

run buildall

@hello-stephen
Copy link
Contributor

hello-stephen commented Sep 19, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@doris-robot
Copy link

doris-robot commented Sep 19, 2025

TPC-H: Total hot run time: 32691 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6b0692e675cf0b9ce26a37a4ceb6fc33e0346a46, data reload: false

------ Round 1 ----------------------------------
q1	17598	5460	5392	5392
q2	2033	404	281	281
q3	11655	1256	763	763
q4	10299	904	459	459
q5	9032	2399	2137	2137
q6	187	166	134	134
q7	908	751	610	610
q8	9331	1437	1169	1169
q9	5217	4985	4926	4926
q10	6769	2305	1835	1835
q11	477	278	268	268
q12	338	352	206	206
q13	17786	3591	2992	2992
q14	227	229	213	213
q15	527	473	456	456
q16	422	429	361	361
q17	611	877	381	381
q18	7126	6410	6355	6355
q19	1212	948	550	550
q20	322	345	207	207
q21	2971	2185	1995	1995
q22	1053	1019	1001	1001
Total cold run time: 106101 ms
Total hot run time: 32691 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5540	5492	5460	5460
q2	235	336	236	236
q3	2244	2626	2304	2304
q4	1379	1793	1380	1380
q5	4383	5003	5021	5003
q6	166	160	129	129
q7	2129	1926	1841	1841
q8	2643	2837	2752	2752
q9	7264	7216	7262	7216
q10	2996	3318	2672	2672
q11	573	514	488	488
q12	627	795	628	628
q13	3369	3796	3138	3138
q14	281	297	272	272
q15	513	471	484	471
q16	451	473	436	436
q17	1222	1716	1267	1267
q18	7639	7324	7309	7309
q19	796	1145	1085	1085
q20	2009	2051	1909	1909
q21	5316	4867	4617	4617
q22	1078	1049	996	996
Total cold run time: 52853 ms
Total hot run time: 51609 ms

@doris-robot
Copy link

doris-robot commented Sep 19, 2025

TPC-DS: Total hot run time: 191815 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6b0692e675cf0b9ce26a37a4ceb6fc33e0346a46, data reload: false

query1	947	411	418	411
query2	6248	1884	1865	1865
query3	8687	204	193	193
query4	33823	23936	23343	23343
query5	3663	589	475	475
query6	289	206	170	170
query7	4193	492	331	331
query8	305	240	229	229
query9	9204	2622	2595	2595
query10	476	331	256	256
query11	17880	15364	15229	15229
query12	163	108	105	105
query13	1562	551	425	425
query14	9026	7166	6579	6579
query15	241	196	176	176
query16	8031	626	523	523
query17	1543	779	609	609
query18	2141	440	351	351
query19	226	188	169	169
query20	127	119	122	119
query21	214	128	106	106
query22	4702	4709	4423	4423
query23	35033	34274	33938	33938
query24	7518	2676	2699	2676
query25	532	494	429	429
query26	818	286	179	179
query27	2472	469	374	374
query28	5439	2271	2244	2244
query29	737	606	450	450
query30	240	197	165	165
query31	1010	894	846	846
query32	96	60	58	58
query33	490	372	330	330
query34	743	875	524	524
query35	797	818	751	751
query36	1003	1056	958	958
query37	109	92	65	65
query38	4076	4049	3973	3973
query39	1537	1485	1473	1473
query40	202	130	107	107
query41	53	50	48	48
query42	128	112	102	102
query43	504	498	471	471
query44	1311	837	856	837
query45	184	178	174	174
query46	891	1068	681	681
query47	2021	2016	1944	1944
query48	411	434	346	346
query49	716	503	403	403
query50	696	703	440	440
query51	7332	7337	7331	7331
query52	105	103	91	91
query53	234	268	185	185
query54	554	553	502	502
query55	77	78	81	78
query56	276	281	253	253
query57	1271	1260	1232	1232
query58	230	221	219	219
query59	3000	3166	3025	3025
query60	294	282	261	261
query61	141	115	120	115
query62	790	764	707	707
query63	243	191	199	191
query64	3747	1016	650	650
query65	3411	3324	3277	3277
query66	840	411	315	315
query67	16099	15869	15615	15615
query68	7910	837	555	555
query69	488	309	267	267
query70	1197	1107	1088	1088
query71	378	289	266	266
query72	5758	3762	3838	3762
query73	642	750	357	357
query74	10522	9165	8980	8980
query75	3283	3140	2646	2646
query76	3145	1156	745	745
query77	579	372	281	281
query78	10449	10592	9610	9610
query79	3423	891	584	584
query80	783	515	427	427
query81	522	254	216	216
query82	830	115	91	91
query83	168	161	145	145
query84	235	108	82	82
query85	785	353	295	295
query86	403	307	303	303
query87	4283	4330	4227	4227
query88	5034	2446	2420	2420
query89	412	341	299	299
query90	1737	191	188	188
query91	137	138	110	110
query92	69	57	53	53
query93	2863	924	560	560
query94	782	413	295	295
query95	347	286	278	278
query96	506	614	288	288
query97	3165	3301	3185	3185
query98	246	222	197	197
query99	1331	1449	1300	1300
Total cold run time: 294514 ms
Total hot run time: 191815 ms

@doris-robot
Copy link

doris-robot commented Sep 19, 2025

ClickBench: Total hot run time: 28.94 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6b0692e675cf0b9ce26a37a4ceb6fc33e0346a46, data reload: false

query1	0.04	0.04	0.04
query2	0.07	0.03	0.03
query3	0.24	0.07	0.07
query4	1.62	0.11	0.10
query5	0.53	0.50	0.54
query6	1.14	0.74	0.73
query7	0.02	0.02	0.02
query8	0.04	0.04	0.03
query9	0.59	0.51	0.50
query10	0.55	0.55	0.56
query11	0.15	0.11	0.10
query12	0.14	0.12	0.11
query13	0.63	0.59	0.59
query14	0.77	0.79	0.80
query15	0.84	0.85	0.82
query16	0.38	0.39	0.39
query17	1.05	1.06	0.97
query18	0.24	0.23	0.23
query19	1.96	1.86	1.88
query20	0.02	0.02	0.01
query21	15.37	0.93	0.58
query22	0.74	0.73	0.65
query23	15.19	1.43	0.61
query24	3.30	1.77	0.89
query25	0.13	0.07	0.09
query26	0.34	0.14	0.13
query27	0.05	0.04	0.04
query28	13.61	0.94	0.43
query29	12.60	4.02	3.26
query30	0.26	0.10	0.06
query31	2.81	0.60	0.37
query32	3.23	0.52	0.46
query33	3.01	3.05	3.03
query34	16.71	5.19	4.54
query35	4.65	4.57	4.58
query36	0.63	0.49	0.49
query37	0.08	0.06	0.06
query38	0.04	0.03	0.04
query39	0.04	0.02	0.02
query40	0.17	0.14	0.13
query41	0.08	0.03	0.03
query42	0.04	0.04	0.02
query43	0.04	0.03	0.03
Total cold run time: 104.14 s
Total hot run time: 28.94 s

@doris-robot
Copy link

doris-robot commented Sep 19, 2025

BE UT Coverage Report

Increment line coverage 38.24% (13/34) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 45.57% (12784/28053)
Line Coverage 36.41% (114043/313247)
Region Coverage 34.02% (65194/191640)
Branch Coverage 31.05% (34215/110202)

@hello-stephen
Copy link
Contributor

hello-stephen commented Sep 19, 2025

BE Regression && UT Coverage Report

Increment line coverage 100.00% (34/34) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 76.49% (21104/27590)
Line Coverage 69.79% (217939/312281)
Region Coverage 67.72% (130312/192417)
Branch Coverage 61.25% (67809/110716)

@morrySnow morrySnow merged commit fefeaf0 into apache:branch-3.1 Sep 19, 2025
22 of 23 checks passed
@morrySnow morrySnow mentioned this pull request Sep 22, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@liaoxin01 @hello-stephen @doris-robot @morrySnow