Skip to content

Conversation

@csun5285
Copy link
Contributor

@csun5285 csun5285 commented Sep 9, 2025

What problem does this PR solve?

In master #55127, the compaction process was made compatible with both the new and old variants, while in 3.1 a defensive strategy was used to prevent errors from occurring.

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@csun5285 csun5285 requested a review from morrySnow as a code owner September 9, 2025 09:53
@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@csun5285
Copy link
Contributor Author

csun5285 commented Sep 9, 2025

run buildall

@csun5285 csun5285 changed the title [fix](variant) Compatibility error when the sparse column is empty branch-3.1: [fix](variant) Compatibility error when the sparse column is empty Sep 9, 2025
@doris-robot
Copy link

TPC-H: Total hot run time: 33902 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 63cda5c11a3168c596f570d23a2e07b71aebe61c, data reload: false

------ Round 1 ----------------------------------
q1	17665	5772	5649	5649
q2	2014	404	289	289
q3	12254	1323	791	791
q4	10556	909	467	467
q5	9722	2521	2261	2261
q6	209	176	138	138
q7	992	751	613	613
q8	9345	1629	1476	1476
q9	5469	5133	5148	5133
q10	6813	2337	1826	1826
q11	488	291	285	285
q12	350	394	217	217
q13	17770	3697	3036	3036
q14	241	227	214	214
q15	561	472	468	468
q16	442	443	386	386
q17	633	918	387	387
q18	7101	6453	6559	6453
q19	1209	1096	596	596
q20	349	349	202	202
q21	3088	2373	2063	2063
q22	1074	1032	952	952
Total cold run time: 108345 ms
Total hot run time: 33902 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5955	5934	5916	5916
q2	244	345	233	233
q3	2305	2735	2346	2346
q4	1464	1829	1385	1385
q5	4749	5200	5172	5172
q6	193	176	129	129
q7	2135	1985	1805	1805
q8	2810	2977	2959	2959
q9	7356	7247	7146	7146
q10	3063	3331	2859	2859
q11	628	523	497	497
q12	682	788	586	586
q13	3413	3899	3183	3183
q14	287	292	275	275
q15	540	472	466	466
q16	452	503	440	440
q17	1309	1896	1290	1290
q18	7630	7556	7260	7260
q19	849	1239	1294	1239
q20	2068	2083	1909	1909
q21	5545	5082	4661	4661
q22	1148	1124	1032	1032
Total cold run time: 54825 ms
Total hot run time: 52788 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192523 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 63cda5c11a3168c596f570d23a2e07b71aebe61c, data reload: false

query1	950	416	418	416
query2	6068	1941	1873	1873
query3	8710	201	194	194
query4	33919	24308	23510	23510
query5	3509	599	452	452
query6	292	211	167	167
query7	4215	501	327	327
query8	301	245	227	227
query9	9587	2613	2627	2613
query10	477	337	264	264
query11	18500	15487	15261	15261
query12	158	108	111	108
query13	1552	544	417	417
query14	9418	6691	7259	6691
query15	248	200	177	177
query16	8072	645	513	513
query17	1520	787	597	597
query18	2162	427	329	329
query19	226	188	171	171
query20	129	129	127	127
query21	209	131	111	111
query22	4636	4685	4423	4423
query23	35476	34526	34284	34284
query24	7753	2691	2733	2691
query25	553	492	443	443
query26	1173	293	173	173
query27	2084	487	364	364
query28	5425	2223	2188	2188
query29	795	624	484	484
query30	241	193	161	161
query31	1021	929	849	849
query32	93	62	61	61
query33	511	413	305	305
query34	750	858	525	525
query35	798	800	737	737
query36	1019	1066	927	927
query37	101	99	70	70
query38	4098	4118	3965	3965
query39	1532	1505	1489	1489
query40	211	116	111	111
query41	49	50	48	48
query42	129	111	112	111
query43	523	521	490	490
query44	1348	811	837	811
query45	183	181	173	173
query46	902	1065	687	687
query47	2013	2022	1946	1946
query48	419	439	338	338
query49	749	522	416	416
query50	686	689	453	453
query51	7303	7343	7290	7290
query52	106	103	97	97
query53	234	259	195	195
query54	553	540	464	464
query55	79	81	76	76
query56	266	279	248	248
query57	1276	1271	1213	1213
query58	240	217	225	217
query59	3075	3234	3075	3075
query60	304	293	267	267
query61	115	115	115	115
query62	792	778	711	711
query63	237	198	199	198
query64	4496	1013	648	648
query65	3420	3275	3305	3275
query66	1043	415	315	315
query67	16273	15898	15573	15573
query68	8050	823	540	540
query69	477	304	259	259
query70	1215	1158	1107	1107
query71	374	298	265	265
query72	5714	3758	3799	3758
query73	634	738	362	362
query74	10536	9289	9028	9028
query75	3292	3196	2751	2751
query76	3141	1134	778	778
query77	547	372	277	277
query78	10446	10457	9593	9593
query79	3639	909	600	600
query80	721	532	438	438
query81	531	262	220	220
query82	600	119	88	88
query83	179	158	143	143
query84	294	97	75	75
query85	794	377	292	292
query86	397	301	298	298
query87	4357	4350	4251	4251
query88	5248	2436	2423	2423
query89	408	338	294	294
query90	1797	193	189	189
query91	139	145	109	109
query92	72	57	50	50
query93	2180	908	544	544
query94	668	418	321	321
query95	342	278	273	273
query96	487	612	281	281
query97	3216	3266	3169	3169
query98	227	207	207	207
query99	1569	1402	1295	1295
Total cold run time: 297596 ms
Total hot run time: 192523 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 28.51 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 63cda5c11a3168c596f570d23a2e07b71aebe61c, data reload: false

query1	0.04	0.03	0.02
query2	0.07	0.03	0.03
query3	0.24	0.07	0.06
query4	1.62	0.11	0.11
query5	0.54	0.51	0.52
query6	1.13	0.72	0.73
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.57	0.50	0.49
query10	0.56	0.55	0.56
query11	0.15	0.11	0.11
query12	0.14	0.11	0.11
query13	0.62	0.59	0.59
query14	0.77	0.80	0.80
query15	0.85	0.82	0.82
query16	0.38	0.41	0.37
query17	0.99	1.04	1.05
query18	0.25	0.23	0.23
query19	1.97	1.86	1.79
query20	0.02	0.01	0.01
query21	15.39	0.93	0.58
query22	0.74	0.91	0.62
query23	15.04	1.45	0.59
query24	3.24	1.27	0.41
query25	0.20	0.15	0.18
query26	0.28	0.15	0.13
query27	0.07	0.05	0.04
query28	13.15	1.08	0.43
query29	12.59	3.99	3.27
query30	0.26	0.10	0.08
query31	2.82	0.59	0.38
query32	3.23	0.54	0.47
query33	2.98	3.04	3.04
query34	16.78	5.13	4.55
query35	4.55	4.60	4.56
query36	0.64	0.50	0.49
query37	0.09	0.06	0.06
query38	0.05	0.04	0.03
query39	0.04	0.03	0.02
query40	0.16	0.14	0.13
query41	0.07	0.02	0.03
query42	0.03	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 103.41 s
Total hot run time: 28.51 s

@hello-stephen
Copy link
Contributor

BE UT Coverage Report

Increment line coverage 100.00% (5/5) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 45.55% (12758/28010)
Line Coverage 36.38% (113745/312695)
Region Coverage 34.01% (65067/191314)
Branch Coverage 31.04% (34150/110014)

Copy link
Member

@eldenmoon eldenmoon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@eldenmoon
Copy link
Member

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32557 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 63cda5c11a3168c596f570d23a2e07b71aebe61c, data reload: false

------ Round 1 ----------------------------------
q1	17593	5490	5433	5433
q2	2017	387	280	280
q3	12115	1216	764	764
q4	10281	862	461	461
q5	8885	2334	2095	2095
q6	182	164	132	132
q7	892	737	633	633
q8	9345	1436	1130	1130
q9	5098	4920	4926	4920
q10	6767	2247	1825	1825
q11	473	284	267	267
q12	351	363	212	212
q13	17765	3586	3021	3021
q14	231	232	212	212
q15	527	467	467	467
q16	421	431	374	374
q17	623	859	373	373
q18	6904	6265	6348	6265
q19	1215	951	537	537
q20	336	327	207	207
q21	3015	2154	1987	1987
q22	1061	1028	962	962
Total cold run time: 106097 ms
Total hot run time: 32557 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5533	5465	5485	5465
q2	236	324	227	227
q3	2264	2643	2322	2322
q4	1374	1745	1431	1431
q5	4460	4904	4963	4904
q6	180	164	132	132
q7	2054	1969	1849	1849
q8	2658	2913	2742	2742
q9	7352	7425	7342	7342
q10	3046	3354	2755	2755
q11	576	516	510	510
q12	661	794	644	644
q13	3535	3930	3254	3254
q14	282	299	303	299
q15	542	463	474	463
q16	454	506	447	447
q17	1218	1734	1261	1261
q18	7591	7449	7226	7226
q19	821	1146	1131	1131
q20	2042	2056	1914	1914
q21	5379	4823	4622	4622
q22	1101	1059	990	990
Total cold run time: 53359 ms
Total hot run time: 51930 ms

@doris-robot
Copy link

BE UT Coverage Report

Increment line coverage 100.00% (5/5) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 45.55% (12758/28010)
Line Coverage 36.38% (113770/312695)
Region Coverage 34.01% (65059/191314)
Branch Coverage 31.04% (34143/110014)

@doris-robot
Copy link

TPC-DS: Total hot run time: 193017 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 63cda5c11a3168c596f570d23a2e07b71aebe61c, data reload: false

query1	953	403	401	401
query2	6279	1930	1856	1856
query3	8692	199	197	197
query4	33468	23753	23849	23753
query5	3645	597	460	460
query6	282	191	178	178
query7	4211	497	336	336
query8	323	245	248	245
query9	9499	2624	2596	2596
query10	485	330	264	264
query11	18552	15486	15287	15287
query12	157	110	107	107
query13	1588	554	426	426
query14	10242	7057	7247	7057
query15	230	192	189	189
query16	8016	658	504	504
query17	1608	796	601	601
query18	2194	433	327	327
query19	250	187	167	167
query20	131	126	124	124
query21	210	129	107	107
query22	4698	4633	4471	4471
query23	35007	34861	34167	34167
query24	7383	2713	2752	2713
query25	562	491	423	423
query26	1195	298	175	175
query27	2113	488	367	367
query28	5248	2230	2175	2175
query29	791	611	464	464
query30	236	198	168	168
query31	990	937	834	834
query32	98	61	55	55
query33	498	378	311	311
query34	759	871	526	526
query35	776	822	745	745
query36	1029	1051	935	935
query37	97	91	64	64
query38	4107	4082	4003	4003
query39	1520	1638	1472	1472
query40	214	116	113	113
query41	51	48	47	47
query42	125	114	107	107
query43	527	533	503	503
query44	1332	839	835	835
query45	193	178	173	173
query46	899	1075	682	682
query47	1980	1982	1902	1902
query48	415	448	343	343
query49	807	492	413	413
query50	673	678	424	424
query51	7420	7249	7251	7249
query52	105	99	91	91
query53	239	259	193	193
query54	551	565	482	482
query55	81	83	86	83
query56	286	288	256	256
query57	1290	1295	1207	1207
query58	244	222	217	217
query59	3099	3260	2959	2959
query60	304	283	275	275
query61	116	114	109	109
query62	772	756	731	731
query63	225	200	197	197
query64	4482	1001	636	636
query65	3379	3360	3351	3351
query66	1028	414	309	309
query67	16474	15766	15557	15557
query68	8014	818	554	554
query69	479	307	265	265
query70	1189	1163	1046	1046
query71	366	297	273	273
query72	5731	3777	3818	3777
query73	626	749	355	355
query74	10487	9192	9211	9192
query75	3189	3190	2664	2664
query76	2978	1183	781	781
query77	509	368	277	277
query78	10399	10399	9572	9572
query79	3557	899	588	588
query80	634	518	437	437
query81	513	255	212	212
query82	581	121	91	91
query83	190	162	144	144
query84	282	97	85	85
query85	796	353	297	297
query86	396	280	300	280
query87	4309	4283	4278	4278
query88	5002	2426	2401	2401
query89	415	331	294	294
query90	1763	192	194	192
query91	141	146	121	121
query92	64	59	54	54
query93	2131	902	549	549
query94	693	407	317	317
query95	359	284	273	273
query96	497	608	294	294
query97	3212	3312	3149	3149
query98	232	205	203	203
query99	1555	1438	1308	1308
Total cold run time: 296722 ms
Total hot run time: 193017 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 28.36 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 63cda5c11a3168c596f570d23a2e07b71aebe61c, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.03	0.03
query3	0.23	0.07	0.07
query4	1.62	0.11	0.10
query5	0.51	0.51	0.53
query6	1.13	0.73	0.72
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.57	0.50	0.50
query10	0.55	0.54	0.55
query11	0.15	0.11	0.11
query12	0.14	0.10	0.11
query13	0.62	0.60	0.61
query14	0.77	0.81	0.79
query15	0.84	0.83	0.82
query16	0.39	0.39	0.38
query17	1.08	1.05	1.05
query18	0.25	0.23	0.24
query19	1.98	1.89	1.77
query20	0.02	0.01	0.02
query21	15.40	0.98	0.59
query22	0.75	0.74	0.66
query23	15.14	1.39	0.57
query24	3.02	1.09	0.30
query25	0.14	0.16	0.13
query26	0.40	0.15	0.13
query27	0.04	0.06	0.05
query28	12.90	1.04	0.44
query29	12.59	3.90	3.27
query30	0.25	0.10	0.06
query31	2.81	0.60	0.39
query32	3.22	0.53	0.46
query33	3.02	3.05	3.04
query34	16.53	5.23	4.53
query35	4.58	4.55	4.55
query36	0.61	0.50	0.48
query37	0.09	0.06	0.07
query38	0.05	0.03	0.04
query39	0.03	0.02	0.02
query40	0.16	0.14	0.13
query41	0.08	0.03	0.02
query42	0.04	0.03	0.02
query43	0.03	0.04	0.03
Total cold run time: 102.9 s
Total hot run time: 28.36 s

@hello-stephen
Copy link
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100.00% (5/5) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 66.17% (18228/27549)
Line Coverage 57.75% (180029/311721)
Region Coverage 55.48% (106569/192091)
Branch Coverage 49.80% (55046/110528)

@hello-stephen
Copy link
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100.00% (3/3) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 66.17% (18228/27549)
Line Coverage 57.75% (180029/311721)
Region Coverage 55.48% (106569/192091)
Branch Coverage 49.80% (55046/110528)

@morrySnow morrySnow merged commit 37088d3 into apache:branch-3.1 Sep 11, 2025
25 of 26 checks passed
@morrySnow morrySnow mentioned this pull request Sep 22, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants