[opt](olap) Optimize the performance of StructFileColumnIterator::read_by_rowids in scenarios where the rowids are continuous #58851

mrhhsg · 2025-12-09T08:23:41Z

What problem does this PR solve?

Avoid seeking and reading row by row.

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

Thearas · 2025-12-09T08:23:47Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

mrhhsg · 2025-12-09T08:23:51Z

run buildall

doris-robot · 2025-12-09T09:05:54Z

TPC-H: Total hot run time: 36336 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f6649923b25f0773b31f9bb8ed8cc20c4782d3d1, data reload: false

------ Round 1 ----------------------------------
q1	17640	5084	4924	4924
q2	2036	356	259	259
q3	10170	1351	756	756
q4	10213	777	345	345
q5	7487	2212	1899	1899
q6	182	171	135	135
q7	1033	850	704	704
q8	9361	1453	1147	1147
q9	7025	5372	5380	5372
q10	6845	2399	1980	1980
q11	547	312	305	305
q12	648	738	587	587
q13	17773	3678	3017	3017
q14	311	296	280	280
q15	600	528	524	524
q16	1022	943	872	872
q17	725	811	488	488
q18	7509	7154	7162	7154
q19	1101	962	589	589
q20	417	378	245	245
q21	4284	3912	3773	3773
q22	1066	989	981	981
Total cold run time: 107995 ms
Total hot run time: 36336 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4965	4945	4919	4919
q2	333	391	313	313
q3	2109	2682	2252	2252
q4	1332	1760	1272	1272
q5	4798	4782	4563	4563
q6	217	169	126	126
q7	2057	1949	1887	1887
q8	2673	2907	2588	2588
q9	7603	7511	7482	7482
q10	3103	3288	2839	2839
q11	599	517	504	504
q12	716	781	577	577
q13	3521	3916	3385	3385
q14	303	302	279	279
q15	558	521	509	509
q16	880	1088	899	899
q17	1222	1467	1309	1309
q18	8024	7647	7749	7647
q19	904	848	882	848
q20	2015	2141	1915	1915
q21	4614	4294	4204	4204
q22	1101	1032	975	975
Total cold run time: 53647 ms
Total hot run time: 51292 ms

doris-robot · 2025-12-09T09:17:21Z

TPC-DS: Total hot run time: 182417 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f6649923b25f0773b31f9bb8ed8cc20c4782d3d1, data reload: false

query5	4409	627	509	509
query6	344	248	234	234
query7	4646	486	285	285
query8	334	274	259	259
query9	8747	2626	2660	2626
query10	579	387	375	375
query11	15652	15188	15013	15013
query12	185	119	119	119
query13	1689	501	425	425
query14	6397	3315	3055	3055
query14_1	2993	2987	2944	2944
query15	214	198	183	183
query16	7516	484	446	446
query17	1274	721	609	609
query18	2015	437	359	359
query19	255	237	221	221
query20	130	119	123	119
query21	228	142	114	114
query22	3861	4109	3884	3884
query23	16524	16143	15910	15910
query23_1	16152	15966	16074	15966
query24	7229	1667	1276	1276
query24_1	1264	1253	1294	1253
query25	625	503	478	478
query26	1283	281	167	167
query27	2906	475	319	319
query28	4418	2192	2168	2168
query29	839	594	480	480
query30	316	248	226	226
query31	832	708	616	616
query32	87	75	75	75
query33	694	342	305	305
query34	874	897	603	603
query35	779	794	735	735
query36	899	941	850	850
query37	122	94	78	78
query38	3979	3847	3804	3804
query39	756	766	815	766
query39_1	696	717	697	697
query40	231	136	124	124
query41	68	65	62	62
query42	133	106	107	106
query43	466	434	401	401
query44	1295	746	752	746
query45	208	190	183	183
query46	942	983	641	641
query47	1684	1707	1645	1645
query48	433	327	249	249
query49	809	441	352	352
query50	690	293	232	232
query51	3860	3874	3936	3874
query52	129	109	102	102
query53	343	356	309	309
query54	336	278	255	255
query55	97	78	76	76
query56	364	310	300	300
query57	1179	1150	1090	1090
query58	292	253	247	247
query59	2320	2391	2284	2284
query60	345	312	284	284
query61	164	155	170	155
query62	781	679	646	646
query63	354	296	306	296
query64	4491	1322	1040	1040
query65	4030	4088	3919	3919
query66	1198	435	332	332
query67	15508	14978	14781	14781
query68	8140	1012	727	727
query69	557	344	312	312
query70	1125	1015	982	982
query71	475	315	299	299
query72	6098	4968	4918	4918
query73	681	561	314	314
query74	8889	8939	8519	8519
query75	3622	3531	3191	3191
query76	3546	1122	756	756
query77	786	412	304	304
query78	9583	9487	8905	8905
query79	1655	870	620	620
query80	748	673	565	565
query81	498	270	239	239
query82	458	133	101	101
query83	261	262	245	245
query84	263	125	104	104
query85	996	518	483	483
query86	359	310	277	277
query87	4008	4107	3985	3985
query88	4119	2297	2311	2297
query89	502	435	404	404
query90	2141	165	159	159
query91	188	180	157	157
query92	80	71	69	69
query93	1511	904	569	569
query94	820	302	287	287
query95	578	324	369	324
query96	592	470	206	206
query97	2682	2665	2605	2605
query98	254	200	212	200
query99	1425	1305	1188	1188
Total cold run time: 273229 ms
Total hot run time: 182417 ms

doris-robot · 2025-12-09T09:22:26Z

ClickBench: Total hot run time: 27.67 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f6649923b25f0773b31f9bb8ed8cc20c4782d3d1, data reload: false

query1	0.05	0.04	0.04
query2	0.10	0.05	0.04
query3	0.26	0.09	0.10
query4	1.62	0.12	0.12
query5	0.28	0.26	0.26
query6	1.16	0.65	0.63
query7	0.04	0.03	0.03
query8	0.06	0.05	0.05
query9	0.56	0.51	0.50
query10	0.56	0.57	0.55
query11	0.15	0.10	0.10
query12	0.14	0.11	0.12
query13	0.63	0.61	0.60
query14	0.99	1.00	0.99
query15	0.81	0.80	0.82
query16	0.40	0.40	0.42
query17	1.11	1.00	1.02
query18	0.23	0.21	0.22
query19	1.94	1.89	1.90
query20	0.02	0.02	0.01
query21	15.44	0.29	0.13
query22	4.61	0.05	0.05
query23	15.95	0.29	0.11
query24	1.47	0.48	0.46
query25	0.09	0.07	0.08
query26	0.15	0.13	0.13
query27	0.07	0.05	0.06
query28	4.69	1.23	1.03
query29	12.62	4.09	3.24
query30	0.28	0.14	0.13
query31	2.82	0.63	0.40
query32	3.23	0.56	0.48
query33	3.01	3.18	3.06
query34	17.06	5.19	4.52
query35	4.56	4.55	4.53
query36	0.68	0.51	0.50
query37	0.11	0.06	0.07
query38	0.07	0.05	0.04
query39	0.05	0.04	0.03
query40	0.18	0.14	0.14
query41	0.09	0.03	0.03
query42	0.05	0.03	0.03
query43	0.04	0.04	0.03
Total cold run time: 98.43 s
Total hot run time: 27.67 s

doris-robot · 2025-12-09T10:18:40Z

BE UT Coverage Report

Increment line coverage 21.62% (8/37) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	53.35% (18708/35065)
Line Coverage	39.05% (172955/442878)
Region Coverage	33.75% (134219/397743)
Branch Coverage	34.64% (57651/166428)

hello-stephen · 2025-12-09T12:06:57Z

BE Regression && UT Coverage Report

Increment line coverage 58.33% (21/36) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	72.22% (24819/34367)
Line Coverage	58.96% (260829/442400)
Region Coverage	53.97% (217220/402466)
Branch Coverage	55.41% (92707/167303)

…d_by_rowids in scenarios where the rowids are continuous

mrhhsg · 2025-12-10T01:40:52Z

run buildall

doris-robot · 2025-12-10T02:25:35Z

TPC-H: Total hot run time: 36370 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9b8949bbc750bb28831805d102b8d2d77c939dd0, data reload: false

------ Round 1 ----------------------------------
q1	17646	5143	4961	4961
q2	2026	365	248	248
q3	10195	1290	745	745
q4	10209	821	319	319
q5	7522	2183	1917	1917
q6	184	170	141	141
q7	997	862	726	726
q8	9342	1409	1120	1120
q9	7049	5328	5323	5323
q10	6842	2381	1965	1965
q11	537	328	302	302
q12	670	721	594	594
q13	17758	3656	3031	3031
q14	291	297	271	271
q15	615	528	523	523
q16	923	902	866	866
q17	714	822	521	521
q18	7653	7084	7168	7084
q19	1266	966	636	636
q20	416	377	251	251
q21	4281	4038	3873	3873
q22	1039	985	953	953
Total cold run time: 108175 ms
Total hot run time: 36370 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4951	4938	4933	4933
q2	333	396	311	311
q3	2126	2680	2265	2265
q4	1329	1767	1299	1299
q5	4751	4801	4591	4591
q6	231	170	132	132
q7	2031	1988	1746	1746
q8	2692	2528	2563	2528
q9	7477	7536	7497	7497
q10	3204	3235	3108	3108
q11	661	501	498	498
q12	662	744	585	585
q13	3466	3966	3440	3440
q14	299	313	280	280
q15	562	523	507	507
q16	851	934	900	900
q17	1246	1398	1358	1358
q18	8160	7705	7486	7486
q19	921	868	859	859
q20	2077	2032	1842	1842
q21	4496	4227	4060	4060
q22	1070	1002	988	988
Total cold run time: 53596 ms
Total hot run time: 51213 ms

doris-robot · 2025-12-10T02:36:40Z

TPC-DS: Total hot run time: 180629 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9b8949bbc750bb28831805d102b8d2d77c939dd0, data reload: false

query5	4817	625	461	461
query6	338	236	213	213
query7	4222	463	270	270
query8	315	248	237	237
query9	8784	2590	2612	2590
query10	533	365	340	340
query11	14940	15123	14536	14536
query12	173	122	119	119
query13	1275	518	405	405
query14	5528	3263	3050	3050
query14_1	2909	2899	2908	2899
query15	205	199	183	183
query16	876	476	465	465
query17	1134	704	602	602
query18	2518	451	354	354
query19	231	240	223	223
query20	122	115	114	114
query21	216	147	119	119
query22	3962	4073	3849	3849
query23	17231	16217	15952	15952
query23_1	15909	16036	15928	15928
query24	7308	1628	1205	1205
query24_1	1220	1188	1258	1188
query25	541	465	405	405
query26	847	271	179	179
query27	2719	462	305	305
query28	4470	2113	2108	2108
query29	742	545	438	438
query30	308	247	217	217
query31	803	690	632	632
query32	76	68	76	68
query33	533	327	293	293
query34	894	885	544	544
query35	798	811	724	724
query36	868	909	809	809
query37	130	93	73	73
query38	3761	3824	3735	3735
query39	743	751	710	710
query39_1	695	710	698	698
query40	231	135	118	118
query41	67	61	62	61
query42	105	109	110	109
query43	420	427	399	399
query44	1324	741	745	741
query45	193	195	179	179
query46	873	970	605	605
query47	1651	1682	1624	1624
query48	314	327	238	238
query49	655	434	374	374
query50	650	299	226	226
query51	3904	3853	3837	3837
query52	105	106	98	98
query53	320	353	302	302
query54	279	266	248	248
query55	77	75	71	71
query56	300	296	290	290
query57	1138	1139	1063	1063
query58	263	253	250	250
query59	2310	2403	2319	2319
query60	308	312	285	285
query61	159	151	149	149
query62	683	681	621	621
query63	334	293	297	293
query64	4392	1258	970	970
query65	3982	4002	3972	3972
query66	1415	447	311	311
query67	15087	15182	14698	14698
query68	3585	1053	734	734
query69	495	345	307	307
query70	1088	964	989	964
query71	346	308	288	288
query72	6149	5028	5187	5028
query73	748	685	318	318
query74	8817	8759	8617	8617
query75	3532	3510	3136	3136
query76	3740	1148	752	752
query77	522	397	298	298
query78	9386	9601	8810	8810
query79	1042	822	619	619
query80	842	629	562	562
query81	514	264	240	240
query82	201	131	107	107
query83	272	257	237	237
query84	255	121	94	94
query85	898	503	455	455
query86	325	282	293	282
query87	3969	4015	4036	4015
query88	3085	2269	2261	2261
query89	452	418	386	386
query90	1878	163	154	154
query91	194	183	157	157
query92	79	71	66	66
query93	968	894	556	556
query94	412	317	305	305
query95	590	347	329	329
query96	596	447	216	216
query97	2648	2646	2571	2571
query98	210	208	191	191
query99	1276	1311	1243	1243
Total cold run time: 254167 ms
Total hot run time: 180629 ms

doris-robot · 2025-12-10T02:41:43Z

ClickBench: Total hot run time: 27.44 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9b8949bbc750bb28831805d102b8d2d77c939dd0, data reload: false

query1	0.06	0.05	0.04
query2	0.10	0.05	0.05
query3	0.25	0.08	0.09
query4	1.60	0.12	0.11
query5	0.28	0.25	0.26
query6	1.18	0.63	0.63
query7	0.04	0.03	0.02
query8	0.05	0.04	0.04
query9	0.57	0.52	0.50
query10	0.55	0.56	0.55
query11	0.16	0.12	0.11
query12	0.16	0.12	0.12
query13	0.62	0.60	0.60
query14	1.00	0.98	0.99
query15	0.82	0.81	0.81
query16	0.41	0.42	0.38
query17	1.05	1.06	1.05
query18	0.23	0.22	0.21
query19	1.85	1.86	1.87
query20	0.02	0.01	0.02
query21	15.43	0.30	0.14
query22	4.64	0.05	0.05
query23	16.10	0.29	0.10
query24	1.26	0.30	0.42
query25	0.12	0.09	0.06
query26	0.14	0.13	0.14
query27	0.09	0.06	0.04
query28	4.37	1.22	1.03
query29	12.59	4.02	3.20
query30	0.28	0.14	0.12
query31	2.82	0.64	0.41
query32	3.24	0.56	0.47
query33	3.04	3.01	3.13
query34	16.68	5.12	4.58
query35	4.57	4.56	4.57
query36	0.67	0.49	0.52
query37	0.10	0.06	0.07
query38	0.07	0.04	0.04
query39	0.05	0.03	0.04
query40	0.18	0.14	0.13
query41	0.09	0.04	0.03
query42	0.05	0.03	0.03
query43	0.05	0.03	0.03
Total cold run time: 97.63 s
Total hot run time: 27.44 s

doris-robot · 2025-12-10T03:05:09Z

BE UT Coverage Report

Increment line coverage 21.62% (8/37) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	53.40% (18728/35070)
Line Coverage	39.10% (173174/442941)
Region Coverage	33.79% (134415/397765)
Branch Coverage	34.69% (57734/166440)

hello-stephen · 2025-12-10T04:56:06Z

BE Regression && UT Coverage Report

Increment line coverage 89.19% (33/37) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	72.26% (24837/34372)
Line Coverage	58.99% (261002/442465)
Region Coverage	53.94% (217084/402488)
Branch Coverage	55.45% (92768/167315)

be/src/olap/rowset/segment_v2/column_reader.cpp

hello-stephen · 2025-12-10T08:09:24Z

BE Regression && UT Coverage Report

Increment line coverage 89.19% (33/37) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	73.15% (25142/34372)
Line Coverage	60.27% (266659/442465)
Region Coverage	55.84% (224731/402488)
Branch Coverage	57.12% (95563/167315)

hello-stephen · 2025-12-10T10:03:39Z

BE Regression && UT Coverage Report

Increment line coverage 89.19% (33/37) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	72.26% (24836/34372)
Line Coverage	58.99% (261002/442465)
Region Coverage	53.92% (217040/402488)
Branch Coverage	55.45% (92771/167315)

eldenmoon

LGTM

github-actions · 2025-12-12T02:08:41Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-12-12T02:08:43Z

PR approved by anyone and no changes requested.

…d_by_rowids in scenarios where the rowids are continuous (apache#58851) ### What problem does this PR solve? Avoid seeking and reading row by row. Issue Number: close #xxx Related PR: #xxx Problem Summary: ### Release note None ### Check List (For Author) - Test  - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason  - Behavior changed: - [ ] No. - [ ] Yes.  - Does this need documentation? - [ ] No. - [ ] Yes.  ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label

…d_by_rowids in scenarios where the rowids are continuous (#58851) ### What problem does this PR solve? Avoid seeking and reading row by row. Issue Number: close #xxx Related PR: #xxx Problem Summary: ### Release note None ### Check List (For Author) - Test  - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason  - Behavior changed: - [ ] No. - [ ] Yes.  - Does this need documentation? - [ ] No. - [ ] Yes.  ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label

…rning (#59286) ### What problem does this PR solve? Problem Summary: ### Release note Cherry-pick #58370 #58354 #59043 #58851 #58485 #58682 #58614 #58373 #57204 #58719 #58471 #58573 #58657 ### Check List (For Author) - Test  - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason  - Behavior changed: - [ ] No. - [ ] Yes.  - Does this need documentation? - [ ] No. - [ ] Yes.  ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label  --------- Co-authored-by: 924060929 <lanhuajian@selectdb.com> Co-authored-by: Jerry Hu <mrhhsg@gmail.com> Co-authored-by: Jerry Hu <hushenggang@selectdb.com> Co-authored-by: lihangyu <lihangyu@selectdb.com>

[opt](olap) Optimize the performance of StructFileColumnIterator::rea…

9b8949b

…d_by_rowids in scenarios where the rowids are continuous

mrhhsg force-pushed the opt_read branch from f664992 to 9b8949b Compare December 10, 2025 01:40

eldenmoon reviewed Dec 10, 2025

View reviewed changes

be/src/olap/rowset/segment_v2/column_reader.cpp Show resolved Hide resolved

eldenmoon approved these changes Dec 12, 2025

View reviewed changes

zhangstar333 approved these changes Dec 12, 2025

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 12, 2025

github-actions bot added the reviewed label Dec 12, 2025

mrhhsg merged commit df86f27 into apache:master Dec 12, 2025
28 of 32 checks passed

mrhhsg deleted the opt_read branch December 12, 2025 02:09

kaka11chen mentioned this pull request Dec 23, 2025

[feature](reader) Optimize Complex Type Column Reading with Column Purning #59286

Merged

16 tasks

yiguolei added the dev/4.0.3-merged label Dec 24, 2025

[opt](olap) Optimize the performance of StructFileColumnIterator::read_by_rowids in scenarios where the rowids are continuous #58851

[opt](olap) Optimize the performance of StructFileColumnIterator::read_by_rowids in scenarios where the rowids are continuous #58851

Uh oh!

Conversation

mrhhsg commented Dec 9, 2025

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

Thearas commented Dec 9, 2025

Uh oh!

mrhhsg commented Dec 9, 2025

Uh oh!

doris-robot commented Dec 9, 2025

Uh oh!

doris-robot commented Dec 9, 2025

Uh oh!

doris-robot commented Dec 9, 2025

Uh oh!

doris-robot commented Dec 9, 2025

BE UT Coverage Report

Uh oh!

hello-stephen commented Dec 9, 2025

BE Regression && UT Coverage Report

Uh oh!

mrhhsg commented Dec 10, 2025

Uh oh!

doris-robot commented Dec 10, 2025

Uh oh!

doris-robot commented Dec 10, 2025

Uh oh!

doris-robot commented Dec 10, 2025

Uh oh!

doris-robot commented Dec 10, 2025

BE UT Coverage Report

Uh oh!

hello-stephen commented Dec 10, 2025

BE Regression && UT Coverage Report

Uh oh!

Uh oh!

hello-stephen commented Dec 10, 2025

BE Regression && UT Coverage Report

Uh oh!

hello-stephen commented Dec 10, 2025

BE Regression && UT Coverage Report

Uh oh!

eldenmoon left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Dec 12, 2025

Uh oh!

github-actions bot commented Dec 12, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants