Skip to content

[fix](olap) The tablet schema cache caused incorrect behavior in the complex-type column pruning functionality #58373

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
mrhhsg merged 1 commit into from
Nov 28, 2025
Merged

[fix](olap) The tablet schema cache caused incorrect behavior in the complex-type column pruning functionality #58373

mrhhsg merged 1 commit into from
Nov 28, 2025

Conversation

@mrhhsg
Copy link
Member

@mrhhsg mrhhsg commented Nov 26, 2025 •
edited
Loading

What problem does this PR solve?

TabletSchema with pruned column type should not be cached.

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

hello-stephen commented Nov 26, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@mrhhsg mrhhsg changed the title [fix](olap) The tablet schema cache caused incorrect behavior in the … [fix](olap) The tablet schema cache caused incorrect behavior in the complex-type column pruning functionality Nov 26, 2025
@mrhhsg
Copy link
Member Author

mrhhsg commented Nov 26, 2025

run buildall

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 27, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Nov 27, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Nov 27, 2025

PR approved by anyone and no changes requested.

@mrhhsg
Copy link
Member Author

mrhhsg commented Nov 27, 2025

run buildall

@hello-stephen
Copy link
Contributor

hello-stephen commented Nov 27, 2025

BE UT Coverage Report

Increment line coverage 0.00% (0/21) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 52.91% (18503/34974)
Line Coverage 38.46% (169539/440855)
Region Coverage 33.26% (131645/395842)
Branch Coverage 34.21% (56678/165687)

@hello-stephen
Copy link
Contributor

hello-stephen commented Nov 27, 2025

BE Regression && UT Coverage Report

Increment line coverage 100.00% (21/21) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 71.54% (24534/34293)
Line Coverage 58.04% (255658/440470)
Region Coverage 53.10% (212768/400656)
Branch Coverage 54.59% (90942/166584)

@hello-stephen
Copy link
Contributor

hello-stephen commented Nov 27, 2025

BE Regression && UT Coverage Report

Increment line coverage 100.00% (21/21) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 71.56% (24539/34293)
Line Coverage 58.08% (255833/440470)
Region Coverage 53.14% (212902/400656)
Branch Coverage 54.64% (91023/166584)

@hello-stephen
Copy link
Contributor

hello-stephen commented Nov 27, 2025

BE Regression && UT Coverage Report

Increment line coverage 100.00% (21/21) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 71.57% (24543/34293)
Line Coverage 58.09% (255868/440470)
Region Coverage 53.14% (212917/400656)
Branch Coverage 54.66% (91056/166584)

@doris-robot
Copy link

doris-robot commented Nov 27, 2025

TPC-H: Total hot run time: 34066 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0d019562aad5685a6a1efca4e707f7579e9719cd, data reload: false

------ Round 1 ----------------------------------
q1	17694	5015	4866	4866
q2	2046	316	202	202
q3	10251	1292	720	720
q4	10247	931	374	374
q5	7533	2477	2190	2190
q6	190	169	136	136
q7	947	760	624	624
q8	9330	1298	982	982
q9	7017	5291	5331	5291
q10	6845	2222	1841	1841
q11	530	307	310	307
q12	343	373	232	232
q13	18061	3718	3024	3024
q14	230	238	231	231
q15	586	547	510	510
q16	1011	1019	963	963
q17	577	797	480	480
q18	7412	6966	7126	6966
q19	1141	942	563	563
q20	318	336	224	224
q21	2929	3131	2402	2402
q22	1039	1009	938	938
Total cold run time: 106277 ms
Total hot run time: 34066 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4897	4877	4859	4859
q2	318	413	318	318
q3	2105	2682	2270	2270
q4	1278	1721	1304	1304
q5	4188	4212	4455	4212
q6	216	172	138	138
q7	2093	1947	1720	1720
q8	2694	2505	2482	2482
q9	7454	7482	7506	7482
q10	3042	3300	2831	2831
q11	598	534	514	514
q12	668	729	615	615
q13	3682	3825	3281	3281
q14	291	315	314	314
q15	539	512	515	512
q16	1085	1121	1096	1096
q17	1116	1363	1400	1363
q18	8005	7904	7625	7625
q19	780	923	773	773
q20	1936	2063	1837	1837
q21	4739	4264	4137	4137
q22	1095	1047	993	993
Total cold run time: 52819 ms
Total hot run time: 50676 ms

@doris-robot
Copy link

doris-robot commented Nov 27, 2025

TPC-DS: Total hot run time: 184940 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0d019562aad5685a6a1efca4e707f7579e9719cd, data reload: false

query1	1053	405	385	385
query2	6564	1586	1576	1576
query3	6753	231	241	231
query4	25605	23230	22954	22954
query5	4856	630	479	479
query6	340	249	217	217
query7	4643	501	307	307
query8	322	277	254	254
query9	8741	2584	2594	2584
query10	537	350	309	309
query11	15487	14743	14852	14743
query12	178	122	114	114
query13	1696	574	463	463
query14	10900	8707	8722	8707
query15	220	202	185	185
query16	7553	688	531	531
query17	1223	786	645	645
query18	2044	441	352	352
query19	218	200	197	197
query20	138	132	128	128
query21	220	144	116	116
query22	4083	4064	3933	3933
query23	33082	32423	32379	32379
query24	8472	2366	2366	2366
query25	624	529	451	451
query26	1232	274	166	166
query27	2746	489	349	349
query28	4348	2131	2113	2113
query29	770	639	532	532
query30	308	249	209	209
query31	813	724	601	601
query32	82	69	72	69
query33	601	385	336	336
query34	823	855	548	548
query35	828	828	744	744
query36	880	937	827	827
query37	124	111	88	88
query38	3311	3360	3278	3278
query39	1489	1401	1403	1401
query40	226	129	121	121
query41	66	64	64	64
query42	129	108	114	108
query43	443	438	417	417
query44	1289	750	759	750
query45	199	192	182	182
query46	868	992	640	640
query47	1692	1745	1684	1684
query48	386	436	318	318
query49	794	484	416	416
query50	657	682	398	398
query51	3859	3938	3817	3817
query52	114	108	100	100
query53	239	254	204	204
query54	327	294	275	275
query55	94	94	99	94
query56	373	335	327	327
query57	1133	1157	1087	1087
query58	293	275	273	273
query59	2280	2422	2317	2317
query60	349	358	342	342
query61	165	159	175	159
query62	786	715	639	639
query63	231	196	194	194
query64	4379	1211	883	883
query65	4077	3958	3996	3958
query66	1124	440	334	334
query67	15061	15019	15104	15019
query68	5790	953	616	616
query69	520	345	300	300
query70	1264	1179	1146	1146
query71	417	332	296	296
query72	6023	4872	4906	4872
query73	660	597	342	342
query74	8924	8930	8668	8668
query75	3316	3373	2821	2821
query76	3320	1128	709	709
query77	525	420	313	313
query78	9598	9867	8908	8908
query79	961	889	579	579
query80	1275	598	498	498
query81	542	271	232	232
query82	418	159	135	135
query83	373	267	250	250
query84	252	119	95	95
query85	1044	502	442	442
query86	391	286	294	286
query87	3453	3500	3409	3409
query88	2859	2286	2274	2274
query89	397	338	299	299
query90	1693	222	216	216
query91	169	182	144	144
query92	74	73	63	63
query93	1075	990	652	652
query94	734	441	328	328
query95	498	408	397	397
query96	520	523	284	284
query97	2912	2975	2874	2874
query98	242	222	210	210
query99	1307	1394	1295	1295
Total cold run time: 267086 ms
Total hot run time: 184940 ms

@doris-robot
Copy link

doris-robot commented Nov 27, 2025

ClickBench: Total hot run time: 27.35 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0d019562aad5685a6a1efca4e707f7579e9719cd, data reload: false

query1	0.05	0.04	0.04
query2	0.09	0.04	0.05
query3	0.26	0.09	0.08
query4	1.60	0.11	0.11
query5	0.27	0.26	0.26
query6	1.17	0.65	0.63
query7	0.03	0.03	0.03
query8	0.05	0.04	0.04
query9	0.57	0.52	0.52
query10	0.56	0.55	0.55
query11	0.15	0.10	0.12
query12	0.15	0.11	0.12
query13	0.62	0.59	0.60
query14	0.98	1.00	0.99
query15	0.81	0.79	0.80
query16	0.40	0.40	0.41
query17	0.98	1.08	1.08
query18	0.23	0.21	0.22
query19	1.87	1.76	1.82
query20	0.01	0.02	0.01
query21	15.47	0.25	0.13
query22	4.80	0.04	0.04
query23	16.03	0.26	0.10
query24	2.02	0.44	0.38
query25	0.10	0.08	0.04
query26	0.14	0.13	0.14
query27	0.06	0.05	0.04
query28	3.99	1.21	1.02
query29	12.62	3.95	3.18
query30	0.28	0.14	0.12
query31	2.81	0.59	0.39
query32	3.22	0.54	0.46
query33	3.09	3.09	3.14
query34	16.95	5.18	4.54
query35	4.51	4.52	4.49
query36	0.66	0.49	0.50
query37	0.10	0.07	0.07
query38	0.06	0.04	0.04
query39	0.05	0.03	0.04
query40	0.18	0.14	0.14
query41	0.08	0.03	0.03
query42	0.05	0.03	0.03
query43	0.05	0.03	0.04
Total cold run time: 98.17 s
Total hot run time: 27.35 s

@mrhhsg mrhhsg merged commit 240f07b into apache:master Nov 28, 2025
26 of 28 checks passed
@mrhhsg mrhhsg deleted the fix_prune branch November 28, 2025 03:04
nagisa-kunhah pushed a commit to nagisa-kunhah/doris that referenced this pull request Dec 14, 2025
@mrhhsg @nagisa-kunhah
[fix](olap) The tablet schema cache caused incorrect behavior in the …
c6d9536 
…complex-type column pruning functionality (apache#58373)

### What problem does this PR solve?

TabletSchema with pruned column type should not be cached.

Related PR: #xxx

Problem Summary:

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
mrhhsg added a commit that referenced this pull request Dec 23, 2025
@mrhhsg
[fix](olap) The tablet schema cache caused incorrect behavior in the …
2aea8f5 
…complex-type column pruning functionality (#58373)

### What problem does this PR solve?

TabletSchema with pruned column type should not be cached.

Related PR: #xxx

Problem Summary:

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
@kaka11chen kaka11chen mentioned this pull request Dec 23, 2025
Merged
16 tasks
yiguolei pushed a commit that referenced this pull request Dec 24, 2025
@kaka11chen @924060929
@mrhhsg @eldenmoon
[feature](reader) Optimize Complex Type Column Reading with Column Pu…
eea2586 
…rning (#59286)

### What problem does this PR solve?

Problem Summary:

### Release note

Cherry-pick #58370 #58354 #59043 #58851 #58485 #58682 #58614 #58373
#57204 #58719 #58471 #58573 #58657

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->

---------

Co-authored-by: 924060929 <lanhuajian@selectdb.com>
Co-authored-by: Jerry Hu <mrhhsg@gmail.com>
Co-authored-by: Jerry Hu <hushenggang@selectdb.com>
Co-authored-by: lihangyu <lihangyu@selectdb.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yiguolei yiguolei yiguolei approved these changes

@zhangstar333 zhangstar333 zhangstar333 approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/4.0.3-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@mrhhsg @hello-stephen @doris-robot @yiguolei @zhangstar333