[fix](index compaction)Add column unique id check before use #47562

qidaye · 2025-02-06T11:00:23Z

What problem does this PR solve?

Problem Summary:

This pull request introduces a new method to check for the existence of a column by its unique ID in the TabletSchema class and utilizes this method in the compaction process to improve error handling and logging.

Enhancements to error handling and logging:

be/src/olap/compaction.cpp: Added a check to skip index compaction for columns not found in the tablet schema and log a warning message.

New method addition:

be/src/olap/tablet_schema.cpp: Implemented the has_column_unique_id method to check for the existence of a column by its unique ID in the TabletSchema class.
be/src/olap/tablet_schema.h: Declared the has_column_unique_id method in the TabletSchema class.

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

Thearas · 2025-02-06T11:00:29Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

qidaye · 2025-02-06T11:02:23Z

run buildall

be/src/olap/tablet_schema.cpp

qidaye · 2025-02-06T11:07:19Z

run buildall

doris-robot · 2025-02-06T12:26:17Z

TPC-H: Total hot run time: 31762 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 54a26c60eaa2d717780e167939651397e7ff5a74, data reload: false

------ Round 1 ----------------------------------
q1	17612	5494	5184	5184
q2	2050	303	175	175
q3	10415	1259	786	786
q4	10203	1028	553	553
q5	7550	2449	2350	2350
q6	188	166	132	132
q7	929	755	596	596
q8	9296	1282	1115	1115
q9	4931	4627	4910	4627
q10	6844	2325	1876	1876
q11	478	288	271	271
q12	348	359	216	216
q13	17786	3699	3061	3061
q14	241	231	212	212
q15	520	485	458	458
q16	630	610	576	576
q17	593	884	344	344
q18	6678	6230	6345	6230
q19	2128	963	563	563
q20	305	327	191	191
q21	2831	2222	1954	1954
q22	370	329	292	292
Total cold run time: 102926 ms
Total hot run time: 31762 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5307	5225	5189	5189
q2	242	328	240	240
q3	2128	2724	2301	2301
q4	1453	1813	1357	1357
q5	4243	4206	4147	4147
q6	215	167	121	121
q7	1916	1851	1743	1743
q8	2683	2640	2595	2595
q9	7328	7254	7200	7200
q10	3020	3246	2789	2789
q11	591	501	498	498
q12	682	782	603	603
q13	3398	3933	3320	3320
q14	285	292	288	288
q15	523	478	459	459
q16	656	669	646	646
q17	1146	1680	1331	1331
q18	7576	7467	7224	7224
q19	867	848	992	848
q20	1964	2005	1916	1916
q21	5517	5093	4783	4783
q22	619	585	528	528
Total cold run time: 52359 ms
Total hot run time: 50126 ms

doris-robot · 2025-02-06T12:38:26Z

TPC-DS: Total hot run time: 191292 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 54a26c60eaa2d717780e167939651397e7ff5a74, data reload: false

query1	1306	984	915	915
query2	6293	1906	1867	1867
query3	11141	4548	4420	4420
query4	56974	25482	23561	23561
query5	5147	536	485	485
query6	347	194	210	194
query7	4965	514	297	297
query8	326	247	228	228
query9	6205	2529	2511	2511
query10	417	320	274	274
query11	15122	15393	14850	14850
query12	162	112	108	108
query13	1106	523	389	389
query14	10259	6766	7012	6766
query15	202	215	186	186
query16	6995	686	504	504
query17	1091	730	582	582
query18	1521	438	317	317
query19	211	201	174	174
query20	135	131	128	128
query21	268	127	102	102
query22	4333	4520	4588	4520
query23	33907	33444	33506	33444
query24	5614	2432	2426	2426
query25	472	463	402	402
query26	718	276	157	157
query27	1761	510	342	342
query28	2773	2431	2409	2409
query29	572	558	408	408
query30	215	191	152	152
query31	911	912	786	786
query32	72	74	79	74
query33	415	369	303	303
query34	788	868	499	499
query35	786	840	759	759
query36	955	1037	904	904
query37	124	100	79	79
query38	4335	4399	4210	4210
query39	1471	1449	1469	1449
query40	207	115	110	110
query41	50	53	50	50
query42	125	110	102	102
query43	492	515	486	486
query44	1318	809	815	809
query45	185	187	168	168
query46	900	1084	689	689
query47	1882	1846	1801	1801
query48	400	405	313	313
query49	707	523	440	440
query50	713	788	442	442
query51	4267	4313	4269	4269
query52	114	114	103	103
query53	236	256	188	188
query54	479	493	422	422
query55	87	87	91	87
query56	283	274	268	268
query57	1167	1176	1140	1140
query58	257	242	244	242
query59	2820	2957	2738	2738
query60	297	282	274	274
query61	120	110	114	110
query62	765	759	683	683
query63	236	193	199	193
query64	1702	1032	674	674
query65	3232	3164	3116	3116
query66	713	390	291	291
query67	16134	15680	15494	15494
query68	5483	796	530	530
query69	523	294	265	265
query70	1170	1127	1095	1095
query71	450	284	258	258
query72	6449	3713	3850	3713
query73	1216	764	375	375
query74	9196	9080	8941	8941
query75	3309	3127	2678	2678
query76	3924	1187	741	741
query77	553	359	276	276
query78	10010	10000	9351	9351
query79	2375	804	601	601
query80	603	527	446	446
query81	481	287	235	235
query82	366	157	120	120
query83	179	171	151	151
query84	290	101	73	73
query85	766	362	318	318
query86	395	302	277	277
query87	4502	4496	4571	4496
query88	3591	2186	2180	2180
query89	395	324	285	285
query90	1824	190	229	190
query91	139	139	105	105
query92	74	59	57	57
query93	1941	1025	596	596
query94	661	403	291	291
query95	352	273	250	250
query96	480	568	271	271
query97	2847	2873	2739	2739
query98	225	211	203	203
query99	1338	1441	1302	1302
Total cold run time: 296410 ms
Total hot run time: 191292 ms

doris-robot · 2025-02-06T12:43:42Z

ClickBench: Total hot run time: 31.11 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 54a26c60eaa2d717780e167939651397e7ff5a74, data reload: false

query1	0.04	0.03	0.03
query2	0.09	0.04	0.03
query3	0.23	0.07	0.06
query4	1.62	0.10	0.10
query5	0.42	0.44	0.41
query6	1.18	0.66	0.67
query7	0.02	0.02	0.02
query8	0.04	0.03	0.04
query9	0.59	0.52	0.52
query10	0.58	0.59	0.58
query11	0.16	0.11	0.11
query12	0.14	0.11	0.11
query13	0.60	0.61	0.61
query14	2.79	2.71	2.82
query15	0.92	0.85	0.85
query16	0.38	0.37	0.38
query17	1.01	1.04	1.03
query18	0.21	0.20	0.20
query19	1.90	1.80	2.03
query20	0.02	0.01	0.02
query21	15.35	0.86	0.55
query22	0.76	1.26	0.74
query23	14.76	1.37	0.64
query24	6.84	1.77	1.19
query25	0.48	0.22	0.08
query26	0.70	0.18	0.13
query27	0.06	0.04	0.05
query28	8.72	0.85	0.43
query29	12.53	3.90	3.26
query30	0.25	0.09	0.06
query31	2.83	0.59	0.38
query32	3.22	0.54	0.46
query33	2.97	3.07	3.02
query34	15.89	5.20	4.52
query35	4.54	4.55	4.58
query36	0.66	0.49	0.48
query37	0.09	0.07	0.07
query38	0.06	0.04	0.04
query39	0.03	0.02	0.02
query40	0.16	0.13	0.13
query41	0.09	0.03	0.03
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 103.99 s
Total hot run time: 31.11 s

doris-robot · 2025-02-06T12:48:46Z

TeamCity be ut coverage result:
Function Coverage: 42.09% (11020/26184)
Line Coverage: 32.37% (93054/287512)
Region Coverage: 31.53% (47713/151346)
Branch Coverage: 27.53% (24135/87656)
Coverage Report: http://coverage.selectdb-in.cc/coverage/54a26c60eaa2d717780e167939651397e7ff5a74_54a26c60eaa2d717780e167939651397e7ff5a74/report/index.html

airborne12

LGTM

airborne12 · 2025-02-07T03:37:34Z

we need to add UT test for index compaction

github-actions · 2025-02-07T03:38:07Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-02-07T03:38:10Z

PR approved by anyone and no changes requested.

zzzxl1993

LGTM

Problem Summary: This pull request introduces a new method to check for the existence of a column by its unique ID in the `TabletSchema` class and utilizes this method in the compaction process to improve error handling and logging.

…use #47562 (#47577) Cherry-picked from #47562 Co-authored-by: qiye <luen@selectdb.com>

…47562) Problem Summary: This pull request introduces a new method to check for the existence of a column by its unique ID in the `TabletSchema` class and utilizes this method in the compaction process to improve error handling and logging.

…50056 (#50732) pick #47562 #50056 from master --------- Co-authored-by: qiye <luen@selectdb.com>

…47562) Problem Summary: This pull request introduces a new method to check for the existence of a column by its unique ID in the `TabletSchema` class and utilizes this method in the compaction process to improve error handling and logging.

[opt](index compaction)Add column unique id check before use

d744114

airborne12 reviewed Feb 6, 2025

View reviewed changes

be/src/olap/tablet_schema.cpp Outdated Show resolved Hide resolved

opt

54a26c6

dataroaring added the dev/3.0.x label Feb 6, 2025

airborne12 approved these changes Feb 7, 2025

View reviewed changes

airborne12 changed the title ~~[opt](index compaction)Add column unique id check before use~~ [fix](index compaction)Add column unique id check before use Feb 7, 2025

github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Feb 7, 2025

zzzxl1993 approved these changes Feb 7, 2025

View reviewed changes

qidaye merged commit a10a2a2 into apache:master Feb 7, 2025
28 of 31 checks passed

qidaye deleted the add_col_unique_id_protection branch February 7, 2025 04:07

github-actions bot mentioned this pull request Feb 7, 2025

branch-3.0: [fix](index compaction)Add column unique id check before use #47562 #47577

Merged

dataroaring pushed a commit that referenced this pull request Feb 8, 2025

branch-3.0: [fix](index compaction)Add column unique id check before …

ba97470

…use #47562 (#47577) Cherry-picked from #47562 Co-authored-by: qiye <luen@selectdb.com>

dataroaring added dev/3.0.4-merged and removed dev/3.0.x labels Feb 8, 2025

airborne12 mentioned this pull request May 9, 2025

branch-2.1: [Fix](inverted index) fix rename column build index bug #50056 #50732

Merged

yiguolei pushed a commit that referenced this pull request May 9, 2025

branch-2.1: [Fix](inverted index) fix rename column build index bug #…

0002500

…50056 (#50732) pick #47562 #50056 from master --------- Co-authored-by: qiye <luen@selectdb.com>

yiguolei added the dev/2.1.10-merged label May 9, 2025

yiguolei mentioned this pull request May 13, 2025

2.1.10 Release Note #50853

Closed

[fix](index compaction)Add column unique id check before use #47562

[fix](index compaction)Add column unique id check before use #47562

Uh oh!

Conversation

qidaye commented Feb 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Enhancements to error handling and logging:

New method addition:

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

Thearas commented Feb 6, 2025

Uh oh!

qidaye commented Feb 6, 2025

Uh oh!

Uh oh!

qidaye commented Feb 6, 2025

Uh oh!

doris-robot commented Feb 6, 2025

Uh oh!

doris-robot commented Feb 6, 2025

Uh oh!

doris-robot commented Feb 6, 2025

Uh oh!

doris-robot commented Feb 6, 2025

Uh oh!

airborne12 left a comment

Choose a reason for hiding this comment

Uh oh!

airborne12 commented Feb 7, 2025

Uh oh!

github-actions bot commented Feb 7, 2025

Uh oh!

github-actions bot commented Feb 7, 2025

Uh oh!

zzzxl1993 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

qidaye commented Feb 6, 2025 •

edited

Loading