Skip to content

Conversation

@liaoxin01
Copy link
Contributor

@liaoxin01 liaoxin01 commented Sep 9, 2025

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #55742

Problem Summary:

Rowsets generated by multiple empty rowset compactions may also lack resource id, and this condition was missed in the DCHECK.

F20250909 15:32:51.967670 309276 compaction.cpp:1575] Check failed: rowset->is_hole_rowset() || rowset->end_version() == 1 Non-hole rowset with version != [0-1] must have non-empty resource_id, rowset_id=0200000000000346484eac5c34cb52957cfa7d080c595eba, version=[2-6], is_hole_rowset=0, tablet_id=1757390702458
*** Check failure stack trace: ***
@ 0x5605002a49cf google::LogMessage::SendToLog()
@ 0x56050029afe0 google::LogMessage::Flush()
@ 0x56050029e6d9 google::LogMessageFatal::~LogMessageFatal()
@ 0x5604e9c77880 doris::CloudCompactionMixin::construct_output_rowset_writer()
@ 0x5604e9c7e2d0 doris::Compaction::merge_input_rowsets()
@ 0x5604e9cae844 doris::CloudCompactionMixin::execute_compact_impl()
@ 0x5604e9c74575 doris::CloudCompactionMixin::execute_compact()
@ 0x5605000be3fb doris::CloudCumulativeCompaction::execute_compact()
@ 0x56050007b679 std::_Function_handler<>::_M_invoke()
@ 0x5604ebdad1f3 doris::ThreadPool::dispatch_thread()
@ 0x5604ebd8a267 doris::Thread::supervise_thread()
@ 0x5604e75c2d27 asan_thread_start()
@ 0x7fd555ff1ac3 (unknown)
@ 0x7fd556083850 (unknown)
@ (nil) (unknown)

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Sep 9, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@liaoxin01
Copy link
Contributor Author

run buildall

@liaoxin01 liaoxin01 force-pushed the fix_empty_compaction branch from c9601d4 to b81cb9f Compare September 9, 2025 14:09
@liaoxin01
Copy link
Contributor Author

run buildall

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 9, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Sep 9, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Sep 9, 2025

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 34406 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b81cb9f9eb73c135505b8a9515716383188ffcec, data reload: false

------ Round 1 ----------------------------------
q1	17665	5325	5014	5014
q2	1992	334	212	212
q3	10245	1283	723	723
q4	10229	1018	524	524
q5	7551	2478	2318	2318
q6	185	172	139	139
q7	939	759	630	630
q8	9354	1399	1099	1099
q9	6916	5092	5178	5092
q10	6967	2364	1997	1997
q11	503	308	292	292
q12	361	360	231	231
q13	17792	3645	3025	3025
q14	240	247	219	219
q15	575	521	489	489
q16	976	997	947	947
q17	588	853	365	365
q18	7708	7087	6984	6984
q19	1362	943	563	563
q20	350	336	240	240
q21	3805	2527	2314	2314
q22	1075	1040	989	989
Total cold run time: 107378 ms
Total hot run time: 34406 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5221	5067	5072	5067
q2	252	338	230	230
q3	2153	2687	2258	2258
q4	1316	1789	1297	1297
q5	4255	4457	4598	4457
q6	233	170	135	135
q7	2080	1995	1820	1820
q8	2680	2582	2600	2582
q9	7502	7252	7318	7252
q10	3100	3283	2966	2966
q11	571	516	504	504
q12	689	782	647	647
q13	3499	4032	3223	3223
q14	298	329	298	298
q15	528	490	489	489
q16	1088	1109	1076	1076
q17	1186	1778	1383	1383
q18	7761	7881	7712	7712
q19	797	818	887	818
q20	1988	2027	1823	1823
q21	4774	4302	4304	4302
q22	1083	1034	1014	1014
Total cold run time: 53054 ms
Total hot run time: 51353 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 187293 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b81cb9f9eb73c135505b8a9515716383188ffcec, data reload: false

query1	1090	461	435	435
query2	6618	1705	1691	1691
query3	6752	225	228	225
query4	26512	23319	23615	23319
query5	4781	645	532	532
query6	349	254	230	230
query7	4678	506	301	301
query8	326	260	251	251
query9	8635	2932	2938	2932
query10	511	349	327	327
query11	15847	15416	14722	14722
query12	173	126	119	119
query13	1705	571	439	439
query14	11312	9335	9297	9297
query15	220	197	177	177
query16	7625	662	495	495
query17	1224	770	662	662
query18	2063	443	357	357
query19	219	207	221	207
query20	134	127	125	125
query21	215	130	117	117
query22	4184	4251	4103	4103
query23	34041	33162	33166	33162
query24	8232	2365	2419	2365
query25	587	530	500	500
query26	1236	282	172	172
query27	2700	513	358	358
query28	4384	2251	2233	2233
query29	789	608	516	516
query30	298	229	203	203
query31	931	814	737	737
query32	96	82	80	80
query33	561	397	356	356
query34	795	849	545	545
query35	822	824	776	776
query36	958	1034	943	943
query37	124	115	92	92
query38	3549	3587	3487	3487
query39	1467	1411	1419	1411
query40	230	142	126	126
query41	63	62	61	61
query42	133	123	120	120
query43	528	513	464	464
query44	1354	872	858	858
query45	190	174	178	174
query46	857	1041	668	668
query47	1790	1801	1763	1763
query48	405	422	325	325
query49	762	509	418	418
query50	658	682	413	413
query51	3927	3959	3868	3868
query52	120	118	109	109
query53	238	280	202	202
query54	619	616	544	544
query55	94	95	88	88
query56	351	355	324	324
query57	1199	1205	1124	1124
query58	302	283	277	277
query59	2538	2641	2508	2508
query60	365	359	353	353
query61	180	165	164	164
query62	826	714	679	679
query63	243	201	201	201
query64	4428	1149	884	884
query65	4061	3969	3990	3969
query66	1132	438	358	358
query67	15417	15465	15039	15039
query68	8580	922	636	636
query69	490	338	295	295
query70	1402	1290	1331	1290
query71	581	348	319	319
query72	5961	2643	5303	2643
query73	807	755	363	363
query74	9247	9043	8661	8661
query75	4088	3226	2838	2838
query76	3692	1151	732	732
query77	803	398	338	338
query78	9615	9770	8838	8838
query79	2796	882	607	607
query80	682	585	529	529
query81	473	266	234	234
query82	463	168	137	137
query83	299	267	254	254
query84	305	117	94	94
query85	944	461	428	428
query86	349	315	298	298
query87	3834	3753	3695	3695
query88	3027	2271	2259	2259
query89	429	336	307	307
query90	1960	244	236	236
query91	164	174	138	138
query92	94	76	78	76
query93	1202	1010	661	661
query94	710	426	318	318
query95	412	336	416	336
query96	493	592	281	281
query97	2970	2959	2861	2861
query98	250	222	216	216
query99	1453	1441	1295	1295
Total cold run time: 278328 ms
Total hot run time: 187293 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.13 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b81cb9f9eb73c135505b8a9515716383188ffcec, data reload: false

query1	0.06	0.05	0.05
query2	0.09	0.05	0.05
query3	0.25	0.08	0.08
query4	1.60	0.12	0.12
query5	0.27	0.28	0.24
query6	1.17	0.66	0.65
query7	0.04	0.03	0.03
query8	0.05	0.04	0.05
query9	0.61	0.53	0.53
query10	0.59	0.59	0.59
query11	0.16	0.11	0.12
query12	0.15	0.12	0.12
query13	0.63	0.64	0.65
query14	1.02	1.04	1.04
query15	0.88	0.87	0.86
query16	0.41	0.40	0.40
query17	1.03	1.03	1.05
query18	0.22	0.21	0.21
query19	2.02	1.82	1.78
query20	0.01	0.02	0.01
query21	15.40	0.96	0.60
query22	0.78	1.27	0.75
query23	14.76	1.40	0.58
query24	6.74	0.84	0.91
query25	0.47	0.35	0.09
query26	0.53	0.15	0.14
query27	0.06	0.05	0.05
query28	10.29	0.96	0.43
query29	12.59	3.91	3.28
query30	0.29	0.13	0.11
query31	2.84	0.61	0.40
query32	3.23	0.56	0.48
query33	3.17	3.21	3.11
query34	16.00	5.45	4.90
query35	4.91	4.93	4.94
query36	0.71	0.51	0.49
query37	0.12	0.07	0.07
query38	0.07	0.04	0.04
query39	0.04	0.03	0.03
query40	0.17	0.15	0.14
query41	0.09	0.04	0.03
query42	0.03	0.03	0.03
query43	0.05	0.03	0.04
Total cold run time: 104.6 s
Total hot run time: 30.13 s

@doris-robot
Copy link

BE UT Coverage Report

Increment line coverage 95.24% (20/21) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 51.94% (17286/33278)
Line Coverage 37.31% (157635/422485)
Region Coverage 31.96% (120285/376400)
Branch Coverage 33.34% (52825/158459)

@hello-stephen
Copy link
Contributor

BE Regression && UT Coverage Report

Increment line coverage 50.00% (12/24) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 71.69% (23428/32680)
Line Coverage 58.57% (247322/422234)
Region Coverage 54.67% (208751/381834)
Branch Coverage 55.76% (88859/159362)

@morningman morningman added Stale and removed Stale labels Sep 9, 2025
@liaoxin01
Copy link
Contributor Author

run cloud_p0 5

@hello-stephen
Copy link
Contributor

BE Regression && UT Coverage Report

Increment line coverage 50.00% (12/24) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 70.75% (23121/32680)
Line Coverage 57.09% (241074/422234)
Region Coverage 52.40% (200073/381834)
Branch Coverage 54.04% (86127/159362)

1 similar comment
@hello-stephen
Copy link
Contributor

BE Regression && UT Coverage Report

Increment line coverage 50.00% (12/24) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 70.75% (23121/32680)
Line Coverage 57.09% (241074/422234)
Region Coverage 52.40% (200073/381834)
Branch Coverage 54.04% (86127/159362)

@liaoxin01 liaoxin01 merged commit bd79515 into apache:master Sep 10, 2025
30 of 33 checks passed
@liaoxin01 liaoxin01 deleted the fix_empty_compaction branch September 10, 2025 06:29
liaoxin01 added a commit to liaoxin01/doris that referenced this pull request Sep 12, 2025
…mpty_rowset_metadata (apache#55837)

Rowsets generated by multiple empty rowset compactions may also lack
resource id, and this condition was missed in the DCHECK.
@morrySnow morrySnow mentioned this pull request Sep 22, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/3.1.1-merged reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants