[opt][editlog] Added the ability to skip certain editlog exceptions when fe is abnormal #54090

deardeng · 2025-07-30T08:23:27Z

…hen fe is abnormal

The previous operation and maintenance method force_skip_journal_ids can only skip a specific logid. If there are many abnormal logids, it is difficult to configure

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.
  https://github.com/apache/doris-website/pull/2731/files

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

…hen fe is abnormal

hello-stephen · 2025-07-30T08:23:41Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

deardeng · 2025-07-30T08:24:46Z

run buildall

github-actions · 2025-07-30T09:20:11Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-07-30T09:20:13Z

PR approved by anyone and no changes requested.

hello-stephen · 2025-07-30T09:41:06Z

FE UT Coverage Report

Increment line coverage 0.00% (0/10) 🎉
Increment coverage report
Complete coverage report

doris-robot · 2025-07-30T09:53:23Z

TPC-H: Total hot run time: 33902 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ce72d770604e4b6bf52e6c30204d4fddf7120854, data reload: false

------ Round 1 ----------------------------------
q1	17701	5735	5483	5483
q2	1915	268	180	180
q3	10677	1348	694	694
q4	10364	998	497	497
q5	10037	2249	2318	2249
q6	219	163	129	129
q7	875	757	623	623
q8	9301	1291	1020	1020
q9	7132	5156	5199	5156
q10	6954	2333	1980	1980
q11	463	274	263	263
q12	368	358	224	224
q13	17846	3445	2947	2947
q14	227	250	220	220
q15	549	476	459	459
q16	416	436	363	363
q17	566	832	348	348
q18	7338	7056	7058	7056
q19	1414	978	517	517
q20	315	310	217	217
q21	5149	2497	2266	2266
q22	1029	1046	1011	1011
Total cold run time: 110855 ms
Total hot run time: 33902 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5898	5740	5750	5740
q2	270	317	211	211
q3	2079	2563	2196	2196
q4	1299	1690	1409	1409
q5	4515	4459	4438	4438
q6	221	174	132	132
q7	1917	1925	1796	1796
q8	2493	2612	2541	2541
q9	7347	7391	7321	7321
q10	3106	3209	2962	2962
q11	557	696	503	503
q12	680	752	611	611
q13	3376	3704	3300	3300
q14	298	322	288	288
q15	503	456	461	456
q16	459	495	454	454
q17	1199	1436	1414	1414
q18	8025	7718	7656	7656
q19	13690	937	1025	937
q20	1922	1892	1754	1754
q21	7017	4255	4222	4222
q22	1031	1019	951	951
Total cold run time: 67902 ms
Total hot run time: 51292 ms

doris-robot · 2025-07-30T10:05:00Z

TPC-DS: Total hot run time: 162563 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ce72d770604e4b6bf52e6c30204d4fddf7120854, data reload: false

reason	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 22:43:56	2023-12-26 22:44:01	NULL	utf-8	NULL	NULL	
============================================
query1	1000	385	461	385
query2	6506	1804	1641	1641
query3	6754	222	220	220
query4	27253	24006	23084	23084
query5	4286	629	543	543
query6	323	243	228	228
query7	4638	519	300	300
query8	293	241	222	222
query9	8596	3182	3106	3106
query10	482	343	285	285
query11	15958	15553	14819	14819
query12	194	144	128	128
query13	1677	558	433	433
query14	9568	7766	7643	7643
query15	217	208	165	165
query16	7850	640	468	468
query17	1661	787	641	641
query18	2104	465	329	329
query19	295	215	190	190
query20	159	141	134	134
query21	219	129	113	113
query22	3976	4003	3854	3854
query23	34323	34180	34299	34180
query24	8149	2376	2449	2376
query25	558	510	443	443
query26	916	291	163	163
query27	2679	501	355	355
query28	4247	2265	2259	2259
query29	715	606	504	504
query30	303	235	203	203
query31	882	803	707	707
query32	88	82	82	82
query33	567	435	377	377
query34	811	845	514	514
query35	825	807	745	745
query36	1042	1027	936	936
query37	139	116	94	94
query38	4099	4008	3979	3979
query39	1435	1392	1380	1380
query40	239	142	139	139
query41	70	57	53	53
query42	139	121	123	121
query43	513	493	486	486
query44	1393	875	880	875
query45	200	189	177	177
query46	970	1048	679	679
query47	1780	1838	1738	1738
query48	403	422	326	326
query49	683	505	408	408
query50	652	681	421	421
query51	4136	4162	4097	4097
query52	129	130	121	121
query53	272	293	213	213
query54	634	641	587	587
query55	93	85	85	85
query56	339	359	341	341
query57	1207	1216	1132	1132
query58	346	341	348	341
query59	2591	2603	2589	2589
query60	398	404	374	374
query61	137	121	130	121
query62	800	741	658	658
query63	250	211	214	211
query64	3514	1108	819	819
query65	4208	4156	4155	4155
query66	943	434	332	332
query67	query68	18228	870	869	869
query69	1165	274	289	274
query70	1384	1131	1118	1118
query71	737	317	322	317
query72	9161	2248	2166	2166
query73	3469	698	369	369
query74	8917	8756	8614	8614
query75	7610	3142	2648	2648
query76	8692	1227	776	776
query77	1155	407	404	404
query78	query79	17719	616	575	575
query80	1793	586	489	489
query81	562	277	246	246
query82	552	155	113	113
query83	353	284	277	277
query84	307	102	83	83
query85	1715	405	344	344
query86	348	341	299	299
query87	4306	4294	4133	4133
query88	5617	2208	2225	2208
query89	542	374	322	322
query90	2595	233	245	233
query91	146	139	112	112
query92	94	77	70	70
query93	6922	997	663	663
query94	1180	390	276	276
query95	429	332	320	320
query96	496	580	277	277
query97	2690	2687	2637	2637
query98	254	222	223	222
query99	1484	1379	1282	1282
Total cold run time: 303448 ms
Total hot run time: 162563 ms

fe/fe-core/src/main/java/org/apache/doris/persist/EditLog.java

deardeng · 2025-08-10T05:35:26Z

run buildall

doris-robot · 2025-08-10T06:24:01Z

TPC-H: Total hot run time: 33304 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ecdd6e0ddef37d0ddaae4e2cc83a0aff0ed6bd6f, data reload: false

------ Round 1 ----------------------------------
q1	16379	5375	5140	5140
q2	1911	258	190	190
q3	9804	1244	678	678
q4	9583	973	499	499
q5	7202	2283	2250	2250
q6	176	164	129	129
q7	890	733	622	622
q8	8984	1252	1076	1076
q9	6721	5094	5056	5056
q10	6897	2377	1979	1979
q11	450	286	271	271
q12	342	354	220	220
q13	17001	3433	2945	2945
q14	230	241	225	225
q15	511	453	457	453
q16	426	419	376	376
q17	556	798	361	361
q18	7156	6917	6908	6908
q19	1030	991	523	523
q20	318	303	203	203
q21	3438	2412	2192	2192
q22	1065	1061	1008	1008
Total cold run time: 101070 ms
Total hot run time: 33304 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5328	5282	5503	5282
q2	233	305	212	212
q3	2068	2565	2157	2157
q4	1331	1672	1332	1332
q5	4093	4028	4058	4028
q6	206	181	136	136
q7	2065	1905	1786	1786
q8	2578	2424	2389	2389
q9	7596	7288	7450	7288
q10	3204	3340	2922	2922
q11	549	514	492	492
q12	651	808	634	634
q13	3347	3715	3163	3163
q14	283	309	300	300
q15	492	457	451	451
q16	438	486	472	472
q17	1147	1398	1349	1349
q18	8362	7823	7692	7692
q19	11425	908	949	908
q20	1971	1994	1755	1755
q21	14903	4260	4215	4215
q22	1039	1036	980	980
Total cold run time: 73309 ms
Total hot run time: 49943 ms

doris-robot · 2025-08-10T06:35:34Z

TPC-DS: Total hot run time: 168967 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ecdd6e0ddef37d0ddaae4e2cc83a0aff0ed6bd6f, data reload: false

============================================
query1	990	390	411	390
query2	6546	1822	1677	1677
query3	6747	234	220	220
query4	26870	23223	22759	22759
query5	4359	649	519	519
query6	334	236	214	214
query7	4621	521	294	294
query8	272	240	232	232
query9	8606	2953	2907	2907
query10	475	365	297	297
query11	15569	14958	14796	14796
query12	185	138	131	131
query13	1671	556	430	430
query14	9161	5762	5705	5705
query15	205	188	166	166
query16	7119	673	490	490
query17	941	756	608	608
query18	2019	435	311	311
query19	215	202	178	178
query20	178	143	143	143
query21	216	125	121	121
query22	4144	3871	3767	3767
query23	34667	34314	34162	34162
query24	5214	2366	2415	2366
query25	492	517	447	447
query26	713	288	158	158
query27	1960	488	346	346
query28	2814	2333	2349	2333
query29	615	609	492	492
query30	294	232	193	193
query31	815	779	731	731
query32	89	76	77	76
query33	453	406	421	406
query34	803	842	512	512
query35	812	837	765	765
query36	1027	1029	943	943
query37	132	109	85	85
query38	3861	3934	3970	3934
query39	1426	1378	1372	1372
query40	232	141	129	129
query41	60	59	54	54
query42	140	126	124	124
query43	526	506	467	467
query44	1381	849	861	849
query45	197	191	182	182
query46	947	1058	663	663
query47	1800	1843	1775	1775
query48	407	430	323	323
query49	679	499	410	410
query50	658	690	411	411
query51	4209	4238	4181	4181
query52	123	132	126	126
query53	263	305	217	217
query54	672	651	580	580
query55	91	89	88	88
query56	372	371	372	371
query57	1216	1230	1139	1139
query58	348	340	339	339
query59	2679	2617	2558	2558
query60	436	394	378	378
query61	129	129	123	123
query62	747	746	716	716
query63	248	211	222	211
query64	2320	1099	785	785
query65	4244	4131	4124	4124
query66	725	454	341	341
query67	query68	17913	931	788	788
query69	1023	274	279	274
query70	1422	1166	1101	1101
query71	721	340	305	305
query72	9189	2309	2371	2309
query73	3491	625	350	350
query74	9046	8967	8706	8706
query75	7679	3187	2663	2663
query76	8928	1206	793	793
query77	1137	403	331	331
query78	9392	9655	8950	8950
query79	14165	611	581	581
query80	991	564	483	483
query81	546	268	229	229
query82	310	151	115	115
query83	287	288	267	267
query84	307	99	81	81
query85	842	383	350	350
query86	355	305	326	305
query87	4238	4206	4049	4049
query88	3688	2240	2244	2240
query89	477	359	329	329
query90	2776	227	230	227
query91	144	144	127	127
query92	91	74	66	66
query93	4032	1026	655	655
query94	893	379	271	271
query95	397	322	321	321
query96	512	595	283	283
query97	2659	2685	2550	2550
query98	244	233	228	228
query99	1493	1444	1300	1300
Total cold run time: 292867 ms
Total hot run time: 168967 ms

doris-robot · 2025-08-10T06:41:13Z

ClickBench: Total hot run time: 34.95 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ecdd6e0ddef37d0ddaae4e2cc83a0aff0ed6bd6f, data reload: false

query1	0.04	0.03	0.03
query2	0.11	0.06	0.06
query3	0.28	0.07	0.07
query4	1.61	0.08	0.08
query5	0.43	0.42	0.42
query6	1.16	0.68	0.69
query7	0.03	0.01	0.02
query8	0.06	0.05	0.05
query9	0.60	0.48	0.49
query10	0.54	0.53	0.56
query11	0.26	0.12	0.12
query12	0.25	0.13	0.13
query13	0.68	0.71	0.69
query14	0.94	1.19	1.03
query15	1.04	0.93	0.95
query16	0.39	0.39	0.38
query17	1.06	1.11	1.09
query18	0.24	0.24	0.25
query19	2.01	1.80	2.01
query20	0.01	0.02	0.01
query21	15.35	1.00	0.71
query22	0.95	1.09	0.90
query23	14.69	1.57	0.96
query24	5.38	0.63	0.36
query25	0.19	0.12	0.11
query26	0.57	0.23	0.19
query27	0.11	0.11	0.10
query28	11.00	1.15	0.64
query29	12.65	4.02	3.50
query30	3.07	3.07	2.97
query31	2.82	0.63	0.46
query32	3.25	0.65	0.57
query33	3.11	3.27	3.26
query34	16.43	5.37	5.01
query35	4.77	5.00	4.98
query36	0.70	0.54	0.53
query37	0.25	0.23	0.22
query38	0.26	0.25	0.24
query39	0.07	0.05	0.05
query40	0.21	0.18	0.16
query41	0.13	0.08	0.07
query42	0.09	0.07	0.07
query43	0.07	0.07	0.06
Total cold run time: 107.86 s
Total hot run time: 34.95 s

hello-stephen · 2025-08-10T06:52:10Z

FE UT Coverage Report

Increment line coverage 0.00% (0/10) 🎉
Increment coverage report
Complete coverage report

dataroaring

LGTM

github-actions · 2025-08-11T13:47:37Z

PR approved by at least one committer and no changes requested.

#54090) …hen fe is abnormal The previous operation and maintenance method force_skip_journal_ids can only skip a specific logid. If there are many abnormal logids, it is difficult to configure

…exceptions w… #54090 (#55202) Cherry-picked from #54090 Co-authored-by: deardeng <dengxin@selectdb.com>

#54090) …hen fe is abnormal The previous operation and maintenance method force_skip_journal_ids can only skip a specific logid. If there are many abnormal logids, it is difficult to configure

…exceptions when fe is abnormal #54090 (#55204) Cherry-picked from #54090 Co-authored-by: deardeng <dengxin@selectdb.com>

[opt][editlog] Added the ability to skip certain editlog exceptions w…

ce72d77

…hen fe is abnormal

w41ter previously approved these changes Jul 30, 2025

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 30, 2025

github-actions bot added the reviewed label Jul 30, 2025

dataroaring reviewed Aug 8, 2025

View reviewed changes

fe/fe-core/src/main/java/org/apache/doris/persist/EditLog.java Outdated Show resolved Hide resolved

fix review

ecdd6e0

deardeng dismissed w41ter’s stale review via ecdd6e0 August 10, 2025 05:35

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Aug 10, 2025

dataroaring approved these changes Aug 11, 2025

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 11, 2025

w41ter approved these changes Aug 19, 2025

View reviewed changes

dataroaring merged commit 56c1272 into apache:master Aug 22, 2025
27 of 30 checks passed

dataroaring added dev/3.0.x dev/2.1.x dev/3.1.x usercase Important user case type label labels Aug 22, 2025

This was referenced Aug 22, 2025

branch-3.0: [opt][editlog] Added the ability to skip certain editlog exceptions w… #54090 #55202

Merged

branch-2.1: [opt][editlog] Added the ability to skip certain editlog exceptions w… #54090 #55203

Open

github-actions bot mentioned this pull request Aug 22, 2025

branch-3.1: [opt][editlog] Added the ability to skip certain editlog exceptions when fe is abnormal #54090 #55204

Merged

dataroaring pushed a commit that referenced this pull request Aug 24, 2025

branch-3.0: [opt][editlog] Added the ability to skip certain editlog …

e8839ea

…exceptions w… #54090 (#55202) Cherry-picked from #54090 Co-authored-by: deardeng <dengxin@selectdb.com>

dataroaring added dev/3.0.8-merged and removed dev/3.0.x labels Aug 24, 2025

morrySnow changed the title ~~[opt][editlog] Added the ability to skip certain editlog exceptions w…~~ [opt][editlog] Added the ability to skip certain editlog exceptions when fe is abnormal Aug 25, 2025

gavinchou mentioned this pull request Sep 1, 2025

Release Note 3.0.8 #55553

Open

morrySnow pushed a commit that referenced this pull request Sep 4, 2025

branch-3.1: [opt][editlog] Added the ability to skip certain editlog …

de7183a

…exceptions when fe is abnormal #54090 (#55204) Cherry-picked from #54090 Co-authored-by: deardeng <dengxin@selectdb.com>

morrySnow added dev/3.1.1-merged and removed dev/3.1.x labels Sep 4, 2025

morrySnow mentioned this pull request Sep 22, 2025

Release Note 3.1.1 #56289

Closed

[opt][editlog] Added the ability to skip certain editlog exceptions when fe is abnormal #54090

[opt][editlog] Added the ability to skip certain editlog exceptions when fe is abnormal #54090

Uh oh!

Conversation

deardeng commented Jul 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Jul 30, 2025

Uh oh!

deardeng commented Jul 30, 2025

Uh oh!

github-actions bot commented Jul 30, 2025

Uh oh!

github-actions bot commented Jul 30, 2025

Uh oh!

hello-stephen commented Jul 30, 2025

FE UT Coverage Report

Uh oh!

doris-robot commented Jul 30, 2025

Uh oh!

doris-robot commented Jul 30, 2025

Uh oh!

Uh oh!

deardeng commented Aug 10, 2025

Uh oh!

doris-robot commented Aug 10, 2025

Uh oh!

doris-robot commented Aug 10, 2025

Uh oh!

doris-robot commented Aug 10, 2025

Uh oh!

hello-stephen commented Aug 10, 2025

FE UT Coverage Report

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Aug 11, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

deardeng commented Jul 30, 2025 •

edited

Loading