[bugfix](k8s) using stdout to output #54129

yiguolei · 2025-07-31T01:46:21Z

What problem does this PR solve?

K8S using stdout to capture BE INFO logs. But when BE crash be will try to output crash stack to be.out file. In the past, it could not output to be.out file. In this feature, I format the behaviors of be log:

-- daemon, output log to be.info file and output crash stack to be.out file.
--console,
2.1 enable_file_logger = true, output all logs to console (stdout) and output all logs to be.info and crash stack to be.out
2.2 enable_file_logger = false, output all logs to console(stdout) and crash stack to be.out

we will always output crash stack to logs/be.out file.

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

Thearas · 2025-07-31T01:46:26Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

github-actions · 2025-07-31T04:38:50Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-07-31T04:38:53Z

PR approved by anyone and no changes requested.

yiguolei · 2025-07-31T04:39:55Z

run buildall

doris-robot · 2025-07-31T05:11:08Z

TPC-H: Total hot run time: 33314 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0deee9177f6359cb6cfaa7134b695d70c87fba38, data reload: false

------ Round 1 ----------------------------------
q1	17617	5759	5342	5342
q2	1966	265	186	186
q3	10298	1244	687	687
q4	10224	958	508	508
q5	7516	2335	2204	2204
q6	177	163	129	129
q7	891	753	610	610
q8	9299	1251	1013	1013
q9	6743	5157	5114	5114
q10	6923	2351	1953	1953
q11	465	285	264	264
q12	337	359	228	228
q13	17770	3418	2930	2930
q14	226	229	219	219
q15	548	456	459	456
q16	418	426	365	365
q17	554	828	345	345
q18	7314	6958	6806	6806
q19	1131	972	525	525
q20	307	308	212	212
q21	3349	3000	2244	2244
q22	1034	1058	974	974
Total cold run time: 105107 ms
Total hot run time: 33314 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5833	5782	5865	5782
q2	227	307	210	210
q3	2079	2572	2222	2222
q4	1291	1698	1332	1332
q5	4093	4339	4537	4339
q6	218	174	135	135
q7	1925	1898	1716	1716
q8	2615	2780	2457	2457
q9	7348	7329	7457	7329
q10	3221	3307	2928	2928
q11	547	495	493	493
q12	682	776	689	689
q13	3333	3706	3454	3454
q14	282	296	282	282
q15	501	440	442	440
q16	434	503	460	460
q17	1126	1448	1309	1309
q18	11829	7821	7421	7421
q19	2537	946	1065	946
q20	1964	2084	1843	1843
q21	11813	4602	4217	4217
q22	1050	1019	975	975
Total cold run time: 64948 ms
Total hot run time: 50979 ms

doris-robot · 2025-07-31T05:22:53Z

TPC-DS: Total hot run time: 163632 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0deee9177f6359cb6cfaa7134b695d70c87fba38, data reload: false

reason	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 22:43:56	2023-12-26 22:44:01	NULL	utf-8	NULL	NULL	
============================================
query1	1020	376	420	376
query2	6524	1792	1676	1676
query3	6749	223	222	222
query4	26852	23885	23476	23476
query5	4323	631	529	529
query6	355	228	213	213
query7	4641	512	291	291
query8	274	226	218	218
query9	8597	3186	3157	3157
query10	473	349	293	293
query11	15747	15107	15113	15107
query12	191	137	136	136
query13	1674	588	440	440
query14	10284	8081	7865	7865
query15	215	204	174	174
query16	7782	651	482	482
query17	1781	789	651	651
query18	2051	459	331	331
query19	240	212	200	200
query20	158	144	137	137
query21	223	137	113	113
query22	3985	4069	3937	3937
query23	34406	34175	34247	34175
query24	7310	2416	2456	2416
query25	546	504	440	440
query26	729	288	167	167
query27	2450	503	350	350
query28	2968	2331	2336	2331
query29	683	616	495	495
query30	285	235	210	210
query31	929	810	712	712
query32	82	78	80	78
query33	545	410	371	371
query34	815	853	525	525
query35	802	814	760	760
query36	1011	1029	965	965
query37	131	109	92	92
query38	4044	4016	3911	3911
query39	1442	1379	1380	1379
query40	233	143	128	128
query41	60	61	59	59
query42	141	122	122	122
query43	542	538	484	484
query44	1385	888	874	874
query45	202	189	180	180
query46	934	1057	674	674
query47	1807	1825	1773	1773
query48	406	433	325	325
query49	700	502	426	426
query50	650	674	415	415
query51	4117	4165	4134	4134
query52	127	130	118	118
query53	265	300	203	203
query54	657	628	552	552
query55	87	86	84	84
query56	360	356	355	355
query57	1221	1193	1132	1132
query58	333	344	338	338
query59	2633	2625	2513	2513
query60	389	400	381	381
query61	126	124	121	121
query62	807	730	633	633
query63	254	213	208	208
query64	2901	1102	838	838
query65	4191	4118	4058	4058
query66	967	445	342	342
query67	query68	20872	903	592	592
query69	999	311	317	311
query70	1424	1110	1122	1110
query71	737	324	321	321
query72	9204	2265	4817	2265
query73	4586	761	360	360
query74	9250	8976	8749	8749
query75	7898	3094	2664	2664
query76	9045	1269	776	776
query77	1174	419	360	360
query78	query79	17300	628	584	584
query80	3246	558	486	486
query81	600	273	245	245
query82	396	150	122	122
query83	490	291	265	265
query84	299	97	83	83
query85	1710	390	329	329
query86	365	326	307	307
query87	4224	4198	4164	4164
query88	4681	2295	2264	2264
query89	571	363	315	315
query90	2651	239	244	239
query91	145	142	113	113
query92	89	70	67	67
query93	6219	957	701	701
query94	1145	381	280	280
query95	400	337	318	318
query96	498	608	295	295
query97	2726	2730	2585	2585
query98	245	223	222	222
query99	1564	1362	1269	1269
Total cold run time: 304565 ms
Total hot run time: 163632 ms

doris-robot · 2025-07-31T05:28:20Z

ClickBench: Total hot run time: 32.67 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0deee9177f6359cb6cfaa7134b695d70c87fba38, data reload: false

query1	0.04	0.04	0.04
query2	0.08	0.04	0.04
query3	0.24	0.07	0.07
query4	1.62	0.10	0.10
query5	0.43	0.43	0.43
query6	1.18	0.67	0.67
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.57	0.48	0.47
query10	0.51	0.53	0.52
query11	0.16	0.10	0.11
query12	0.15	0.11	0.12
query13	0.65	0.68	0.64
query14	1.05	0.97	1.04
query15	0.92	0.89	0.88
query16	0.39	0.39	0.39
query17	1.07	1.02	1.04
query18	0.22	0.21	0.20
query19	2.03	1.84	1.85
query20	0.02	0.01	0.01
query21	15.44	0.84	0.56
query22	0.78	1.26	0.65
query23	14.87	1.08	0.67
query24	6.82	1.78	0.41
query25	0.44	0.21	0.20
query26	0.65	0.15	0.13
query27	0.07	0.05	0.06
query28	8.94	0.80	0.44
query29	12.57	3.78	3.24
query30	2.99	2.97	2.95
query31	2.82	0.56	0.40
query32	3.23	0.57	0.51
query33	2.98	3.20	3.18
query34	16.17	5.30	4.95
query35	4.93	4.98	4.90
query36	0.71	0.52	0.51
query37	0.09	0.08	0.08
query38	0.05	0.05	0.04
query39	0.03	0.03	0.03
query40	0.18	0.14	0.14
query41	0.07	0.03	0.03
query42	0.04	0.03	0.02
query43	0.04	0.04	0.03
Total cold run time: 106.31 s
Total hot run time: 32.67 s

doris-robot · 2025-07-31T06:22:46Z

BE UT Coverage Report

Increment line coverage 0.00% (0/35) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	57.80% (16084/27826)
Line Coverage	46.54% (144674/310890)
Region Coverage	35.78% (108835/304191)
Branch Coverage	38.42% (48043/125039)

hello-stephen · 2025-07-31T07:32:48Z

BE Regression && UT Coverage Report

Increment line coverage 0.00% (0/35) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	81.11% (22156/27315)
Line Coverage	73.82% (229210/310517)
Region Coverage	61.51% (191845/311908)
Branch Coverage	65.28% (82525/126411)

yiguolei · 2025-07-31T09:39:00Z

skip check_coverage

hello-stephen · 2025-07-31T11:16:12Z

BE Regression && UT Coverage Report

Increment line coverage 0.00% (0/35) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	81.13% (22160/27315)
Line Coverage	73.83% (229261/310517)
Region Coverage	61.55% (191966/311908)
Branch Coverage	65.30% (82550/126411)

yiguolei · 2025-08-01T01:06:03Z

skip check_coverage

### What problem does this PR solve? K8S using stdout to capture BE INFO logs. But when BE crash be will try to output crash stack to be.out file. In the past, it could not output to be.out file. In this feature, I format the behaviors of be log: 1. -- daemon, output log to be.info file and output crash stack to be.out file. 2. --console, 2.1 enable_file_logger = true, output all logs to console (stdout) and output all logs to be.info and crash stack to be.out 2.2 enable_file_logger = false, output all logs to console(stdout) and crash stack to be.out we will always output crash stack to logs/be.out file.

Cherry-picked from #54129 Co-authored-by: yiguolei <guolei@selectdb.com>

yiguolei added 6 commits July 30, 2025 20:08

add glog sink

f7bff94

f

f2762ab

f

4a6adc1

f

0d7aa24

f

adbe2b0

f

8888321

yiguolei added 4 commits July 31, 2025 09:47

f

f5af4cd

f

a888c5c

f

e6eeead

f

0deee91

yiguolei added usercase Important user case type label dev/3.0.x dev/3.1.x labels Jul 31, 2025

morningman approved these changes Jul 31, 2025

View reviewed changes

github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Jul 31, 2025

jacktengg approved these changes Jul 31, 2025

View reviewed changes

yiguolei merged commit 88b62c3 into apache:master Aug 1, 2025
31 of 33 checks passed

github-actions bot mentioned this pull request Aug 1, 2025

branch-3.0: [bugfix](k8s) using stdout to output #54129 #54171

Merged

github-actions bot mentioned this pull request Aug 1, 2025

branch-3.1: [bugfix](k8s) using stdout to output #54129 #54172

Merged

morrySnow pushed a commit that referenced this pull request Aug 1, 2025

branch-3.1: [bugfix](k8s) using stdout to output #54129 (#54172)

ff8868f

Cherry-picked from #54129 Co-authored-by: yiguolei <guolei@selectdb.com>

morrySnow added dev/3.1.0-merged and removed dev/3.1.x labels Aug 1, 2025

dataroaring pushed a commit that referenced this pull request Aug 1, 2025

branch-3.0: [bugfix](k8s) using stdout to output #54129 (#54171)

9eca445

Cherry-picked from #54129 Co-authored-by: yiguolei <guolei@selectdb.com>

dataroaring added dev/3.0.7-merged and removed dev/3.0.x labels Aug 1, 2025

[bugfix](k8s) using stdout to output #54129

[bugfix](k8s) using stdout to output #54129

Uh oh!

Conversation

yiguolei commented Jul 31, 2025 • edited by morningman Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

Thearas commented Jul 31, 2025

Uh oh!

github-actions bot commented Jul 31, 2025

Uh oh!

github-actions bot commented Jul 31, 2025

Uh oh!

yiguolei commented Jul 31, 2025

Uh oh!

doris-robot commented Jul 31, 2025

Uh oh!

doris-robot commented Jul 31, 2025

Uh oh!

doris-robot commented Jul 31, 2025

Uh oh!

doris-robot commented Jul 31, 2025

BE UT Coverage Report

Uh oh!

hello-stephen commented Jul 31, 2025

BE Regression && UT Coverage Report

Uh oh!

yiguolei commented Jul 31, 2025

Uh oh!

hello-stephen commented Jul 31, 2025

BE Regression && UT Coverage Report

Uh oh!

yiguolei commented Aug 1, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

yiguolei commented Jul 31, 2025 •

edited by morningman

Loading