branch-3.1: [opt](audit) use one line in audit log and origin statement in audit table (#52032) #53022

morningman · 2025-07-09T13:43:35Z

bp #52032 #52587 #52968

…table (apache#52032) Previously, when auditing, we use replace all `\n`, `\t` in origin sql string with `\\n`, `\\t`, so that the sql string can be written in one line. But this lead to some problem: 1. User can not direct use the sql in audit log to execute. 2. Some replacement is wrong, eg, replace the `\n` in a quota string. This PR changes the logic: 1. For audit log, only replace `\n` with `\\n` to keep SQL in one line. 2. For audit table, keep the origin string. 3. Use special column and line separator for audit log load data, to avoid conflict with char in SQL

### What problem does this PR solve? Related PR: In this PR apache#52032 , the audit_log schema had changed, so the test need to fix

hello-stephen · 2025-07-09T13:43:41Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

…52968) followup apache#52032, the separator is wrong

morningman · 2025-07-09T13:56:24Z

run buildall

morningman · 2025-07-10T02:35:18Z

run buildall

doris-robot · 2025-07-10T03:12:06Z

TPC-H: Total hot run time: 39570 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ae78ba8517f2a51cec8fb5b91fead2e667d10381, data reload: false

------ Round 1 ----------------------------------
q1	17601	6717	6579	6579
q2	2050	212	194	194
q3	10471	1104	1096	1096
q4	10224	727	737	727
q5	7719	2868	2733	2733
q6	215	133	134	133
q7	977	630	615	615
q8	9354	1918	1951	1918
q9	6638	6467	6471	6467
q10	7044	2251	2289	2251
q11	465	255	264	255
q12	411	213	216	213
q13	17784	2954	2974	2954
q14	241	206	211	206
q15	501	461	472	461
q16	477	379	369	369
q17	980	546	514	514
q18	7183	6672	6688	6672
q19	1324	1000	933	933
q20	450	200	195	195
q21	3959	3137	3090	3090
q22	1111	995	1013	995
Total cold run time: 107179 ms
Total hot run time: 39570 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6589	6543	6526	6526
q2	327	232	234	232
q3	2865	2825	2942	2825
q4	1990	1786	1769	1769
q5	5726	5675	5684	5675
q6	207	123	123	123
q7	2162	1857	1772	1772
q8	3394	3469	3478	3469
q9	8879	9003	8904	8904
q10	3550	3514	3526	3514
q11	581	506	504	504
q12	808	586	613	586
q13	7768	3185	3114	3114
q14	292	283	280	280
q15	507	459	465	459
q16	500	444	431	431
q17	1823	1590	1596	1590
q18	8298	7917	7746	7746
q19	1646	1487	1480	1480
q20	2021	1805	1877	1805
q21	5147	4889	5058	4889
q22	1144	1073	1043	1043
Total cold run time: 66224 ms
Total hot run time: 58736 ms

doris-robot · 2025-07-10T03:24:20Z

TPC-DS: Total hot run time: 195922 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ae78ba8517f2a51cec8fb5b91fead2e667d10381, data reload: false

query1	1263	902	908	902
query2	6330	1945	1861	1861
query3	10934	4322	4337	4322
query4	33025	23362	23438	23362
query5	3666	472	450	450
query6	277	181	176	176
query7	4011	315	323	315
query8	295	237	230	230
query9	9559	2575	2560	2560
query10	487	271	256	256
query11	17946	15066	15249	15066
query12	160	107	105	105
query13	1559	426	415	415
query14	9361	7019	6888	6888
query15	268	180	193	180
query16	8064	452	496	452
query17	1682	599	608	599
query18	2181	336	341	336
query19	276	169	176	169
query20	124	118	116	116
query21	208	109	106	106
query22	4908	4529	4527	4527
query23	35118	33926	33640	33640
query24	11329	2896	2924	2896
query25	646	436	418	418
query26	987	172	169	169
query27	2473	359	364	359
query28	7095	2180	2137	2137
query29	826	452	462	452
query30	264	161	168	161
query31	1046	825	879	825
query32	96	63	55	55
query33	796	295	318	295
query34	1057	514	525	514
query35	922	740	738	738
query36	1123	937	956	937
query37	133	68	66	66
query38	3987	3980	3981	3980
query39	1521	1458	1468	1458
query40	202	105	107	105
query41	50	51	48	48
query42	111	107	105	105
query43	534	481	481	481
query44	1242	812	855	812
query45	186	167	174	167
query46	1169	751	727	727
query47	1992	1933	1953	1933
query48	439	338	346	338
query49	942	390	391	390
query50	819	434	432	432
query51	7443	7195	7315	7195
query52	99	91	95	91
query53	256	189	184	184
query54	1217	500	478	478
query55	82	77	79	77
query56	279	270	252	252
query57	1310	1220	1214	1214
query58	221	204	215	204
query59	3139	2930	3136	2930
query60	318	256	271	256
query61	112	114	130	114
query62	860	689	702	689
query63	225	190	195	190
query64	4229	720	657	657
query65	3317	3251	3302	3251
query66	788	315	313	313
query67	16213	15647	15503	15503
query68	4508	587	594	587
query69	450	271	306	271
query70	1180	1073	1102	1073
query71	332	270	250	250
query72	6305	4172	4122	4122
query73	761	350	359	350
query74	9386	8859	8828	8828
query75	3401	2629	2673	2629
query76	2802	1140	1110	1110
query77	387	275	279	275
query78	10583	9607	9480	9480
query79	2386	608	621	608
query80	1189	434	430	430
query81	559	222	226	222
query82	897	91	86	86
query83	230	143	148	143
query84	243	81	83	81
query85	1316	309	293	293
query86	444	292	282	282
query87	4371	4140	4196	4140
query88	4251	2397	2403	2397
query89	419	287	289	287
query90	1915	184	186	184
query91	140	108	109	108
query92	61	53	51	51
query93	2300	561	556	556
query94	771	291	290	290
query95	360	257	255	255
query96	621	290	287	287
query97	3325	3118	3166	3118
query98	214	210	206	206
query99	1548	1316	1294	1294
Total cold run time: 301452 ms
Total hot run time: 195922 ms

doris-robot · 2025-07-10T03:29:44Z

ClickBench: Total hot run time: 31.15 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ae78ba8517f2a51cec8fb5b91fead2e667d10381, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.03	0.03
query3	0.24	0.07	0.07
query4	1.63	0.10	0.10
query5	0.54	0.52	0.52
query6	1.13	0.72	0.72
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.55	0.50	0.50
query10	0.56	0.54	0.55
query11	0.14	0.10	0.10
query12	0.14	0.11	0.12
query13	0.61	0.59	0.60
query14	0.76	0.78	0.79
query15	0.84	0.83	0.82
query16	0.39	0.38	0.38
query17	0.98	1.04	1.03
query18	0.23	0.21	0.21
query19	1.95	1.86	1.84
query20	0.01	0.01	0.02
query21	15.42	0.60	0.59
query22	2.34	2.53	1.94
query23	16.97	0.92	0.82
query24	2.85	1.63	2.44
query25	0.27	0.15	0.14
query26	0.50	0.14	0.14
query27	0.04	0.05	0.05
query28	8.77	0.51	0.46
query29	12.58	3.25	3.24
query30	0.25	0.06	0.06
query31	2.86	0.40	0.38
query32	3.24	0.46	0.46
query33	2.93	3.03	3.02
query34	17.00	4.50	4.52
query35	4.57	4.55	4.57
query36	0.66	0.48	0.46
query37	0.09	0.06	0.05
query38	0.04	0.03	0.04
query39	0.04	0.02	0.02
query40	0.16	0.13	0.12
query41	0.08	0.03	0.02
query42	0.03	0.03	0.02
query43	0.03	0.03	0.03
Total cold run time: 102.59 s
Total hot run time: 31.15 s

morningman and others added 2 commits July 9, 2025 06:42

[fix](test) fix audit_log regression test (apache#52587)

c01a607

### What problem does this PR solve? Related PR: In this PR apache#52032 , the audit_log schema had changed, so the test need to fix

morningman requested a review from morrySnow as a code owner July 9, 2025 13:43

morningman and others added 2 commits July 9, 2025 06:47

[fix](audit) fix invalid audit log column and line separator (apache#…

4f200d7

…52968) followup apache#52032, the separator is wrong

1

af8b517

morningman force-pushed the 31_bp52032 branch from f8ad726 to af8b517 Compare July 9, 2025 13:56

morningman changed the title ~~[opt](audit) use one line in audit log and origin statement in audit table (#52032)~~ branch-3.1: [opt](audit) use one line in audit log and origin statement in audit table (#52032) Jul 9, 2025

morrySnow previously approved these changes Jul 10, 2025

View reviewed changes

Merge branch 'branch-3.1' into 31_bp52032

ae78ba8

morningman dismissed morrySnow’s stale review via ae78ba8 July 10, 2025 02:35

morrySnow approved these changes Jul 10, 2025

View reviewed changes

morrySnow merged commit ec5936c into apache:branch-3.1 Jul 10, 2025
20 of 21 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

branch-3.1: [opt](audit) use one line in audit log and origin statement in audit table (#52032) #53022

branch-3.1: [opt](audit) use one line in audit log and origin statement in audit table (#52032) #53022

Uh oh!

morningman commented Jul 9, 2025 •

edited

Loading

Uh oh!

hello-stephen commented Jul 9, 2025

Uh oh!

morningman commented Jul 9, 2025

Uh oh!

morningman commented Jul 10, 2025

Uh oh!

doris-robot commented Jul 10, 2025

Uh oh!

doris-robot commented Jul 10, 2025

Uh oh!

doris-robot commented Jul 10, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

branch-3.1: [opt](audit) use one line in audit log and origin statement in audit table (#52032) #53022

branch-3.1: [opt](audit) use one line in audit log and origin statement in audit table (#52032) #53022

Uh oh!

Conversation

morningman commented Jul 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hello-stephen commented Jul 9, 2025

Uh oh!

morningman commented Jul 9, 2025

Uh oh!

morningman commented Jul 10, 2025

Uh oh!

doris-robot commented Jul 10, 2025

Uh oh!

doris-robot commented Jul 10, 2025

Uh oh!

doris-robot commented Jul 10, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

morningman commented Jul 9, 2025 •

edited

Loading