[opt](audit) use one line in audit log and origin statement in audit table #52032

morningman · 2025-06-20T05:04:18Z

What problem does this PR solve?

Previously, when auditing, we use replace all \n, \t in origin sql string with \\n, \\t,
so that the sql string can be written in one line. But this lead to some problem:

User can not direct use the sql in audit log to execute.
Some replacement is wrong, eg, replace the \n in a quota string.

This PR changes the logic:

For audit log, only replace \n with \\n to keep SQL in one line.
For audit table, keep the origin string.
Use special column and line separator for audit log load data, to avoid conflict with char in SQL

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes. [feat] modify audit log and profile http api doris-website#2516

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

Thearas · 2025-06-20T05:04:24Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

…table

morningman · 2025-06-20T07:44:11Z

run buildall

doris-robot · 2025-06-20T08:29:42Z

TPC-H: Total hot run time: 34577 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 72fdf8d3794b06b334aaa0e23a15bff9dcd8f0bb, data reload: false

------ Round 1 ----------------------------------
q1	17587	5218	5040	5040
q2	1927	280	199	199
q3	10308	1340	708	708
q4	10219	1007	504	504
q5	7503	2453	2293	2293
q6	195	173	133	133
q7	897	736	593	593
q8	9305	1325	1169	1169
q9	6819	5124	5158	5124
q10	6914	2381	1957	1957
q11	492	290	279	279
q12	342	345	209	209
q13	17791	3837	3642	3642
q14	221	231	221	221
q15	573	484	480	480
q16	430	427	363	363
q17	617	876	385	385
q18	7726	7161	7281	7161
q19	1730	967	570	570
q20	340	343	228	228
q21	3677	3169	2350	2350
q22	1072	1024	969	969
Total cold run time: 106685 ms
Total hot run time: 34577 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5161	5071	5094	5071
q2	254	321	226	226
q3	2151	2637	2297	2297
q4	1327	1813	1400	1400
q5	4216	4138	4452	4138
q6	219	171	127	127
q7	2021	1920	1789	1789
q8	2646	2565	2582	2565
q9	7158	7175	7129	7129
q10	3057	3296	2828	2828
q11	576	496	483	483
q12	666	757	612	612
q13	3473	3869	3168	3168
q14	273	297	265	265
q15	524	470	465	465
q16	425	478	456	456
q17	1184	1569	1335	1335
q18	7685	7483	7345	7345
q19	799	800	923	800
q20	1966	2036	1864	1864
q21	4938	4453	4413	4413
q22	1044	1014	983	983
Total cold run time: 51763 ms
Total hot run time: 49759 ms

doris-robot · 2025-06-20T08:41:14Z

TPC-DS: Total hot run time: 185993 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 72fdf8d3794b06b334aaa0e23a15bff9dcd8f0bb, data reload: false

query1	998	396	394	394
query2	6523	1808	1840	1808
query3	6754	220	219	219
query4	26546	23790	23472	23472
query5	4350	609	462	462
query6	315	223	221	221
query7	4642	504	286	286
query8	266	222	216	216
query9	8608	2641	2642	2641
query10	503	344	283	283
query11	15506	14991	14804	14804
query12	157	111	113	111
query13	1644	516	412	412
query14	9149	6043	6210	6043
query15	188	187	174	174
query16	7171	606	466	466
query17	1172	696	570	570
query18	1983	402	303	303
query19	204	184	162	162
query20	118	113	110	110
query21	213	125	105	105
query22	4033	4241	4192	4192
query23	33929	33113	33058	33058
query24	8514	2351	2366	2351
query25	555	485	381	381
query26	1234	266	148	148
query27	2781	508	340	340
query28	4325	2122	2109	2109
query29	758	541	428	428
query30	283	217	196	196
query31	924	837	762	762
query32	79	65	63	63
query33	555	350	305	305
query34	803	859	525	525
query35	783	841	709	709
query36	944	963	850	850
query37	106	101	83	83
query38	4132	4091	4023	4023
query39	1449	1454	1400	1400
query40	219	144	106	106
query41	63	61	61	61
query42	121	105	107	105
query43	481	498	475	475
query44	1281	831	813	813
query45	174	169	161	161
query46	840	1018	627	627
query47	1770	1809	1741	1741
query48	391	425	313	313
query49	730	478	395	395
query50	635	671	406	406
query51	4071	4077	4118	4077
query52	112	105	99	99
query53	221	252	189	189
query54	567	583	504	504
query55	86	83	83	83
query56	338	310	279	279
query57	1192	1185	1130	1130
query58	262	270	259	259
query59	2537	2602	2633	2602
query60	326	329	310	310
query61	152	152	153	152
query62	808	729	682	682
query63	232	192	191	191
query64	4417	1079	767	767
query65	4218	4146	4196	4146
query66	1149	427	386	386
query67	15697	15552	15374	15374
query68	8746	877	538	538
query69	475	305	266	266
query70	1175	1134	1085	1085
query71	464	327	302	302
query72	5649	4649	4689	4649
query73	726	603	351	351
query74	8851	9253	9036	9036
query75	4120	3186	2704	2704
query76	3603	1182	749	749
query77	797	394	285	285
query78	10043	10290	9399	9399
query79	1916	790	579	579
query80	618	497	431	431
query81	471	263	219	219
query82	450	129	101	101
query83	284	249	229	229
query84	284	168	99	99
query85	781	351	314	314
query86	325	296	282	282
query87	4370	4409	4282	4282
query88	2839	2275	2318	2275
query89	391	309	287	287
query90	1913	201	203	201
query91	142	142	113	113
query92	72	60	56	56
query93	1112	928	595	595
query94	674	422	299	299
query95	374	292	283	283
query96	483	572	281	281
query97	2750	2771	2634	2634
query98	221	206	199	199
query99	1441	1378	1252	1252
Total cold run time: 272847 ms
Total hot run time: 185993 ms

doris-robot · 2025-06-20T08:46:26Z

ClickBench: Total hot run time: 29.25 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 72fdf8d3794b06b334aaa0e23a15bff9dcd8f0bb, data reload: false

query1	0.04	0.04	0.02
query2	0.07	0.04	0.03
query3	0.24	0.06	0.07
query4	1.61	0.10	0.11
query5	0.41	0.44	0.42
query6	1.16	0.66	0.67
query7	0.02	0.02	0.02
query8	0.05	0.03	0.03
query9	0.59	0.52	0.51
query10	0.56	0.58	0.56
query11	0.16	0.11	0.11
query12	0.14	0.12	0.12
query13	0.62	0.61	0.61
query14	0.80	0.80	0.81
query15	0.90	0.86	0.86
query16	0.39	0.39	0.40
query17	1.06	1.06	1.05
query18	0.22	0.21	0.21
query19	1.95	1.87	1.80
query20	0.01	0.01	0.01
query21	15.43	0.88	0.53
query22	0.76	1.22	0.72
query23	14.80	1.33	0.62
query24	7.19	1.20	0.62
query25	0.50	0.21	0.08
query26	0.58	0.16	0.15
query27	0.06	0.05	0.04
query28	9.55	0.89	0.44
query29	12.55	3.97	3.30
query30	0.25	0.09	0.06
query31	2.83	0.60	0.39
query32	3.27	0.55	0.48
query33	3.06	3.02	3.07
query34	16.11	5.36	4.79
query35	4.85	4.80	4.79
query36	0.68	0.52	0.51
query37	0.09	0.06	0.07
query38	0.05	0.04	0.04
query39	0.03	0.03	0.03
query40	0.18	0.14	0.13
query41	0.08	0.03	0.03
query42	0.03	0.02	0.02
query43	0.04	0.04	0.03
Total cold run time: 103.97 s
Total hot run time: 29.25 s

hello-stephen · 2025-06-20T09:09:07Z

FE UT Coverage Report

Increment line coverage 51.11% (46/90) 🎉
Increment coverage report
Complete coverage report

morningman · 2025-06-20T13:33:50Z

run buildall

github-actions · 2025-06-20T13:42:27Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-06-20T13:42:29Z

PR approved by anyone and no changes requested.

doris-robot · 2025-06-20T13:55:00Z

TPC-H: Total hot run time: 34266 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 37f0c2d5fbbb3093845585fd5d9013c620bac663, data reload: false

------ Round 1 ----------------------------------
q1	17601	5263	5159	5159
q2	1952	294	192	192
q3	10324	1286	723	723
q4	10244	1018	532	532
q5	7678	3096	2298	2298
q6	192	164	134	134
q7	919	746	614	614
q8	9340	1286	1140	1140
q9	6806	5091	5064	5064
q10	6952	2359	1970	1970
q11	490	309	279	279
q12	341	352	217	217
q13	17807	3673	3172	3172
q14	229	223	224	223
q15	576	480	496	480
q16	441	425	376	376
q17	645	893	383	383
q18	7967	7337	7181	7181
q19	1897	970	557	557
q20	327	327	224	224
q21	3968	3167	2389	2389
q22	1110	1029	959	959
Total cold run time: 107806 ms
Total hot run time: 34266 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5303	5119	5170	5119
q2	244	318	228	228
q3	2181	2687	2341	2341
q4	1384	1781	1297	1297
q5	4231	4322	4386	4322
q6	220	172	131	131
q7	2040	2004	1764	1764
q8	2610	2608	2555	2555
q9	7148	7092	7173	7092
q10	3132	3222	2855	2855
q11	586	517	506	506
q12	693	794	640	640
q13	3486	3975	3386	3386
q14	285	293	272	272
q15	546	480	473	473
q16	462	491	445	445
q17	1161	1582	1405	1405
q18	7418	7191	7064	7064
q19	779	776	824	776
q20	1909	1993	1810	1810
q21	4755	4344	4322	4322
q22	1071	1032	961	961
Total cold run time: 51644 ms
Total hot run time: 49764 ms

doris-robot · 2025-06-20T14:06:35Z

TPC-DS: Total hot run time: 186068 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 37f0c2d5fbbb3093845585fd5d9013c620bac663, data reload: false

query1	992	400	408	400
query2	6549	1775	1810	1775
query3	6743	224	239	224
query4	27154	23367	23302	23302
query5	4382	646	464	464
query6	312	221	214	214
query7	4632	491	288	288
query8	262	241	212	212
query9	8632	2615	2655	2615
query10	451	332	270	270
query11	15681	15546	15112	15112
query12	168	114	106	106
query13	1661	541	430	430
query14	9426	6219	6172	6172
query15	214	193	179	179
query16	7555	618	463	463
query17	1208	724	584	584
query18	2014	406	316	316
query19	200	208	160	160
query20	123	116	118	116
query21	220	136	106	106
query22	4214	4067	3988	3988
query23	34078	33357	33126	33126
query24	8429	2354	2428	2354
query25	518	460	377	377
query26	1226	269	148	148
query27	2746	515	342	342
query28	4303	2141	2097	2097
query29	752	555	424	424
query30	289	223	194	194
query31	937	839	772	772
query32	77	67	65	65
query33	560	359	344	344
query34	794	860	546	546
query35	789	816	740	740
query36	956	988	879	879
query37	115	96	76	76
query38	4134	4098	4106	4098
query39	1568	1423	1398	1398
query40	215	116	108	108
query41	64	62	59	59
query42	124	108	109	108
query43	485	502	463	463
query44	1308	814	815	814
query45	191	176	173	173
query46	837	1026	653	653
query47	1713	1790	1709	1709
query48	380	426	313	313
query49	745	491	402	402
query50	638	674	409	409
query51	4160	4215	4147	4147
query52	113	117	102	102
query53	228	253	183	183
query54	579	576	509	509
query55	86	84	87	84
query56	286	309	300	300
query57	1151	1200	1123	1123
query58	264	260	277	260
query59	2572	2587	2515	2515
query60	333	328	299	299
query61	127	126	124	124
query62	781	720	654	654
query63	230	187	196	187
query64	4287	998	664	664
query65	4269	4151	4202	4151
query66	1180	406	319	319
query67	15715	15405	15387	15387
query68	8548	899	544	544
query69	470	307	265	265
query70	1114	1065	1085	1065
query71	465	369	308	308
query72	5773	4665	4647	4647
query73	709	585	353	353
query74	8852	8865	8964	8865
query75	3923	3197	2728	2728
query76	3752	1191	753	753
query77	778	387	308	308
query78	9928	10250	9457	9457
query79	2484	819	571	571
query80	634	519	439	439
query81	476	269	221	221
query82	438	133	101	101
query83	281	245	233	233
query84	290	119	84	84
query85	800	351	309	309
query86	329	302	283	283
query87	4479	4509	4337	4337
query88	2960	2233	2243	2233
query89	393	310	294	294
query90	1947	204	202	202
query91	143	155	113	113
query92	77	61	58	58
query93	1206	939	592	592
query94	680	422	304	304
query95	369	301	283	283
query96	501	564	281	281
query97	2816	2759	2640	2640
query98	234	217	200	200
query99	1434	1390	1302	1302
Total cold run time: 275408 ms
Total hot run time: 186068 ms

doris-robot · 2025-06-20T14:11:48Z

ClickBench: Total hot run time: 29.29 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 37f0c2d5fbbb3093845585fd5d9013c620bac663, data reload: false

query1	0.03	0.03	0.03
query2	0.08	0.03	0.04
query3	0.23	0.07	0.06
query4	1.60	0.11	0.11
query5	0.44	0.43	0.44
query6	1.19	0.66	0.66
query7	0.02	0.02	0.02
query8	0.04	0.04	0.04
query9	0.58	0.52	0.52
query10	0.57	0.59	0.57
query11	0.16	0.11	0.10
query12	0.14	0.12	0.12
query13	0.63	0.61	0.60
query14	0.80	0.81	0.82
query15	0.89	0.87	0.86
query16	0.39	0.39	0.39
query17	1.12	1.06	1.06
query18	0.23	0.21	0.21
query19	1.96	1.92	1.85
query20	0.02	0.01	0.02
query21	15.40	0.90	0.53
query22	0.77	1.15	0.75
query23	14.81	1.38	0.65
query24	8.08	1.35	0.36
query25	0.50	0.21	0.23
query26	0.68	0.17	0.14
query27	0.06	0.04	0.04
query28	9.16	0.89	0.45
query29	12.60	3.95	3.30
query30	0.26	0.09	0.06
query31	2.84	0.60	0.40
query32	3.24	0.55	0.46
query33	3.06	3.12	3.17
query34	15.96	5.37	4.75
query35	4.82	4.77	4.80
query36	0.69	0.50	0.49
query37	0.08	0.07	0.07
query38	0.05	0.04	0.03
query39	0.03	0.02	0.02
query40	0.17	0.14	0.14
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.04	0.04	0.03
Total cold run time: 104.53 s
Total hot run time: 29.29 s

hello-stephen · 2025-06-20T14:35:25Z

FE UT Coverage Report

Increment line coverage 51.11% (46/90) 🎉
Increment coverage report
Complete coverage report

…table (apache#52032) Previously, when auditing, we use replace all `\n`, `\t` in origin sql string with `\\n`, `\\t`, so that the sql string can be written in one line. But this lead to some problem: 1. User can not direct use the sql in audit log to execute. 2. Some replacement is wrong, eg, replace the `\n` in a quota string. This PR changes the logic: 1. For audit log, only replace `\n` with `\\n` to keep SQL in one line. 2. For audit table, keep the origin string. 3. Use special column and line separator for audit log load data, to avoid conflict with char in SQL

…table (#52032) (#52205) bp #52032

### What problem does this PR solve? Related PR: In this PR #52032 , the audit_log schema had changed, so the test need to fix

followup #52032, the separator is wrong

…table (apache#52032) Previously, when auditing, we use replace all `\n`, `\t` in origin sql string with `\\n`, `\\t`, so that the sql string can be written in one line. But this lead to some problem: 1. User can not direct use the sql in audit log to execute. 2. Some replacement is wrong, eg, replace the `\n` in a quota string. This PR changes the logic: 1. For audit log, only replace `\n` with `\\n` to keep SQL in one line. 2. For audit table, keep the origin string. 3. Use special column and line separator for audit log load data, to avoid conflict with char in SQL

### What problem does this PR solve? Related PR: In this PR apache#52032 , the audit_log schema had changed, so the test need to fix

…52968) followup apache#52032, the separator is wrong

…nt in audit table #52032 #52587 #52968 (#53022) bp --------- Co-authored-by: yagagagaga <zhangminke@selectdb.com>

morningman added the kind/behavior-changed label Jun 20, 2025

morningman force-pushed the fix_audit_format branch from 98b06c5 to 6735857 Compare June 20, 2025 07:01

morningman added 3 commits June 20, 2025 15:36

[fix](audit) use one line in audit log and origin statement in audit …

8231808

…table

2

bc0abc3

3

72fdf8d

morningman force-pushed the fix_audit_format branch from 06fd181 to 72fdf8d Compare June 20, 2025 07:36

1

37f0c2d

morrySnow approved these changes Jun 20, 2025

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Jun 20, 2025

github-actions bot added the reviewed label Jun 20, 2025

CalvinKirs approved these changes Jun 20, 2025

View reviewed changes

morningman merged commit 890d11d into apache:master Jun 20, 2025
24 of 27 checks passed

morningman added the dev/3.0.x label Jun 20, 2025

github-actions bot added the dev/3.0.x-conflict label Jun 20, 2025

morningman mentioned this pull request Jun 24, 2025

[opt](audit) use one line in audit log and origin statement in audit table (#52032) #52205

Merged

morningman added a commit that referenced this pull request Jun 24, 2025

[opt](audit) use one line in audit log and origin statement in audit …

cf68a64

…table (#52032) (#52205) bp #52032

morningman added dev/3.0.7-merged and removed dev/3.0.x dev/3.0.x-conflict labels Jun 24, 2025

morrySnow added the dev/3.1.x label Jun 24, 2025

github-actions bot added the dev/3.1.x-conflict label Jun 24, 2025

yagagagaga mentioned this pull request Jul 1, 2025

[fix](test) fix audit_log regression test #52587

Merged

16 tasks

morningman added dev/3.0.x-experimental and removed dev/3.0.x-experimental labels Jul 4, 2025

hello-stephen pushed a commit that referenced this pull request Jul 7, 2025

[fix](test) fix audit_log regression test (#52587)

581872a

### What problem does this PR solve? Related PR: In this PR #52032 , the audit_log schema had changed, so the test need to fix

morningman mentioned this pull request Jul 8, 2025

[fix](audit) fix invalid audit log column and line separator #52968

Merged

morningman added a commit that referenced this pull request Jul 9, 2025

[fix](audit) fix invalid audit log column and line separator (#52968)

fa3b5dd

followup #52032, the separator is wrong

morningman pushed a commit to morningman/doris that referenced this pull request Jul 9, 2025

[fix](test) fix audit_log regression test (apache#52587)

c01a607

### What problem does this PR solve? Related PR: In this PR apache#52032 , the audit_log schema had changed, so the test need to fix

morningman mentioned this pull request Jul 9, 2025

branch-3.1: [opt](audit) use one line in audit log and origin statement in audit table (#52032) #53022

Merged

morningman added a commit to morningman/doris that referenced this pull request Jul 9, 2025

[fix](audit) fix invalid audit log column and line separator (apache#…

4f200d7

…52968) followup apache#52032, the separator is wrong

morrySnow pushed a commit that referenced this pull request Jul 10, 2025

branch-3.1: [opt](audit) use one line in audit log and origin stateme…

ec5936c

…nt in audit table #52032 #52587 #52968 (#53022) bp --------- Co-authored-by: yagagagaga <zhangminke@selectdb.com>

morrySnow added dev/3.1.0-merged and removed dev/3.1.x dev/3.1.x-conflict labels Jul 10, 2025

[opt](audit) use one line in audit log and origin statement in audit table #52032

[opt](audit) use one line in audit log and origin statement in audit table #52032

Uh oh!

Conversation

morningman commented Jun 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

Thearas commented Jun 20, 2025

Uh oh!

morningman commented Jun 20, 2025

Uh oh!

doris-robot commented Jun 20, 2025

Uh oh!

doris-robot commented Jun 20, 2025

Uh oh!

doris-robot commented Jun 20, 2025

Uh oh!

hello-stephen commented Jun 20, 2025

FE UT Coverage Report

Uh oh!

morningman commented Jun 20, 2025

Uh oh!

github-actions bot commented Jun 20, 2025

Uh oh!

github-actions bot commented Jun 20, 2025

Uh oh!

doris-robot commented Jun 20, 2025

Uh oh!

doris-robot commented Jun 20, 2025

Uh oh!

doris-robot commented Jun 20, 2025

Uh oh!

hello-stephen commented Jun 20, 2025

FE UT Coverage Report

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

morningman commented Jun 20, 2025 •

edited

Loading