Skip to content

branch-3.0:[fix](audit)Fixed an issue that the audit log would record the previo… #54008

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 1 commit into from
Jul 30, 2025
Merged

branch-3.0:[fix](audit)Fixed an issue that the audit log would record the previo… #54008

dataroaring merged 1 commit into from
Jul 30, 2025

Conversation

@zddr
Copy link
Contributor

@zddr zddr commented Jul 29, 2025

…us queryId when parseSQL fails. (#53107)

pick: #53107

@zddr
[fix](audit)Fixed an issue that the audit log would record the previo…
43dcad9 
…us queryId when parseSQL fails. (apache#53107)

### What problem does this PR solve?

Issue:
- When parsing SQL, the queryId has not yet been generated. However, the
audit log retrieves the queryId from ConnectContext, which means it uses
the queryId from the previous execution.
- The queryId stored in queryCache is from the previous query, not the
current one.

Solution:
- Since parsing may produce multiple SQL statements, each should have
its own queryId. Therefore, the queryId should not be set during
parsing. As a compromise, we clear the queryId before parsing, ensuring
that the audit log records an empty queryId if parsing fails.
- Set the queryId when adding the cache, not in the constructor.
# Conflicts:
#	fe/fe-core/src/main/java/org/apache/doris/common/cache/NereidsSqlCacheManager.java
#	fe/fe-core/src/main/java/org/apache/doris/qe/ConnectContext.java
#	fe/fe-core/src/test/java/org/apache/doris/qe/ConnectContextTest.java
@zddr zddr requested a review from dataroaring as a code owner July 29, 2025 03:39
@Thearas
Copy link
Contributor

Thearas commented Jul 29, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@zddr
Copy link
Contributor Author

zddr commented Jul 29, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jul 29, 2025

TPC-H: Total hot run time: 39777 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 43dcad9dfcdbe9d6b809c2a748c4299f84f1c717, data reload: false

------ Round 1 ----------------------------------
q1	17658	7368	6689	6689
q2	2067	177	164	164
q3	10667	1132	1212	1132
q4	10509	773	750	750
q5	7741	2865	2768	2768
q6	219	134	135	134
q7	983	605	604	604
q8	9578	1959	2040	1959
q9	7868	6347	6446	6347
q10	7047	2299	2303	2299
q11	457	260	255	255
q12	398	219	215	215
q13	17820	2999	2999	2999
q14	224	201	206	201
q15	529	474	478	474
q16	497	375	376	375
q17	973	563	535	535
q18	7283	6551	6587	6551
q19	1398	1045	1064	1045
q20	469	201	209	201
q21	3925	3106	3132	3106
q22	1101	986	974	974
Total cold run time: 109411 ms
Total hot run time: 39777 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6653	6719	6655	6655
q2	327	230	228	228
q3	3052	2967	2965	2965
q4	2055	1846	1795	1795
q5	5717	5727	5760	5727
q6	221	133	128	128
q7	2194	1789	1841	1789
q8	3326	3587	3499	3499
q9	8795	8919	8810	8810
q10	3592	3561	3561	3561
q11	579	496	500	496
q12	788	595	615	595
q13	7572	3132	3141	3132
q14	302	268	268	268
q15	527	471	472	471
q16	487	438	438	438
q17	1843	1665	1633	1633
q18	8244	7653	7699	7653
q19	1673	1565	1539	1539
q20	2078	1886	1859	1859
q21	5313	5208	5115	5115
q22	1145	1123	1051	1051
Total cold run time: 66483 ms
Total hot run time: 59407 ms

@doris-robot
Copy link

doris-robot commented Jul 29, 2025

TPC-DS: Total hot run time: 195831 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 43dcad9dfcdbe9d6b809c2a748c4299f84f1c717, data reload: false

query1	1271	900	909	900
query2	6339	1931	1837	1837
query3	10954	4245	4467	4245
query4	33159	23381	23551	23381
query5	3746	472	439	439
query6	270	181	180	180
query7	3996	313	314	313
query8	280	237	221	221
query9	9683	2568	2567	2567
query10	479	257	259	257
query11	18142	15226	15205	15205
query12	175	101	103	101
query13	1567	427	421	421
query14	9325	7456	6781	6781
query15	220	192	188	188
query16	7731	495	494	494
query17	1633	625	592	592
query18	2037	334	323	323
query19	237	170	168	168
query20	134	116	113	113
query21	211	107	106	106
query22	4901	4443	4566	4443
query23	34747	34379	33763	33763
query24	11237	3009	2905	2905
query25	678	441	428	428
query26	1376	176	175	175
query27	2545	351	362	351
query28	7564	2141	2193	2141
query29	932	488	472	472
query30	269	153	160	153
query31	1069	828	855	828
query32	96	59	55	55
query33	769	292	297	292
query34	1177	514	525	514
query35	937	730	726	726
query36	1073	963	933	933
query37	123	76	65	65
query38	4115	3934	3910	3910
query39	1518	1475	1479	1475
query40	212	98	97	97
query41	50	49	48	48
query42	112	104	100	100
query43	531	482	489	482
query44	1305	809	807	807
query45	186	172	168	168
query46	1177	713	716	713
query47	2010	1932	1931	1931
query48	475	377	391	377
query49	959	395	405	395
query50	855	437	439	437
query51	7338	7199	7368	7199
query52	100	96	90	90
query53	265	197	188	188
query54	1273	481	465	465
query55	81	80	78	78
query56	261	245	259	245
query57	1299	1227	1203	1203
query58	228	212	236	212
query59	3264	3017	2923	2923
query60	288	258	296	258
query61	124	106	114	106
query62	870	715	738	715
query63	240	199	197	197
query64	4001	676	645	645
query65	3344	3276	3316	3276
query66	898	295	300	295
query67	15868	15509	15530	15509
query68	4428	593	559	559
query69	431	269	266	266
query70	1131	1118	1082	1082
query71	338	269	261	261
query72	6310	4153	3900	3900
query73	755	346	360	346
query74	10280	9259	9081	9081
query75	3366	2617	2642	2617
query76	2618	1182	1051	1051
query77	396	266	279	266
query78	10555	9596	9477	9477
query79	1758	598	586	586
query80	1173	439	434	434
query81	550	215	218	215
query82	949	88	89	88
query83	235	145	142	142
query84	233	82	80	80
query85	1348	299	295	295
query86	449	275	298	275
query87	4352	4223	4234	4223
query88	3562	2381	2355	2355
query89	415	291	293	291
query90	1877	190	189	189
query91	189	150	145	145
query92	67	50	51	50
query93	2440	550	561	550
query94	825	302	265	265
query95	354	260	258	258
query96	622	284	277	277
query97	3268	3103	3131	3103
query98	223	202	200	200
query99	1507	1297	1284	1284
Total cold run time: 301333 ms
Total hot run time: 195831 ms

@doris-robot
Copy link

doris-robot commented Jul 29, 2025

ClickBench: Total hot run time: 30.3 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 43dcad9dfcdbe9d6b809c2a748c4299f84f1c717, data reload: false

query1	0.03	0.04	0.03
query2	0.07	0.02	0.03
query3	0.23	0.07	0.06
query4	1.63	0.11	0.10
query5	0.52	0.52	0.53
query6	1.13	0.72	0.73
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.57	0.50	0.50
query10	0.55	0.55	0.59
query11	0.14	0.10	0.10
query12	0.14	0.10	0.11
query13	0.60	0.59	0.60
query14	0.79	0.78	0.80
query15	0.84	0.82	0.83
query16	0.39	0.39	0.37
query17	0.96	1.03	1.04
query18	0.23	0.21	0.23
query19	1.95	1.78	1.72
query20	0.01	0.00	0.01
query21	15.40	0.60	0.56
query22	2.38	1.52	1.93
query23	16.97	0.88	0.97
query24	2.86	1.57	1.40
query25	0.32	0.15	0.11
query26	0.36	0.13	0.13
query27	0.04	0.04	0.04
query28	9.76	0.54	0.46
query29	12.65	3.21	3.24
query30	0.24	0.06	0.06
query31	2.85	0.38	0.39
query32	3.22	0.47	0.45
query33	2.96	2.97	3.02
query34	16.93	4.57	4.55
query35	4.55	4.58	4.55
query36	0.67	0.48	0.47
query37	0.08	0.06	0.06
query38	0.04	0.03	0.04
query39	0.04	0.02	0.02
query40	0.16	0.12	0.13
query41	0.08	0.02	0.03
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 103.48 s
Total hot run time: 30.3 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Jul 29, 2025

FE UT Coverage Report

Increment line coverage 45.45% (5/11) 🎉
Increment coverage report
Complete coverage report

dataroaring
dataroaring approved these changes Jul 30, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit d8f734a into apache:branch-3.0 Jul 30, 2025
24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@zddr @Thearas @doris-robot @hello-stephen @dataroaring