Skip to content

Conversation

@luwei16
Copy link
Contributor

@luwei16 luwei16 commented Sep 12, 2025

pick 3.1 #55692

@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@luwei16
Copy link
Contributor Author

luwei16 commented Sep 12, 2025

run buildall

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 12, 2025
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CRITICAL: Buffer Overflow Risk

This code has a critical buffer overflow vulnerability:

std::vector<uint8_t> magic_code_buf;
magic_code_buf.reserve(sizeof(uint64_t));  // ❌ CRITICAL BUG
Slice magic_code(magic_code_buf.data(), sizeof(uint64_t));

Issue: reserve() only allocates capacity but doesn't resize the vector. The vector size remains 0, so data() points to uninitialized memory.

Fix: Use resize() instead:

std::vector<uint8_t> magic_code_buf;
magic_code_buf.resize(sizeof(uint64_t));  // ✅ CORRECT
Slice magic_code(magic_code_buf.data(), sizeof(uint64_t));

This must be fixed before merging as it could cause crashes or undefined behavior.

@dataroaring
Copy link
Contributor

Hardcoded Magic Numbers Need Documentation

File: be/src/http/action/check_encryption_action.cpp line 105

std::vector<uint8_t> answer = {'A', 'B', 'C', 'D', 'E', 'A', 'B', 'C'};

Issue: Magic byte sequence is hardcoded without explanation or named constants.

Problems:

  • Code is unclear - what do these bytes represent?
  • Maintenance difficulty if magic bytes need to change
  • No documentation of the encryption validation protocol

Recommendation: Define as named constants with documentation:

// Magic bytes used to identify encrypted tablet segments
static constexpr std::array<uint8_t, 8> ENCRYPTION_MAGIC_BYTES = 
    {'A', 'B', 'C', 'D', 'E', 'A', 'B', 'C'};
std::vector<uint8_t> answer(ENCRYPTION_MAGIC_BYTES.begin(), ENCRYPTION_MAGIC_BYTES.end());

This improves code readability and maintainability.

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Sep 12, 2025
@dataroaring
Copy link
Contributor

Missing Header Guards

File: be/src/http/action/check_encryption_action.h

Issue: This header file lacks proper include guards, which can cause multiple inclusion problems during compilation.

Risk:

  • Compilation errors in complex build scenarios
  • Potential symbol redefinition issues
  • Non-standard header practices

Fix: Add header guards at the beginning of the file:

#pragma once

// OR traditional guards:
#ifndef BE_SRC_HTTP_ACTION_CHECK_ENCRYPTION_ACTION_H
#define BE_SRC_HTTP_ACTION_CHECK_ENCRYPTION_ACTION_H

// ... existing content ...

#endif // BE_SRC_HTTP_ACTION_CHECK_ENCRYPTION_ACTION_H

Recommend using #pragma once as it's simpler and widely supported by modern compilers.

@dataroaring
Copy link
Contributor

Remove Commented Code Block

File: fe/fe-core/src/main/java/org/apache/doris/planner/external/paimon/PaimonScanNode.java and PropertyAnalyzer.java

Issue: Large blocks of commented-out code are present, particularly around lines 1903-1912 in PropertyAnalyzer.java.

Problems:

  • Makes code harder to read and maintain
  • Suggests incomplete refactoring
  • Version control already maintains code history
  • Unclear whether code should be restored or permanently removed

Recommendation:

  • Remove the commented code blocks entirely
  • If the code needs to be preserved for future reference, document the reasoning in a TODO comment
  • Use git history to reference old implementations if needed

Example:

// Remove these commented blocks:
// if (properties.containsKey(PropertyAnalyzer.PROPERTIES_COLOCATE_WITH)) {
//     ... (multiple lines of commented code)

// If truly needed for future work, replace with:
// TODO: Implement colocate_with property validation for TDE scenarios (see commit ABC123)

@dataroaring
Copy link
Contributor

Question: Privilege Type Consistency

File: be/src/http/action/check_encryption_action.cpp

Issue: The new encryption check endpoint uses TPrivilegeType::ALL, which differs from other admin endpoints that typically use TPrivilegeType::ADMIN.

Code:

Status CheckEncryptionAction::check_tablet_encryption_request(HttpRequest* req, TCheckTabletEncryptionReq& request) {
    // Uses TPrivilegeType::ALL

Questions:

  1. Is TPrivilegeType::ALL the intended privilege level for encryption checks?
  2. Should this be TPrivilegeType::ADMIN to match other administrative endpoints?
  3. What's the security reasoning for requiring ALL privileges vs ADMIN privileges?

Security Consideration:

  • ALL privilege is typically broader than ADMIN
  • Encryption status might be sensitive information that should be restricted

Recommendation: Please clarify the intended privilege level and ensure it aligns with the security model for TDE-related operations.

@hello-stephen
Copy link
Contributor

FE UT Coverage Report

Increment line coverage 1.54% (1/65) 🎉
Increment coverage report
Complete coverage report

@doris-robot
Copy link

TPC-H: Total hot run time: 35601 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0c9e018d6d794fada1cdd95eacd8a940e1eb2805, data reload: false

------ Round 1 ----------------------------------
q1	17608	5235	5083	5083
q2	2005	322	214	214
q3	10273	1293	739	739
q4	10219	1049	531	531
q5	7512	2417	2369	2369
q6	191	173	139	139
q7	942	793	647	647
q8	9357	1344	1220	1220
q9	7072	5154	5175	5154
q10	6985	2407	1975	1975
q11	533	305	279	279
q12	382	371	234	234
q13	17806	3716	3061	3061
q14	239	237	222	222
q15	585	501	496	496
q16	1029	1006	938	938
q17	626	887	366	366
q18	7497	7204	6988	6988
q19	1095	973	595	595
q20	359	359	239	239
q21	3958	3278	3142	3142
q22	1062	1024	970	970
Total cold run time: 107335 ms
Total hot run time: 35601 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5165	5123	5335	5123
q2	253	346	231	231
q3	2208	2713	2271	2271
q4	1410	1813	1337	1337
q5	4458	4610	4541	4541
q6	222	184	139	139
q7	1970	2043	1912	1912
q8	2674	2723	2779	2723
q9	7339	7528	7376	7376
q10	3135	3318	2857	2857
q11	616	527	490	490
q12	714	763	654	654
q13	3617	4039	3317	3317
q14	301	315	288	288
q15	525	496	487	487
q16	1072	1225	1064	1064
q17	1181	1578	1444	1444
q18	8001	7855	7840	7840
q19	869	840	943	840
q20	2013	2133	1851	1851
q21	4819	4260	4298	4260
q22	1094	1028	1005	1005
Total cold run time: 53656 ms
Total hot run time: 52050 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 189363 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0c9e018d6d794fada1cdd95eacd8a940e1eb2805, data reload: false

query1	1050	464	405	405
query2	6582	1727	1691	1691
query3	6745	233	227	227
query4	26583	23630	23296	23296
query5	4441	690	530	530
query6	338	239	227	227
query7	4649	516	312	312
query8	305	261	251	251
query9	8629	2941	2918	2918
query10	505	374	301	301
query11	16125	15076	14896	14896
query12	181	124	132	124
query13	1684	552	469	469
query14	10348	9139	9202	9139
query15	209	201	174	174
query16	7595	665	488	488
query17	1259	770	662	662
query18	2050	440	351	351
query19	251	193	169	169
query20	142	129	123	123
query21	219	136	126	126
query22	4192	4299	4149	4149
query23	33886	33026	33251	33026
query24	8266	2387	2397	2387
query25	586	509	446	446
query26	1239	276	168	168
query27	2734	507	368	368
query28	4404	2253	2239	2239
query29	782	623	499	499
query30	296	220	202	202
query31	917	795	726	726
query32	91	79	79	79
query33	593	404	351	351
query34	795	848	531	531
query35	835	829	737	737
query36	970	1046	930	930
query37	130	114	99	99
query38	3551	3506	3481	3481
query39	1528	1424	1566	1424
query40	229	137	158	137
query41	63	63	62	62
query42	136	122	130	122
query43	528	526	468	468
query44	1334	884	862	862
query45	188	173	180	173
query46	850	1018	656	656
query47	1792	1860	1769	1769
query48	416	434	310	310
query49	755	519	416	416
query50	646	683	400	400
query51	3904	3887	3898	3887
query52	114	118	110	110
query53	261	291	204	204
query54	605	609	545	545
query55	95	95	95	95
query56	371	328	345	328
query57	1193	1211	1145	1145
query58	299	286	285	285
query59	2598	2604	2570	2570
query60	368	357	366	357
query61	196	158	164	158
query62	809	751	695	695
query63	234	198	203	198
query64	4454	1145	846	846
query65	4039	3981	3952	3952
query66	1169	443	346	346
query67	15625	15467	15086	15086
query68	8648	981	589	589
query69	494	328	297	297
query70	1335	1311	1291	1291
query71	577	352	338	338
query72	5936	5001	5133	5001
query73	757	642	360	360
query74	8989	9197	8693	8693
query75	4151	3248	2780	2780
query76	3608	1186	747	747
query77	812	412	347	347
query78	9679	9779	8848	8848
query79	2356	868	596	596
query80	650	590	522	522
query81	484	259	229	229
query82	459	176	138	138
query83	299	259	245	245
query84	311	116	97	97
query85	869	531	434	434
query86	358	307	333	307
query87	3709	3710	3631	3631
query88	3227	2221	2229	2221
query89	411	334	301	301
query90	1902	224	221	221
query91	164	172	140	140
query92	91	76	75	75
query93	1239	981	651	651
query94	698	424	328	328
query95	411	337	326	326
query96	483	568	283	283
query97	2976	2995	2905	2905
query98	242	222	219	219
query99	1550	1407	1300	1300
Total cold run time: 276831 ms
Total hot run time: 189363 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.41 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0c9e018d6d794fada1cdd95eacd8a940e1eb2805, data reload: false

query1	0.06	0.05	0.05
query2	0.09	0.05	0.05
query3	0.25	0.08	0.08
query4	1.61	0.11	0.11
query5	0.27	0.27	0.25
query6	1.18	0.66	0.64
query7	0.03	0.03	0.02
query8	0.06	0.04	0.05
query9	0.60	0.54	0.52
query10	0.58	0.58	0.59
query11	0.17	0.11	0.11
query12	0.16	0.12	0.12
query13	0.63	0.63	0.63
query14	1.01	1.06	1.03
query15	0.86	0.86	0.88
query16	0.40	0.39	0.40
query17	1.06	1.05	1.02
query18	0.22	0.20	0.20
query19	1.93	1.87	1.82
query20	0.02	0.01	0.02
query21	15.41	0.95	0.58
query22	0.76	1.12	0.97
query23	14.72	1.40	0.64
query24	6.75	1.03	1.35
query25	0.49	0.12	0.09
query26	0.64	0.18	0.15
query27	0.06	0.06	0.05
query28	9.98	0.92	0.44
query29	12.60	3.96	3.23
query30	0.29	0.13	0.11
query31	2.84	0.58	0.39
query32	3.24	0.59	0.48
query33	3.13	3.05	3.06
query34	15.92	5.45	4.89
query35	4.95	4.90	4.88
query36	0.70	0.54	0.50
query37	0.10	0.08	0.08
query38	0.07	0.05	0.05
query39	0.04	0.04	0.03
query40	0.17	0.17	0.15
query41	0.09	0.04	0.03
query42	0.04	0.04	0.03
query43	0.04	0.04	0.03
Total cold run time: 104.22 s
Total hot run time: 30.41 s

@doris-robot
Copy link

BE UT Coverage Report

Increment line coverage 17.11% (26/152) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 52.20% (17427/33387)
Line Coverage 37.42% (158365/423259)
Region Coverage 31.98% (120734/377499)
Branch Coverage 33.36% (52992/158832)

@luwei16
Copy link
Contributor Author

luwei16 commented Sep 12, 2025

run cloud_p0

@luwei16
Copy link
Contributor Author

luwei16 commented Sep 12, 2025

run nonConcurrent

@luwei16
Copy link
Contributor Author

luwei16 commented Sep 12, 2025

run p0

@hello-stephen
Copy link
Contributor

BE Regression && UT Coverage Report

Increment line coverage 23.03% (35/152) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 70.67% (23072/32648)
Line Coverage 57.00% (240686/422247)
Region Coverage 52.50% (200793/382492)
Branch Coverage 54.11% (86325/159533)

@hello-stephen
Copy link
Contributor

FE Regression Coverage Report

Increment line coverage 1.54% (1/65) 🎉
Increment coverage report
Complete coverage report

@hello-stephen
Copy link
Contributor

BE Regression && UT Coverage Report

Increment line coverage 23.03% (35/152) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 70.68% (23076/32648)
Line Coverage 57.02% (240764/422247)
Region Coverage 52.51% (200853/382492)
Branch Coverage 54.12% (86335/159533)

1 similar comment
@hello-stephen
Copy link
Contributor

BE Regression && UT Coverage Report

Increment line coverage 23.03% (35/152) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 70.68% (23076/32648)
Line Coverage 57.02% (240764/422247)
Region Coverage 52.51% (200853/382492)
Branch Coverage 54.12% (86335/159533)

@hello-stephen
Copy link
Contributor

FE Regression Coverage Report

Increment line coverage 1.54% (1/65) 🎉
Increment coverage report
Complete coverage report

1 similar comment
@hello-stephen
Copy link
Contributor

FE Regression Coverage Report

Increment line coverage 1.54% (1/65) 🎉
Increment coverage report
Complete coverage report

@hello-stephen
Copy link
Contributor

BE Regression && UT Coverage Report

Increment line coverage 23.03% (35/152) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 70.68% (23077/32648)
Line Coverage 57.02% (240761/422247)
Region Coverage 52.50% (200826/382492)
Branch Coverage 54.11% (86329/159533)

@luwei16
Copy link
Contributor Author

luwei16 commented Sep 12, 2025

run cloud_p0

@hello-stephen
Copy link
Contributor

FE Regression Coverage Report

Increment line coverage 1.54% (1/65) 🎉
Increment coverage report
Complete coverage report

@hello-stephen
Copy link
Contributor

BE Regression && UT Coverage Report

Increment line coverage 23.03% (35/152) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 70.68% (23077/32648)
Line Coverage 57.02% (240761/422247)
Region Coverage 52.50% (200826/382492)
Branch Coverage 54.11% (86329/159533)

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 14, 2025
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@gavinchou gavinchou merged commit 1bee89a into apache:master Sep 14, 2025
30 of 36 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants