branch-3.1: [refactor](plugin) unify the plugin dir #45936 #52921

morningman · 2025-07-08T05:42:02Z

This pull request primarily updates the directory structure for plugins and JDBC drivers, ensuring compatibility with previous versions. Additionally, it includes changes to various configuration files and scripts to reflect these updates. * [`be/src/common/config.cpp`](diffhunk://#diff-b626e6ab16bc72abf40db76bf5094fcc8ca3c37534c2eb83b63b7805e1b601ffL54-R54): Changed the default directory for `jdbc_drivers_dir` and `trino_connector_plugin_dir` to `${DORIS_HOME}/plugins/*` for better organization. [[1]](diffhunk://#diff-b626e6ab16bc72abf40db76bf5094fcc8ca3c37534c2eb83b63b7805e1b601ffL54-R54) [[2]](diffhunk://#diff-b626e6ab16bc72abf40db76bf5094fcc8ca3c37534c2eb83b63b7805e1b601ffL1288-R1288) * [`build.sh`](diffhunk://#diff-4d2a8eefdf2a9783512a35da4dc7676a66404b6f3826a8af9aad038722da6823L755-R759): Updated the script to create new plugin directories under `${DORIS_OUTPUT}/fe/plugins/` and `${DORIS_OUTPUT}/be/plugins/`. [[1]](diffhunk://#diff-4d2a8eefdf2a9783512a35da4dc7676a66404b6f3826a8af9aad038722da6823L755-R759) [[2]](diffhunk://#diff-4d2a8eefdf2a9783512a35da4dc7676a66404b6f3826a8af9aad038722da6823L882-R890) * `be/src/runtime/user_function_cache.cpp` and `be/src/runtime/user_function_cache.h`: Introduced `_check_and_return_default_driver_url` method to maintain compatibility with the old default directory for JDBC drivers. [[1]](diffhunk://#diff-08d48161cad0bb4c6ec31c154c3017407caaff2402f0feb599f014f2d87ae9edL328-R354) [[2]](diffhunk://#diff-0cfb7bc4e6df1282c7dc7f9ec6677ffc9a03ed6cd08f4fea11ba43bcc297f0c3R79-R80) * [`fe/be-java-extensions/trino-connector-scanner/src/main/java/org/apache/doris/trinoconnector/TrinoConnectorPluginLoader.java`](diffhunk://#diff-19838c8c525ec05716e434f18d1ef270dfa03029385258c53b8fbe7da5ba25b9L43-R43): Added `checkAndReturnPluginDir` method to ensure compatibility with the old plugin directory. [[1]](diffhunk://#diff-19838c8c525ec05716e434f18d1ef270dfa03029385258c53b8fbe7da5ba25b9L43-R43) [[2]](diffhunk://#diff-19838c8c525ec05716e434f18d1ef270dfa03029385258c53b8fbe7da5ba25b9L75-R84) [[3]](diffhunk://#diff-19838c8c525ec05716e434f18d1ef270dfa03029385258c53b8fbe7da5ba25b9R94-R116) * [`fe/fe-common/src/main/java/org/apache/doris/common/Config.java`](diffhunk://#diff-ed7dee981e5322b45ed92fbc3c9e049191574baa990894ae36a97567ce198c74L144-R144): Updated various configuration paths to use `EnvUtils.getDorisHome()` instead of `System.getenv("DORIS_HOME")` for consistency and maintainability. [[1]](diffhunk://#diff-ed7dee981e5322b45ed92fbc3c9e049191574baa990894ae36a97567ce198c74L144-R144) [[2]](diffhunk://#diff-ed7dee981e5322b45ed92fbc3c9e049191574baa990894ae36a97567ce198c74L153-R153) [[3]](diffhunk://#diff-ed7dee981e5322b45ed92fbc3c9e049191574baa990894ae36a97567ce198c74L204-R207) [[4]](diffhunk://#diff-ed7dee981e5322b45ed92fbc3c9e049191574baa990894ae36a97567ce198c74L358-R358) [[5]](diffhunk://#diff-ed7dee981e5322b45ed92fbc3c9e049191574baa990894ae36a97567ce198c74L630-R630) [[6]](diffhunk://#diff-ed7dee981e5322b45ed92fbc3c9e049191574baa990894ae36a97567ce198c74L639-R642) [[7]](diffhunk://#diff-ed7dee981e5322b45ed92fbc3c9e049191574baa990894ae36a97567ce198c74L1281-R1281) [[8]](diffhunk://#diff-ed7dee981e5322b45ed92fbc3c9e049191574baa990894ae36a97567ce198c74L2364-R2371) [[9]](diffhunk://#diff-ed7dee981e5322b45ed92fbc3c9e049191574baa990894ae36a97567ce198c74L2903-R2903) * [`fe/fe-core/src/main/java/org/apache/doris/analysis/CreateFunctionStmt.java`](diffhunk://#diff-1481395f49d6c346e93c01ca884ae2b7e65c85247d02cc91a20cd48b98bdf349L271-L273): Removed commented-out code for better readability. * [`fe/fe-core/src/main/java/org/apache/doris/catalog/JdbcResource.java`](diffhunk://#diff-24260bdb6ba0b3ac0f5c719e5c48c3ea70d13405af93670adc8692de523730cfR38): Refactored `getFullDriverUrl` method to use `checkAndReturnDefaultDriverUrl` for compatibility with the old JDBC drivers directory. [[1]](diffhunk://#diff-24260bdb6ba0b3ac0f5c719e5c48c3ea70d13405af93670adc8692de523730cfR38) [[2]](diffhunk://#diff-24260bdb6ba0b3ac0f5c719e5c48c3ea70d13405af93670adc8692de523730cfL296-R297) [[3]](diffhunk://#diff-24260bdb6ba0b3ac0f5c719e5c48c3ea70d13405af93670adc8692de523730cfR316-R336) There are following kinds of plugins in Doris: 1. jdbc drivers 2. java udf 3. trino connectors 4. hadoop conf 5. other custom jars After this PR, all these plugins will be located in `${DORIS_HOME}/plugins/`. ``` ${DORIS_HOME}/plugins/jdbc_drivers/ ${DORIS_HOME}/plugins/java_udf/ ${DORIS_HOME}/plugins/connectors/ ${DORIS_HOME}/plugins/hadoop_conf/ ``` I tested: 1. Put jars in old dir 2. Put jars in new dir 3. Put jars in custom dir and set dir in fe/be conf file All works well.

hello-stephen · 2025-07-08T05:42:06Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

morningman · 2025-07-08T05:47:05Z

run buildall

doris-robot · 2025-07-08T06:25:00Z

TPC-H: Total hot run time: 40036 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3db59ca0a96a01cc943b34e6e8052e1f83e36bd1, data reload: false

------ Round 1 ----------------------------------
q1	17654	6764	6661	6661
q2	2074	224	164	164
q3	10599	1139	1195	1139
q4	10430	708	747	708
q5	7739	2844	2807	2807
q6	214	134	141	134
q7	995	643	623	623
q8	9361	1951	1980	1951
q9	6559	6385	6399	6385
q10	7039	2268	2245	2245
q11	464	266	274	266
q12	435	220	210	210
q13	17804	2997	3028	2997
q14	247	219	211	211
q15	498	464	462	462
q16	483	387	382	382
q17	993	554	555	554
q18	7181	6780	6790	6780
q19	1643	996	1023	996
q20	485	208	220	208
q21	4006	3310	3155	3155
q22	1166	1025	998	998
Total cold run time: 108069 ms
Total hot run time: 40036 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6662	6656	6629	6629
q2	337	237	243	237
q3	2981	3007	3043	3007
q4	2075	1936	1915	1915
q5	5706	5754	5716	5716
q6	211	129	129	129
q7	2217	1795	1791	1791
q8	3371	3524	3560	3524
q9	8728	8886	8819	8819
q10	3536	3500	3535	3500
q11	601	502	505	502
q12	824	607	593	593
q13	6973	3112	3259	3112
q14	304	262	266	262
q15	525	471	471	471
q16	502	462	453	453
q17	1878	1651	1606	1606
q18	8252	7929	7750	7750
q19	1705	1684	1556	1556
q20	2069	1820	1823	1820
q21	5276	5144	4965	4965
q22	1133	1020	1065	1020
Total cold run time: 65866 ms
Total hot run time: 59377 ms

doris-robot · 2025-07-08T06:37:16Z

TPC-DS: Total hot run time: 196705 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3db59ca0a96a01cc943b34e6e8052e1f83e36bd1, data reload: false

query1	1278	904	914	904
query2	6312	1906	1874	1874
query3	10791	4428	4261	4261
query4	32820	23449	23071	23071
query5	3590	480	480	480
query6	267	188	177	177
query7	3993	324	339	324
query8	301	237	229	229
query9	9642	2618	2611	2611
query10	461	276	283	276
query11	17799	15058	15198	15058
query12	168	100	109	100
query13	1554	453	425	425
query14	9853	6662	7266	6662
query15	257	181	185	181
query16	8052	468	482	468
query17	1596	591	599	591
query18	2173	333	322	322
query19	263	165	174	165
query20	131	125	120	120
query21	209	108	110	108
query22	4801	4386	4346	4346
query23	34590	34214	34019	34019
query24	11226	2965	2962	2962
query25	663	431	433	431
query26	1283	180	176	176
query27	2451	375	369	369
query28	7513	2193	2204	2193
query29	896	461	467	461
query30	274	165	163	163
query31	1019	847	847	847
query32	97	60	62	60
query33	770	317	327	317
query34	943	524	553	524
query35	885	778	771	771
query36	1108	943	946	943
query37	126	67	72	67
query38	4200	4017	4036	4017
query39	1553	1494	1518	1494
query40	217	112	108	108
query41	54	54	49	49
query42	116	104	104	104
query43	533	496	487	487
query44	1267	830	820	820
query45	194	171	172	171
query46	1165	737	756	737
query47	1998	1917	1890	1890
query48	438	344	331	331
query49	904	410	400	400
query50	838	444	443	443
query51	7525	7279	7296	7279
query52	102	92	89	89
query53	265	183	194	183
query54	1198	466	476	466
query55	78	82	79	79
query56	273	258	271	258
query57	1310	1201	1195	1195
query58	249	221	230	221
query59	3202	3056	3061	3056
query60	294	261	262	261
query61	114	120	114	114
query62	857	706	730	706
query63	236	205	196	196
query64	4115	709	650	650
query65	3389	3319	3307	3307
query66	782	308	306	306
query67	16310	15714	15551	15551
query68	4607	587	590	587
query69	449	274	269	269
query70	1118	1098	1130	1098
query71	348	263	260	260
query72	6291	3948	4014	3948
query73	754	363	362	362
query74	10693	9183	9264	9183
query75	3376	2621	2685	2621
query76	2673	1118	1088	1088
query77	374	278	276	276
query78	10476	9530	9605	9530
query79	1709	594	620	594
query80	1182	436	441	436
query81	544	225	231	225
query82	894	91	94	91
query83	243	143	150	143
query84	240	83	77	77
query85	1369	317	292	292
query86	436	279	306	279
query87	4453	4246	4215	4215
query88	3496	2421	2385	2385
query89	403	288	297	288
query90	1899	190	187	187
query91	146	105	106	105
query92	70	51	52	51
query93	2011	567	563	563
query94	825	301	297	297
query95	358	257	264	257
query96	615	278	288	278
query97	3312	3199	3182	3182
query98	222	201	207	201
query99	1529	1328	1313	1313
Total cold run time: 301046 ms
Total hot run time: 196705 ms

doris-robot · 2025-07-08T06:42:40Z

ClickBench: Total hot run time: 30.08 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3db59ca0a96a01cc943b34e6e8052e1f83e36bd1, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.23	0.06	0.07
query4	1.63	0.11	0.10
query5	0.53	0.52	0.50
query6	1.14	0.73	0.73
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.57	0.51	0.50
query10	0.54	0.54	0.56
query11	0.15	0.11	0.10
query12	0.14	0.11	0.11
query13	0.62	0.60	0.60
query14	0.80	0.79	0.81
query15	0.86	0.82	0.84
query16	0.39	0.36	0.38
query17	1.02	1.03	1.00
query18	0.24	0.22	0.21
query19	2.02	1.81	1.85
query20	0.01	0.01	0.01
query21	15.42	0.58	0.56
query22	2.48	2.70	1.71
query23	16.88	0.92	0.90
query24	3.35	1.90	1.05
query25	0.29	0.13	0.07
query26	0.41	0.15	0.14
query27	0.04	0.05	0.04
query28	9.33	0.48	0.51
query29	12.60	3.26	3.23
query30	0.25	0.06	0.06
query31	2.86	0.39	0.39
query32	3.23	0.46	0.47
query33	3.02	3.04	2.99
query34	17.10	4.43	4.49
query35	4.60	4.46	4.53
query36	0.66	0.48	0.48
query37	0.08	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.03	0.02
query40	0.16	0.13	0.12
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.02
Total cold run time: 104.04 s
Total hot run time: 30.08 s

hello-stephen · 2025-07-08T07:31:11Z

BE UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	45.23% (12465/27557)
Line Coverage	36.20% (110485/305245)
Region Coverage	35.35% (57319/162168)
Branch Coverage	32.43% (31085/95842)

Followup #52921 Some code is missing after cherry-pick When upgrading from 3.0.x to 3.1.x, if user put jdbc driver jar under `${DORIS_HOME}/jdbc_drivers`, the system will fail to find the jar, which is unexpected. For compatibility, the system should check both `${DORIS_HOME}/jdbc_drivers`(old) and `${DORIS_HOME}/plugins/jdbc_drivers`(new) to find the jar. This PR fix it. Only for branch-3.1. No issue on master branch

morningman requested a review from morrySnow as a code owner July 8, 2025 05:42

morrySnow approved these changes Jul 8, 2025

View reviewed changes

morrySnow changed the title ~~[refactor](plugin) unify the plugin dir (#45936)~~ branch-3.1: [refactor](plugin) unify the plugin dir #45936 Jul 8, 2025

morrySnow merged commit 4325cce into apache:branch-3.1 Jul 8, 2025
22 of 24 checks passed

morningman mentioned this pull request Sep 15, 2025

branch-3.1: [fix](plugin) fix plugin dir compatible issue #56060

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

branch-3.1: [refactor](plugin) unify the plugin dir #45936 #52921

branch-3.1: [refactor](plugin) unify the plugin dir #45936 #52921

Uh oh!

morningman commented Jul 8, 2025 •

edited by morrySnow

Loading

Uh oh!

hello-stephen commented Jul 8, 2025

Uh oh!

morningman commented Jul 8, 2025

Uh oh!

doris-robot commented Jul 8, 2025

Uh oh!

doris-robot commented Jul 8, 2025

Uh oh!

doris-robot commented Jul 8, 2025

Uh oh!

hello-stephen commented Jul 8, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

branch-3.1: [refactor](plugin) unify the plugin dir #45936 #52921

branch-3.1: [refactor](plugin) unify the plugin dir #45936 #52921

Uh oh!

Conversation

morningman commented Jul 8, 2025 • edited by morrySnow Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hello-stephen commented Jul 8, 2025

Uh oh!

morningman commented Jul 8, 2025

Uh oh!

doris-robot commented Jul 8, 2025

Uh oh!

doris-robot commented Jul 8, 2025

Uh oh!

doris-robot commented Jul 8, 2025

Uh oh!

hello-stephen commented Jul 8, 2025

BE UT Coverage Report

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

morningman commented Jul 8, 2025 •

edited by morrySnow

Loading