-
Notifications
You must be signed in to change notification settings - Fork 3.7k
[refactor](plugin) unify the plugin dir #45936
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
|
run buildall |
TPC-H: Total hot run time: 32418 ms |
TPC-DS: Total hot run time: 197860 ms |
ClickBench: Total hot run time: 32.46 s |
|
TeamCity be ut coverage result: |
532d7a4 to
e6e7a25
Compare
|
run buildall |
TPC-H: Total hot run time: 31260 ms |
TPC-DS: Total hot run time: 184610 ms |
ClickBench: Total hot run time: 30.88 s |
|
PR approved by at least one committer and no changes requested. |
|
PR approved by anyone and no changes requested. |
|
TeamCity be ut coverage result: |
|
run buildall |
TPC-H: Total hot run time: 31592 ms |
TPC-DS: Total hot run time: 190387 ms |
ClickBench: Total hot run time: 30.88 s |
|
TeamCity be ut coverage result: |
|
run buildall |
TPC-H: Total hot run time: 31528 ms |
TPC-DS: Total hot run time: 190000 ms |
ClickBench: Total hot run time: 30.62 s |
|
TeamCity be ut coverage result: |
|
PR approved by at least one committer and no changes requested. |
### What problem does this PR solve?
This pull request primarily updates the directory structure for plugins
and JDBC drivers, ensuring compatibility with previous versions.
Additionally, it includes changes to various configuration files and
scripts to reflect these updates.
#### Directory Structure Updates:
*
[`be/src/common/config.cpp`](diffhunk://#diff-b626e6ab16bc72abf40db76bf5094fcc8ca3c37534c2eb83b63b7805e1b601ffL54-R54):
Changed the default directory for `jdbc_drivers_dir` and
`trino_connector_plugin_dir` to `${DORIS_HOME}/plugins/*` for better
organization.
[[1]](diffhunk://#diff-b626e6ab16bc72abf40db76bf5094fcc8ca3c37534c2eb83b63b7805e1b601ffL54-R54)
[[2]](diffhunk://#diff-b626e6ab16bc72abf40db76bf5094fcc8ca3c37534c2eb83b63b7805e1b601ffL1288-R1288)
*
[`build.sh`](diffhunk://#diff-4d2a8eefdf2a9783512a35da4dc7676a66404b6f3826a8af9aad038722da6823L755-R759):
Updated the script to create new plugin directories under
`${DORIS_OUTPUT}/fe/plugins/` and `${DORIS_OUTPUT}/be/plugins/`.
[[1]](diffhunk://#diff-4d2a8eefdf2a9783512a35da4dc7676a66404b6f3826a8af9aad038722da6823L755-R759)
[[2]](diffhunk://#diff-4d2a8eefdf2a9783512a35da4dc7676a66404b6f3826a8af9aad038722da6823L882-R890)
#### Code Refactoring for Compatibility:
* `be/src/runtime/user_function_cache.cpp` and
`be/src/runtime/user_function_cache.h`: Introduced
`_check_and_return_default_driver_url` method to maintain compatibility
with the old default directory for JDBC drivers.
[[1]](diffhunk://#diff-08d48161cad0bb4c6ec31c154c3017407caaff2402f0feb599f014f2d87ae9edL328-R354)
[[2]](diffhunk://#diff-0cfb7bc4e6df1282c7dc7f9ec6677ffc9a03ed6cd08f4fea11ba43bcc297f0c3R79-R80)
*
[`fe/be-java-extensions/trino-connector-scanner/src/main/java/org/apache/doris/trinoconnector/TrinoConnectorPluginLoader.java`](diffhunk://#diff-19838c8c525ec05716e434f18d1ef270dfa03029385258c53b8fbe7da5ba25b9L43-R43):
Added `checkAndReturnPluginDir` method to ensure compatibility with the
old plugin directory.
[[1]](diffhunk://#diff-19838c8c525ec05716e434f18d1ef270dfa03029385258c53b8fbe7da5ba25b9L43-R43)
[[2]](diffhunk://#diff-19838c8c525ec05716e434f18d1ef270dfa03029385258c53b8fbe7da5ba25b9L75-R84)
[[3]](diffhunk://#diff-19838c8c525ec05716e434f18d1ef270dfa03029385258c53b8fbe7da5ba25b9R94-R116)
#### Configuration Updates:
*
[`fe/fe-common/src/main/java/org/apache/doris/common/Config.java`](diffhunk://#diff-ed7dee981e5322b45ed92fbc3c9e049191574baa990894ae36a97567ce198c74L144-R144):
Updated various configuration paths to use `EnvUtils.getDorisHome()`
instead of `System.getenv("DORIS_HOME")` for consistency and
maintainability.
[[1]](diffhunk://#diff-ed7dee981e5322b45ed92fbc3c9e049191574baa990894ae36a97567ce198c74L144-R144)
[[2]](diffhunk://#diff-ed7dee981e5322b45ed92fbc3c9e049191574baa990894ae36a97567ce198c74L153-R153)
[[3]](diffhunk://#diff-ed7dee981e5322b45ed92fbc3c9e049191574baa990894ae36a97567ce198c74L204-R207)
[[4]](diffhunk://#diff-ed7dee981e5322b45ed92fbc3c9e049191574baa990894ae36a97567ce198c74L358-R358)
[[5]](diffhunk://#diff-ed7dee981e5322b45ed92fbc3c9e049191574baa990894ae36a97567ce198c74L630-R630)
[[6]](diffhunk://#diff-ed7dee981e5322b45ed92fbc3c9e049191574baa990894ae36a97567ce198c74L639-R642)
[[7]](diffhunk://#diff-ed7dee981e5322b45ed92fbc3c9e049191574baa990894ae36a97567ce198c74L1281-R1281)
[[8]](diffhunk://#diff-ed7dee981e5322b45ed92fbc3c9e049191574baa990894ae36a97567ce198c74L2364-R2371)
[[9]](diffhunk://#diff-ed7dee981e5322b45ed92fbc3c9e049191574baa990894ae36a97567ce198c74L2903-R2903)
#### Minor Cleanup:
*
[`fe/fe-core/src/main/java/org/apache/doris/analysis/CreateFunctionStmt.java`](diffhunk://#diff-1481395f49d6c346e93c01ca884ae2b7e65c85247d02cc91a20cd48b98bdf349L271-L273):
Removed commented-out code for better readability.
*
[`fe/fe-core/src/main/java/org/apache/doris/catalog/JdbcResource.java`](diffhunk://#diff-24260bdb6ba0b3ac0f5c719e5c48c3ea70d13405af93670adc8692de523730cfR38):
Refactored `getFullDriverUrl` method to use
`checkAndReturnDefaultDriverUrl` for compatibility with the old JDBC
drivers directory.
[[1]](diffhunk://#diff-24260bdb6ba0b3ac0f5c719e5c48c3ea70d13405af93670adc8692de523730cfR38)
[[2]](diffhunk://#diff-24260bdb6ba0b3ac0f5c719e5c48c3ea70d13405af93670adc8692de523730cfL296-R297)
[[3]](diffhunk://#diff-24260bdb6ba0b3ac0f5c719e5c48c3ea70d13405af93670adc8692de523730cfR316-R336)
There are following kinds of plugins in Doris:
1. jdbc drivers
2. java udf
3. trino connectors
4. hadoop conf
5. other custom jars
After this PR, all these plugins will be located in
`${DORIS_HOME}/plugins/`.
```
${DORIS_HOME}/plugins/jdbc_drivers/
${DORIS_HOME}/plugins/java_udf/
${DORIS_HOME}/plugins/connectors/
${DORIS_HOME}/plugins/hadoop_conf/
```
I tested:
1. Put jars in old dir
2. Put jars in new dir
3. Put jars in custom dir and set dir in fe/be conf file
All works well.
### What problem does this PR solve?
This pull request primarily updates the directory structure for plugins
and JDBC drivers, ensuring compatibility with previous versions.
Additionally, it includes changes to various configuration files and
scripts to reflect these updates.
#### Directory Structure Updates:
*
[`be/src/common/config.cpp`](diffhunk://#diff-b626e6ab16bc72abf40db76bf5094fcc8ca3c37534c2eb83b63b7805e1b601ffL54-R54):
Changed the default directory for `jdbc_drivers_dir` and
`trino_connector_plugin_dir` to `${DORIS_HOME}/plugins/*` for better
organization.
[[1]](diffhunk://#diff-b626e6ab16bc72abf40db76bf5094fcc8ca3c37534c2eb83b63b7805e1b601ffL54-R54)
[[2]](diffhunk://#diff-b626e6ab16bc72abf40db76bf5094fcc8ca3c37534c2eb83b63b7805e1b601ffL1288-R1288)
*
[`build.sh`](diffhunk://#diff-4d2a8eefdf2a9783512a35da4dc7676a66404b6f3826a8af9aad038722da6823L755-R759):
Updated the script to create new plugin directories under
`${DORIS_OUTPUT}/fe/plugins/` and `${DORIS_OUTPUT}/be/plugins/`.
[[1]](diffhunk://#diff-4d2a8eefdf2a9783512a35da4dc7676a66404b6f3826a8af9aad038722da6823L755-R759)
[[2]](diffhunk://#diff-4d2a8eefdf2a9783512a35da4dc7676a66404b6f3826a8af9aad038722da6823L882-R890)
#### Code Refactoring for Compatibility:
* `be/src/runtime/user_function_cache.cpp` and
`be/src/runtime/user_function_cache.h`: Introduced
`_check_and_return_default_driver_url` method to maintain compatibility
with the old default directory for JDBC drivers.
[[1]](diffhunk://#diff-08d48161cad0bb4c6ec31c154c3017407caaff2402f0feb599f014f2d87ae9edL328-R354)
[[2]](diffhunk://#diff-0cfb7bc4e6df1282c7dc7f9ec6677ffc9a03ed6cd08f4fea11ba43bcc297f0c3R79-R80)
*
[`fe/be-java-extensions/trino-connector-scanner/src/main/java/org/apache/doris/trinoconnector/TrinoConnectorPluginLoader.java`](diffhunk://#diff-19838c8c525ec05716e434f18d1ef270dfa03029385258c53b8fbe7da5ba25b9L43-R43):
Added `checkAndReturnPluginDir` method to ensure compatibility with the
old plugin directory.
[[1]](diffhunk://#diff-19838c8c525ec05716e434f18d1ef270dfa03029385258c53b8fbe7da5ba25b9L43-R43)
[[2]](diffhunk://#diff-19838c8c525ec05716e434f18d1ef270dfa03029385258c53b8fbe7da5ba25b9L75-R84)
[[3]](diffhunk://#diff-19838c8c525ec05716e434f18d1ef270dfa03029385258c53b8fbe7da5ba25b9R94-R116)
#### Configuration Updates:
*
[`fe/fe-common/src/main/java/org/apache/doris/common/Config.java`](diffhunk://#diff-ed7dee981e5322b45ed92fbc3c9e049191574baa990894ae36a97567ce198c74L144-R144):
Updated various configuration paths to use `EnvUtils.getDorisHome()`
instead of `System.getenv("DORIS_HOME")` for consistency and
maintainability.
[[1]](diffhunk://#diff-ed7dee981e5322b45ed92fbc3c9e049191574baa990894ae36a97567ce198c74L144-R144)
[[2]](diffhunk://#diff-ed7dee981e5322b45ed92fbc3c9e049191574baa990894ae36a97567ce198c74L153-R153)
[[3]](diffhunk://#diff-ed7dee981e5322b45ed92fbc3c9e049191574baa990894ae36a97567ce198c74L204-R207)
[[4]](diffhunk://#diff-ed7dee981e5322b45ed92fbc3c9e049191574baa990894ae36a97567ce198c74L358-R358)
[[5]](diffhunk://#diff-ed7dee981e5322b45ed92fbc3c9e049191574baa990894ae36a97567ce198c74L630-R630)
[[6]](diffhunk://#diff-ed7dee981e5322b45ed92fbc3c9e049191574baa990894ae36a97567ce198c74L639-R642)
[[7]](diffhunk://#diff-ed7dee981e5322b45ed92fbc3c9e049191574baa990894ae36a97567ce198c74L1281-R1281)
[[8]](diffhunk://#diff-ed7dee981e5322b45ed92fbc3c9e049191574baa990894ae36a97567ce198c74L2364-R2371)
[[9]](diffhunk://#diff-ed7dee981e5322b45ed92fbc3c9e049191574baa990894ae36a97567ce198c74L2903-R2903)
#### Minor Cleanup:
*
[`fe/fe-core/src/main/java/org/apache/doris/analysis/CreateFunctionStmt.java`](diffhunk://#diff-1481395f49d6c346e93c01ca884ae2b7e65c85247d02cc91a20cd48b98bdf349L271-L273):
Removed commented-out code for better readability.
*
[`fe/fe-core/src/main/java/org/apache/doris/catalog/JdbcResource.java`](diffhunk://#diff-24260bdb6ba0b3ac0f5c719e5c48c3ea70d13405af93670adc8692de523730cfR38):
Refactored `getFullDriverUrl` method to use
`checkAndReturnDefaultDriverUrl` for compatibility with the old JDBC
drivers directory.
[[1]](diffhunk://#diff-24260bdb6ba0b3ac0f5c719e5c48c3ea70d13405af93670adc8692de523730cfR38)
[[2]](diffhunk://#diff-24260bdb6ba0b3ac0f5c719e5c48c3ea70d13405af93670adc8692de523730cfL296-R297)
[[3]](diffhunk://#diff-24260bdb6ba0b3ac0f5c719e5c48c3ea70d13405af93670adc8692de523730cfR316-R336)
There are following kinds of plugins in Doris:
1. jdbc drivers
2. java udf
3. trino connectors
4. hadoop conf
5. other custom jars
After this PR, all these plugins will be located in
`${DORIS_HOME}/plugins/`.
```
${DORIS_HOME}/plugins/jdbc_drivers/
${DORIS_HOME}/plugins/java_udf/
${DORIS_HOME}/plugins/connectors/
${DORIS_HOME}/plugins/hadoop_conf/
```
I tested:
1. Put jars in old dir
2. Put jars in new dir
3. Put jars in custom dir and set dir in fe/be conf file
All works well.
This pull request primarily updates the directory structure for plugins
and JDBC drivers, ensuring compatibility with previous versions.
Additionally, it includes changes to various configuration files and
scripts to reflect these updates.
*
[`be/src/common/config.cpp`](diffhunk://#diff-b626e6ab16bc72abf40db76bf5094fcc8ca3c37534c2eb83b63b7805e1b601ffL54-R54):
Changed the default directory for `jdbc_drivers_dir` and
`trino_connector_plugin_dir` to `${DORIS_HOME}/plugins/*` for better
organization.
[[1]](diffhunk://#diff-b626e6ab16bc72abf40db76bf5094fcc8ca3c37534c2eb83b63b7805e1b601ffL54-R54)
[[2]](diffhunk://#diff-b626e6ab16bc72abf40db76bf5094fcc8ca3c37534c2eb83b63b7805e1b601ffL1288-R1288)
*
[`build.sh`](diffhunk://#diff-4d2a8eefdf2a9783512a35da4dc7676a66404b6f3826a8af9aad038722da6823L755-R759):
Updated the script to create new plugin directories under
`${DORIS_OUTPUT}/fe/plugins/` and `${DORIS_OUTPUT}/be/plugins/`.
[[1]](diffhunk://#diff-4d2a8eefdf2a9783512a35da4dc7676a66404b6f3826a8af9aad038722da6823L755-R759)
[[2]](diffhunk://#diff-4d2a8eefdf2a9783512a35da4dc7676a66404b6f3826a8af9aad038722da6823L882-R890)
* `be/src/runtime/user_function_cache.cpp` and
`be/src/runtime/user_function_cache.h`: Introduced
`_check_and_return_default_driver_url` method to maintain compatibility
with the old default directory for JDBC drivers.
[[1]](diffhunk://#diff-08d48161cad0bb4c6ec31c154c3017407caaff2402f0feb599f014f2d87ae9edL328-R354)
[[2]](diffhunk://#diff-0cfb7bc4e6df1282c7dc7f9ec6677ffc9a03ed6cd08f4fea11ba43bcc297f0c3R79-R80)
*
[`fe/be-java-extensions/trino-connector-scanner/src/main/java/org/apache/doris/trinoconnector/TrinoConnectorPluginLoader.java`](diffhunk://#diff-19838c8c525ec05716e434f18d1ef270dfa03029385258c53b8fbe7da5ba25b9L43-R43):
Added `checkAndReturnPluginDir` method to ensure compatibility with the
old plugin directory.
[[1]](diffhunk://#diff-19838c8c525ec05716e434f18d1ef270dfa03029385258c53b8fbe7da5ba25b9L43-R43)
[[2]](diffhunk://#diff-19838c8c525ec05716e434f18d1ef270dfa03029385258c53b8fbe7da5ba25b9L75-R84)
[[3]](diffhunk://#diff-19838c8c525ec05716e434f18d1ef270dfa03029385258c53b8fbe7da5ba25b9R94-R116)
*
[`fe/fe-common/src/main/java/org/apache/doris/common/Config.java`](diffhunk://#diff-ed7dee981e5322b45ed92fbc3c9e049191574baa990894ae36a97567ce198c74L144-R144):
Updated various configuration paths to use `EnvUtils.getDorisHome()`
instead of `System.getenv("DORIS_HOME")` for consistency and
maintainability.
[[1]](diffhunk://#diff-ed7dee981e5322b45ed92fbc3c9e049191574baa990894ae36a97567ce198c74L144-R144)
[[2]](diffhunk://#diff-ed7dee981e5322b45ed92fbc3c9e049191574baa990894ae36a97567ce198c74L153-R153)
[[3]](diffhunk://#diff-ed7dee981e5322b45ed92fbc3c9e049191574baa990894ae36a97567ce198c74L204-R207)
[[4]](diffhunk://#diff-ed7dee981e5322b45ed92fbc3c9e049191574baa990894ae36a97567ce198c74L358-R358)
[[5]](diffhunk://#diff-ed7dee981e5322b45ed92fbc3c9e049191574baa990894ae36a97567ce198c74L630-R630)
[[6]](diffhunk://#diff-ed7dee981e5322b45ed92fbc3c9e049191574baa990894ae36a97567ce198c74L639-R642)
[[7]](diffhunk://#diff-ed7dee981e5322b45ed92fbc3c9e049191574baa990894ae36a97567ce198c74L1281-R1281)
[[8]](diffhunk://#diff-ed7dee981e5322b45ed92fbc3c9e049191574baa990894ae36a97567ce198c74L2364-R2371)
[[9]](diffhunk://#diff-ed7dee981e5322b45ed92fbc3c9e049191574baa990894ae36a97567ce198c74L2903-R2903)
*
[`fe/fe-core/src/main/java/org/apache/doris/analysis/CreateFunctionStmt.java`](diffhunk://#diff-1481395f49d6c346e93c01ca884ae2b7e65c85247d02cc91a20cd48b98bdf349L271-L273):
Removed commented-out code for better readability.
*
[`fe/fe-core/src/main/java/org/apache/doris/catalog/JdbcResource.java`](diffhunk://#diff-24260bdb6ba0b3ac0f5c719e5c48c3ea70d13405af93670adc8692de523730cfR38):
Refactored `getFullDriverUrl` method to use
`checkAndReturnDefaultDriverUrl` for compatibility with the old JDBC
drivers directory.
[[1]](diffhunk://#diff-24260bdb6ba0b3ac0f5c719e5c48c3ea70d13405af93670adc8692de523730cfR38)
[[2]](diffhunk://#diff-24260bdb6ba0b3ac0f5c719e5c48c3ea70d13405af93670adc8692de523730cfL296-R297)
[[3]](diffhunk://#diff-24260bdb6ba0b3ac0f5c719e5c48c3ea70d13405af93670adc8692de523730cfR316-R336)
There are following kinds of plugins in Doris:
1. jdbc drivers
2. java udf
3. trino connectors
4. hadoop conf
5. other custom jars
After this PR, all these plugins will be located in
`${DORIS_HOME}/plugins/`.
```
${DORIS_HOME}/plugins/jdbc_drivers/
${DORIS_HOME}/plugins/java_udf/
${DORIS_HOME}/plugins/connectors/
${DORIS_HOME}/plugins/hadoop_conf/
```
I tested:
1. Put jars in old dir
2. Put jars in new dir
3. Put jars in custom dir and set dir in fe/be conf file
All works well.
What problem does this PR solve?
This pull request primarily updates the directory structure for plugins and JDBC drivers, ensuring compatibility with previous versions. Additionally, it includes changes to various configuration files and scripts to reflect these updates.
Directory Structure Updates:
be/src/common/config.cpp: Changed the default directory forjdbc_drivers_dirandtrino_connector_plugin_dirto${DORIS_HOME}/plugins/*for better organization. [1] [2]build.sh: Updated the script to create new plugin directories under${DORIS_OUTPUT}/fe/plugins/and${DORIS_OUTPUT}/be/plugins/. [1] [2]Code Refactoring for Compatibility:
be/src/runtime/user_function_cache.cppandbe/src/runtime/user_function_cache.h: Introduced_check_and_return_default_driver_urlmethod to maintain compatibility with the old default directory for JDBC drivers. [1] [2]fe/be-java-extensions/trino-connector-scanner/src/main/java/org/apache/doris/trinoconnector/TrinoConnectorPluginLoader.java: AddedcheckAndReturnPluginDirmethod to ensure compatibility with the old plugin directory. [1] [2] [3]Configuration Updates:
fe/fe-common/src/main/java/org/apache/doris/common/Config.java: Updated various configuration paths to useEnvUtils.getDorisHome()instead ofSystem.getenv("DORIS_HOME")for consistency and maintainability. [1] [2] [3] [4] [5] [6] [7] [8] [9]Minor Cleanup:
fe/fe-core/src/main/java/org/apache/doris/analysis/CreateFunctionStmt.java: Removed commented-out code for better readability.fe/fe-core/src/main/java/org/apache/doris/catalog/JdbcResource.java: RefactoredgetFullDriverUrlmethod to usecheckAndReturnDefaultDriverUrlfor compatibility with the old JDBC drivers directory. [1] [2] [3]There are following kinds of plugins in Doris:
After this PR, all these plugins will be located in
${DORIS_HOME}/plugins/.I tested:
All works well.
Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)