[fix](hive)Modify the Hive notification event processing method when using meta cache and add parameters to the Hive catalog. #39239
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thank you for your contribution to Apache Doris. Since 2024-03-18, the Document has been moved to doris-website. |
|
run buildall |
|
run buildall |
TPC-H: Total hot run time: 37508 ms |
|
run buildall |
TPC-H: Total hot run time: 37485 ms |
TPC-DS: Total hot run time: 189905 ms |
ClickBench: Total hot run time: 30.43 s |
|
run buildall |
TPC-H: Total hot run time: 37386 ms |
|
run buildall |
1 similar comment
|
run buildall |
TPC-H: Total hot run time: 37886 ms |
TPC-DS: Total hot run time: 195789 ms |
ClickBench: Total hot run time: 31.73 s |
|
run buildall |
TPC-H: Total hot run time: 38639 ms |
TPC-DS: Total hot run time: 191458 ms |
ClickBench: Total hot run time: 31.58 s |
hubgeter
changed the title
morningman
reviewed
Aug 17, 2024
| @@ -101,6 +106,12 @@ public void checkProperties() throws DdlException { | |||
| "The parameter " + FILE_META_CACHE_TTL_SECOND + " is wrong, value is " + fileMetaCacheTtlSecond); | |||
| } | |||
|
|
|||
| enableHmsEventsIncrementalSync = | |||
| catalogProperty.getOrDefault(HMSProperties.ENABLE_HMS_EVENTS_INCREMENTAL_SYNC, "false").equals("true"); | |||
There was a problem hiding this comment.
The default value should be Config.enable_hms_events_incremental_sync
| catalogProperty.getOrDefault(HMSProperties.ENABLE_HMS_EVENTS_INCREMENTAL_SYNC, "false").equals("true"); | ||
|
|
||
| hmsEventsBatchSizePerRpc = | ||
| Integer.valueOf(catalogProperty.getOrDefault(HMSProperties.HMS_EVENTIS_BATCH_SIZE_PER_RPC, "-1")); |
There was a problem hiding this comment.
Originally I wanted to indicate that this value was not set, but now I want its default value to be Config.hms_events_batch_size_per_rpc
fe/fe-core/src/main/java/org/apache/doris/datasource/hive/event/AlterPartitionEvent.java
Show resolved
Hide resolved
| @@ -125,6 +128,8 @@ private void realRun() { | |||
| } catch (MetastoreNotificationFetchException e) { | |||
| LOG.warn("Failed to fetch hms events on {}. msg: ", hmsExternalCatalog.getName(), e); | |||
| } catch (Exception ex) { | |||
| hmsExternalCatalog.onRefresh(true); | |||
There was a problem hiding this comment.
What is this 2 lines for?
There was a problem hiding this comment.
Suppose you receive a set of events, and when processing the insert event, the initialization reports a nullpointer. The event processing fails, and our data with Hive is out of sync? Refresh it and you can process other events next time. i will modify it to onRefreshCache
|
run buildall |
TPC-H: Total hot run time: 39318 ms |
TPC-DS: Total hot run time: 195658 ms |
ClickBench: Total hot run time: 31.87 s |
|
run buildall |
TPC-H: Total hot run time: 38306 ms |
hubgeter
force-pushed
the
append_hms_auto_update_case
branch
from
0255473
to
b428fa7
Compare
|
run buildall |
TPC-H: Total hot run time: 38084 ms |
TPC-DS: Total hot run time: 185898 ms |
ClickBench: Total hot run time: 31.14 s |
wuwenchi
approved these changes
Aug 23, 2024
CalvinKirs
approved these changes
Aug 23, 2024
github-actions
bot
added
the
approved
|
PR approved by at least one committer and no changes requested. |
morningman
pushed a commit
to morningman/doris
that referenced
this pull request
Aug 23, 2024
…using meta cache and add parameters to the Hive catalog. (apache#39239) before apache#15401 and apache#38244 1. Add the parameter `hive.enable_hms_events_incremental_sync` to the hive catalog, which is used to switch the catalog to read hive notification events (default is false). The default value is `enable_hms_events_incremental_sync` in fe.conf 2. Add the parameter `hive.hms_events_batch_size_per_rpc` to the hive catalog, which is used to set the size of notification events read by the catalog each time. The default value is `hms_events_batch_size_per_rpc` in fe.conf (default is 500) 3. append hms event notification case . 4. Remove the `use_meta_cache` setting in catalog that is forced to true. Example : ``` create catalog if not exists catalog_name properties ( "type"="hms", 'hive.metastore.uris' = 'thrift://externalEnvIp:hms_port', "hive.enable_hms_events_incremental_sync" ="true", "hive.hms_events_batch_size_per_rpc" = "1000" ); ```
morningman
added a commit
that referenced
this pull request
Aug 23, 2024
pick part of #39239 set default value of use_meta_cache to false, same as previous version, to avoid compatibility issue. This should be totally fixed in doris 3.0.2
gavinchou
pushed a commit
that referenced
this pull request
Aug 23, 2024
pick part of #39239 set default value of use_meta_cache to false, same as previous version, to avoid compatibility issue. This should be totally fixed in doris 3.0.2
dataroaring
pushed a commit
that referenced
this pull request
Aug 26, 2024
…using meta cache and add parameters to the Hive catalog. (#39239) before #15401 and #38244 ## Proposed changes 1. Add the parameter `hive.enable_hms_events_incremental_sync` to the hive catalog, which is used to switch the catalog to read hive notification events (default is false). The default value is `enable_hms_events_incremental_sync` in fe.conf 2. Add the parameter `hive.hms_events_batch_size_per_rpc` to the hive catalog, which is used to set the size of notification events read by the catalog each time. The default value is `hms_events_batch_size_per_rpc` in fe.conf (default is 500) 3. append hms event notification case . 4. Remove the `use_meta_cache` setting in catalog that is forced to true. Example : ``` create catalog if not exists catalog_name properties ( "type"="hms", 'hive.metastore.uris' = 'thrift://externalEnvIp:hms_port', "hive.enable_hms_events_incremental_sync" ="true", "hive.hms_events_batch_size_per_rpc" = "1000" ); ```
hubgeter
added a commit
to hubgeter/doris
that referenced
this pull request
Sep 7, 2024
…using meta cache and add parameters to the Hive catalog. (apache#39239) (apache#39865) bp apache#39239 Co-authored-by: daidai <2017501503@qq.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
before #15401 and #38244
Proposed changes
Add the parameter
hive.enable_hms_events_incremental_syncto the hive catalog, which is used to switch the catalog to read hive notification events (default is false). The default value isenable_hms_events_incremental_syncin fe.confAdd the parameter
hive.hms_events_batch_size_per_rpcto the hive catalog, which is used to set the size of notification events read by the catalog each time. The default value ishms_events_batch_size_per_rpcin fe.conf (default is 500)append hms event notification case .
Remove the
use_meta_cachesetting in catalog that is forced to true.Example :