Skip to content

[opt](hive) add more log for insert hive partition and replay editlog #59198

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
zy-kkk wants to merge 1 commit into from
Closed

[opt](hive) add more log for insert hive partition and replay editlog #59198

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 5 additions & 2 deletions fe/fe-core/src/main/java/org/apache/doris/catalog/RefreshManager.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -167,7 +167,8 @@ public void replayRefreshTable(ExternalObjectLog log) {
}
// See comment in refreshDbInternal for why db and table may be null.
if (!db.isPresent()) {
LOG.warn("failed to find db when replaying refresh table: {}", log.debugForRefreshTable());
LOG.warn("failed to find db when replaying refresh table: {}, catalog={}, catalogInitialized={}",
log.debugForRefreshTable(), catalog.getName(), catalog.isInitialized());
return;
}
Optional<? extends ExternalTable> table;
Expand All @@ -177,7 +178,8 @@ public void replayRefreshTable(ExternalObjectLog log) {
table = db.get().getTableForReplay(log.getTableId());
}
if (!table.isPresent()) {
LOG.warn("failed to find table when replaying refresh table: {}", log.debugForRefreshTable());
LOG.warn("failed to find table when replaying refresh table: {}, db={}, dbInitialized={}",
log.debugForRefreshTable(), db.get().getFullName(), db.get().isInitialized());
return;
}
if (!Strings.isNullOrEmpty(log.getNewTableName())) {
Expand All @@ -201,6 +203,7 @@ public void replayRefreshTable(ExternalObjectLog log) {
newPartNames == null ? 0 : newPartNames.size());
} else {
// Full table cache invalidation
LOG.info("replay refresh table {} with full invalidation", table.get().getName());
refreshTableInternal(db.get(), table.get(), log.getLastUpdateTime());
}
}
Expand Down
2 changes: 2 additions & 0 deletions fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HiveMetaStoreCache.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -636,6 +636,8 @@ public void addPartitionsCache(NameMapping nameMapping, List<String> partitionNa
PartitionValueCacheKey key = new PartitionValueCacheKey(nameMapping, partitionColumnTypes);
HivePartitionValues partitionValues = partitionValuesCache.getIfPresent(key);
if (partitionValues == null) {
LOG.warn("addPartitionsCache: cache miss for table {}.{}, partitions {} will not be added",
nameMapping.getLocalDbName(), nameMapping.getLocalTblName(), partitionNames);
return;
}
HivePartitionValues copy = partitionValues.copy();
Expand Down
2 changes: 2 additions & 0 deletions ...rc/main/java/org/apache/doris/nereids/trees/plans/commands/insert/HiveInsertExecutor.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,8 @@ protected void doAfterCommit() throws DdlException {
table.getName(),
modifiedPartNames,
newPartNames);
LOG.info("doAfterCommit for table {}: writing editlog, modified={}, new={}",
table.getName(), modifiedPartNames, newPartNames);
} else {
// Full table refresh for other FEs
log = ExternalObjectLog.createForRefreshTable(
Expand Down
Loading