Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Improvement]: Add TableFormat information to the return value of the Catalog listTable method. #2150

Merged
merged 21 commits into from
Oct 23, 2023

Conversation

baiyangtx
Copy link
Contributor

@baiyangtx baiyangtx commented Oct 19, 2023

Why are the changes needed?

This PR is part of this issue #1061 .

In order for a catalog to support multiple formats, we need to include TableFormat information in the return value of the catalog's listTable interface. The code changes related to the #1061 are very large. To reduce the burden of review, some code refactoring has been proposed as a separate PR.

Brief change log

  • Change the return type of listTable from List<TableIdentifier> to List<TableIDWithFormat>
  • Add format to ServerTableIdentifier
  • Remove format from table_runtime
  • Change the input parameter of ServerTableDescriptor from ServerTableIdentifier to TableIdentifier.

How was this patch tested?

  • Add some test cases that check the changes thoroughly including negative and positive cases if possible

  • Add screenshots for manual tests if appropriate

  • Run test locally before making a pull request

Documentation

  • Does this pull request introduce a new feature? (yes / no)
  • If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

@github-actions github-actions bot added module:core Core module module:ams-dashboard Ams dashboard module labels Oct 19, 2023
# Conflicts:
#	ams/server/src/main/java/com/netease/arctic/server/table/DefaultTableService.java
@baiyangtx baiyangtx marked this pull request as ready for review October 19, 2023 09:53
@codecov
Copy link

codecov bot commented Oct 19, 2023

Codecov Report

Attention: 56 lines in your changes are missing coverage. Please review.

Files Coverage Δ
...tease/arctic/server/IcebergRestCatalogService.java 79.77% <100.00%> (+0.07%) ⬆️
...netease/arctic/server/catalog/ExternalCatalog.java 100.00% <100.00%> (ø)
...arctic/server/catalog/MixedIcebergCatalogImpl.java 52.63% <100.00%> (ø)
...ase/arctic/server/catalog/PaimonServerCatalog.java 52.63% <100.00%> (-6.46%) ⬇️
...m/netease/arctic/server/catalog/ServerCatalog.java 100.00% <ø> (ø)
...ease/arctic/server/optimizing/OptimizingQueue.java 73.88% <100.00%> (+0.07%) ⬆️
...tease/arctic/server/table/DefaultTableService.java 86.98% <100.00%> (-0.56%) ⬇️
...ase/arctic/server/table/ServerTableIdentifier.java 87.23% <100.00%> (+1.86%) ⬆️
.../com/netease/arctic/server/table/TableManager.java 0.00% <ø> (ø)
...com/netease/arctic/server/table/TableMetadata.java 70.75% <100.00%> (-0.81%) ⬇️
... and 9 more

... and 2 files with indirect coverage changes

📢 Thoughts on this report? Let us know!.

@@ -21,16 +21,16 @@
import com.netease.arctic.ams.api.TableFormat;
import com.netease.arctic.table.TableIdentifier;

public class TableMeta {
public class TableIDWithFormat {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about TableIDAndFormat?

core/src/main/java/com/netease/arctic/UnifiedCatalog.java Outdated Show resolved Hide resolved
Copy link
Contributor

@wangtaohz wangtaohz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I left some comments, PTAL @baiyangtx

# Conflicts:
#	ams/server/src/main/java/com/netease/arctic/server/dashboard/ServerTableDescriptor.java
#	ams/server/src/main/java/com/netease/arctic/server/dashboard/controller/TableController.java
Copy link
Contributor

@wangtaohz wangtaohz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@shidayang
Copy link
Contributor

LGTM

@zhoujinsong zhoujinsong merged commit 568e877 into apache:master Oct 23, 2023
5 of 6 checks passed
@baiyangtx baiyangtx deleted the table-id-with-format branch October 23, 2023 02:06
@shidayang shidayang mentioned this pull request Oct 24, 2023
70 tasks
ShawHee pushed a commit to ShawHee/arctic that referenced this pull request Dec 29, 2023
… Catalog listTable method. (apache#2150)

* ServerCatalog listTable return TableIDWithFormat

* ServerCatalog listTable return TableIDWithFormat

* fix checkstyle in ams/server

* fix checkstyles

* remove table_format from table_runtime

* fix checkstyle.

* fix reviewers

* fix compile error

* fix compile

* fix compile

* remove useless code

* fix checkstyle

* unit test passed.

* fix merge errors

* fix ResultMap for ServerTableIdentifier

* remove table format in table_metadata.

* fix typo
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
module:ams-dashboard Ams dashboard module module:core Core module
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants