[HotFix] Add table filter for catalog properties #2310

HuangFru · 2023-11-15T07:31:04Z

Why are the changes needed?

We've lost table-filter in #2232.

Brief change log

Add table filter in UnifiedCatalog

How was this patch tested?

Add some test cases that check the changes thoroughly including negative and positive cases if possible
Add screenshots for manual tests if appropriate
Run test locally before making a pull request

Documentation

Does this pull request introduce a new feature? (no)
If yes, how is the feature documented? (not applicable)

codecov · 2023-11-15T07:37:49Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (0384d8e) 53.39% compared to head (24a40a5) 53.53%.

Additional details and impacted files

@@             Coverage Diff              @@
##             master    #2310      +/-   ##
============================================
+ Coverage     53.39%   53.53%   +0.13%     
- Complexity     4354     4369      +15     
============================================
  Files           517      517              
  Lines         29856    29872      +16     
  Branches       2908     2910       +2     
============================================
+ Hits          15942    15991      +49     
+ Misses        12637    12594      -43     
- Partials       1277     1287      +10

Flag	Coverage Δ
core	`54.01% <100.00%> (+0.06%)`	⬆️
trino	`50.95% <ø> (+0.50%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

baiyangtx

BasicMixedIcebergCatalog also need this.

wangtaohz · 2023-11-15T08:05:12Z

I think table-filter should be a generic feature for the external catalog. Maybe you can help confirm if it is feasible. @HuangFru

HuangFru · 2023-11-15T08:08:59Z

I think table-filter should be a generic feature for the external catalog. Maybe you can help confirm if it is feasible. @HuangFru

What do you mean? Judging from the current code structure, it should be effective for all external catalogs.

HuangFru · 2023-11-15T08:16:40Z

BasicMixedIcebergCatalog also need this.

Why? For external Mix-iceberg?

wangtaohz · 2023-11-15T11:13:24Z

What do you mean? Judging from the current code structure, it should be effective for all external catalogs.

Yes, but I find a databaseFilterPattern in com.netease.arctic.mixed.BasicMixedIcebergCatalog. Maybe @baiyangtx can provide more information.

shidayang · 2023-11-16T01:58:42Z

Table filter is only for external catalog, Unified includes internal and external

HuangFru · 2023-11-16T02:07:41Z

Table filter is only for external catalog, Unified includes internal and external

I got you.

ams/server/src/main/java/com/netease/arctic/server/catalog/ExternalCatalog.java

CLAassistant · 2023-11-22T08:41:55Z

All committers have signed the CLA.

baiyangtx · 2023-11-22T11:15:00Z

Table filter is only for external catalog, Unified includes internal and external

Table filter works for AMS, but InternalCatalog will not use UnifiedCatalog for ServerCatalog implementation.

baiyangtx · 2023-11-22T11:17:35Z

BasicMixedIcebergCatalog

What do you mean? Judging from the current code structure, it should be effective for all external catalogs.

Yes, but I find a databaseFilterPattern in com.netease.arctic.mixed.BasicMixedIcebergCatalog. Maybe @baiyangtx can provide more information.

BasicMixedIcebergCatalog is used in external catalog.

The catalog type[Internal/External] is a concept of AMS Instance, for engine side, all catalogs have no different.

HuangFru · 2023-11-23T01:48:23Z

BasicMixedIcebergCatalog

What do you mean? Judging from the current code structure, it should be effective for all external catalogs.

Yes, but I find a databaseFilterPattern in com.netease.arctic.mixed.BasicMixedIcebergCatalog. Maybe @baiyangtx can provide more information.

BasicMixedIcebergCatalog is used in external catalog.

The catalog type[Internal/External] is a concept of AMS Instance, for engine side, all catalogs have no different.

So you mean the engine side also needs the table filter?

…-table-filter # Conflicts: # core/src/main/java/com/netease/arctic/mixed/BasicMixedIcebergCatalog.java

zhoujinsong · 2023-11-29T03:14:39Z

AFAIK, the table filter feature helps AMS filter out tables that are not relevant for attention, and currently, only the external catalogs require it. Although the BasicMixedIcebergCatalog now has an implementation of a database filter, I believe it should be removed as the engine side does not have any scenarios that require filtering tables or databases. This feature should be specific to AMS.

HDYT? @HuangFru @wangtaohz @baiyangtx

HuangFru · 2023-11-29T07:59:03Z

AFAIK, the table filter feature helps AMS filter out tables that are not relevant for attention, and currently, only the external catalogs require it. Although the BasicMixedIcebergCatalog now has an implementation of a database filter, I believe it should be removed as the engine side does not have any scenarios that require filtering tables or databases. This feature should be specific to AMS.

HDYT? @HuangFru @wangtaohz @baiyangtx

There is indeed no scenario on the engine side that requires filtering tables or databases, I'll rollback the change.

ams/server/src/test/java/com/netease/arctic/server/catalog/TestServerCatalog.java

baiyangtx

Should table-filter needs to be applied to the listDatabase interface ?

…-table-filter

wangtaohz

LGTM

HuangFru · 2023-12-06T07:35:55Z

Should table-filter needs to be applied to the listDatabase interface ?

I think it's hard to cut a db filter from the table filter to apply to the listDatabase, have some ideas?

zhongqishang · 2023-12-06T07:54:51Z

Should table-filter needs to be applied to the listDatabase interface ?

I think it's hard to cut a db filter from the table filter to apply to the listDatabase, have some ideas?

Have you considered restoring the previous parameter database.filter-regular-expression to support listDatabase?
I've seen something similar in Flink MongoDB CDC connectors.

HuangFru · 2023-12-06T09:38:45Z

Should table-filter needs to be applied to the listDatabase interface ?

I think it's hard to cut a db filter from the table filter to apply to the listDatabase, have some ideas?

Have you considered restoring the previous parameter database.filter-regular-expression to support listDatabase? I've seen something similar in Flink MongoDB CDC connectors.

I don’t know if the community felt that table filter could replace database.filter-regular-expression at that time and marked it as deprecated. In my opinion, the table filter is good enough to use.
WDYT @wangtaohz @baiyangtx @zhoujinsong ?

wangtaohz · 2023-12-06T11:11:47Z

IMO, it seems unnecessary to filter databases now, and the current implementation of this PR is sufficient.

If there is a need to filter databases, it can be addressed in another issue.

baiyangtx

LGTM, This parameter was originally introduced to reduce the number of tables processed by AMS to improve stability. From this perspective, listDatabase is unnecessary.

* re add table filter * implement in external catalog * add ut case * fix comment * fix comment * fix comment * fix ut * fix update properties * roll back the engine side's filter * resolve conflicts * add ut --------- Co-authored-by: baiyangtx <xiangnebula@163.com> Co-authored-by: ZhouJinsong <zhoujinsong0505@163.com>

* [AMORO-1812] support spark-based external optimizer * resolve code style error * [AMORO-1951] Support parallelized planning in one optimizer group (#2282) * [AMORO-1951] Support parallelized planning in one optimizer group * [AMORO-1951] add unit test for OptimizingQueue and DefaultOptimizingService * [AMORO-1951] optimize default parameters * fix bugs * fix warnings and spotless issues * merge from #2290 * add apache license and fix spotless * fix config error * Update ams/server/src/main/java/com/netease/arctic/server/DefaultOptimizingService.java Co-authored-by: ZhouJinsong <zhoujinsong0505@163.com> * add annotations * fix compile errors * fix import problem * remove isDebugEnabled() * spotless apply * Update ArcticManagementConf.java * fix reboot bug and supply document content * use MoreObjects.toStringHelper for OptimizerThread.java * Merged from [AMORO-2376] Print right log info after calculating and sorting tables * fix import problem * remove unused codes * spotless * remove incorrect comments * add max-planning-parallelism to config --------- Co-authored-by: majin1102 <majin1102@163.com> Co-authored-by: ZhouJinsong <zhoujinsong0505@163.com> * [AMORO-2378] The optimizer based on Flink on YARN should prioritize loading the optimizer-job.jar (#2379) * load optimizer jar first * fix code style * change config name * add config taskmanager.memory.managed.fraction * fix * [AMORO-2222] [Improvement]: Skip cleaning up dangling delete files for Iceberg V1 table (#2361) * [AMORO-2222] [Improvement]: Skip cleaning up dangling delete files for Iceberg V1 table * Update IcebergTableMaintainer.java The `total-delete-files` could be 0. --------- Co-authored-by: wangtaohz <103108928+wangtaohz@users.noreply.github.com> * [AMORO-2404] fix Mixed Hive table mistakenly deletes hive files during expiring snapshots (#2405) get hive locations return the uri path * [AMORO-2407] Fix access data file from dashboard of non-partitioned table (#2408) * fix null partition * fix listing files of non-partitioned iceberg table * [AMORO-2383] Add serialVersionUID to RewriteFilesInput (#2384) * add serialVersionUID * fix comment * [AMORO-1720] Fix Mixed Format KeyedTable expiring all the snapshots with optimized sequence (#2394) * should not expire the latest snapshot contains optimized sequence * add visible for testing * add fetchLatestNonOptimizedSnapshotTime for base store * get hive locations return the uri path * refactor codes and fix comments * improve for exclude files is empty for expring snapshots --------- Co-authored-by: ZhouJinsong <zhoujinsong0505@163.com> * [AMORO-2386][AMS] Configure `iceberg.worker.num-threads` in the config.yaml (#2393) * [AMORO-2386][AMS] Configure `iceberg.worker.num-threads` in the config.yaml * Fix * [AMORO-2386][AMS] reuse config `table-manifest-io.thread-count` and reuse thread pool * Add comment * [AMORO-1716] [Improvement]: sort the table list returned by server (#2362) * improve: sort the table list returned by server * optimize: sort tables by format * optimize: optimiz tables sorting * style: udpate comment --------- Co-authored-by: chenyuzhi <chenyuzhi@corp.netease.com> Co-authored-by: ZhouJinsong <zhoujinsong0505@163.com> * [HotFix] Re-add table-filter to Server ExternalCatalog (#2310) * re add table filter * implement in external catalog * add ut case * fix comment * fix comment * fix comment * fix ut * fix update properties * roll back the engine side's filter * resolve conflicts * add ut --------- Co-authored-by: baiyangtx <xiangnebula@163.com> Co-authored-by: ZhouJinsong <zhoujinsong0505@163.com> * [AMORO-2299]: Cancel the running optimizing process from ams web console (#2297) * cancel the running opimizing process from ams web console * refact code to avoid NPE * add o comment for com.netease.arctic.server.table.TableService#getServerTableIdentifier * change the cancel post api to be more restful style * [AMORO-2415] Print GC date stamps (#2416) add gc timestamp * Update wrong comments in SnapshotsExpiringExecutor.java (#2422) * [AMORO-2276]: UnifiiedCatalog for Spark Engine (#2269) * Add UnifiedSparkCatalog under spark common module * Extract MixedSparkCatalogBase and MixedSparkSessionCatalogBase to spark common module * Refactor spark unit test framework to adapt unifed catalog tests and mixed format tests. * [AMORO-2261] Extract the deleting dangling files from the cleaning orphan files (#2403) * [Improvement]: Extract the deleting dangling files from the cleaning orphan files * [Improvement]: Extract the deleting dangling files from the cleaning orphan files * [Improvement]: Extract the deleting dangling files from the cleaning orphan files * [AMORO-1341] [Flink]: Support UnifiedCatalog to contain Mixed format table in Flink Engine (#2370) * [AMORO-1341] [Flink]: Support UnifiedCatalog to contain Mixed format table in Flink Engine * [AMORO-2413] Need to select the first db after switching to another Catalog (#2419) * fix: If the current catalog is not the one in the query, the first db is selected by default. * build dashboard frontend --------- Co-authored-by: wangtao <wangtao3@corp.netease.com> * [HotFix] Fix loading the optimizing snapshot id of change store for Mixed Format KeyedTable (#2430) fix load target change snapshot id * [AMORO-2260] Show the format version of iceberg table (#2425) [AMORO-2260] Show the format version of Iceberg Table Signed-off-by: tcodehuber <tcodehuber@gmail.com> * [AMORO-2115] Support displaying Optimizing tasks (#2322) * dashboard: rename optimized to optimizing * dashboard: support optimizing taskes * add optimizer token * dashboard: modify column width * dashboard: build * sort the metrics field and change record cnt to long * modify MetricsSummary Compatibility * dashbard: build * Update ams/server/src/main/java/com/netease/arctic/server/optimizing/TaskRuntime.java Co-authored-by: Qishang Zhong <zhongqishang@gmail.com> * fix * support input metrics and output metrics for optimizing process * dashboard: support optimizing metrics * dashbard: build * dashboard:rebuild * support MetricsSummary to map * optimizing task supports input output * dashboard: optimizing tasks support input and output * dashboard: not display seconds when longer than 1 hour * dashboard: optimizing process show summary * remove useless import * dashboard: build * as head * dashbard: build * change process status to CLOSED after cancel process * remove useless log * dashboard: refresh after cancelled * support cancel optimizing tasks * dashboard: handle exception when can't cancel optimizing process * throw exception when can't cancel optimizing process * dashboard: build * dashboard: refresh optimizing process when exist optimizing detail page * dashboard: build * fix cost time is 0ms * change metrics name * fix task startTime and endTime * fix costTime * using Preconditions.checkArgument * fix task reset * add comments * cancel tasks before closing optimizing process * fix unit test * fix cancel task * as head * Revert "as head" This reverts commit e469e71. * dashboard: build --------- Co-authored-by: Qishang Zhong <zhongqishang@gmail.com> * [AMORO-2385] Make the maximum input file size for per optimize thread configurable (#2387) * add config self-optimizing.max-input-file-size-per-thread * add doc * add resource group property max-input-file-size-per-thread * add doc * fix compile * [Hotfix] Add database filter to Server ExternalCatalog (#2414) * [Hotfix] Add database filter to Server ExternalCatalog * [Hotfix] Add database filter to Server ExternalCatalog * Rename config database.filter-regular-expression to database-filter --------- Co-authored-by: baiyangtx <xiangnebula@163.com> * [AMORO-2423] [Flink]: Using 'mixed_iceberg' and 'mixed_hive' indentifier to CREATE CATALOG and deprecate 'arctic' identifier (#2424) * [AMORO-2423] [Flink]: Using 'mixed_iceberg' and 'mixed_hive' identifiers to CREATE CATALOG and deprecate 'arctic' identifier * [AMORO-2316] The Files page supports filtering by partition name and sorting by dictionary value. (#2420) * AMORO-2316: The Files page supports filtering by partition name and sorting by dictionary value. * build dashboard frontend * dashboard: build --------- Co-authored-by: wangtao <wangtao3@corp.netease.com> * [AMORO-1892] Improve the SQL Shortcuts in Terminal Web UI (#2434) * [AMORO-1892] Improve the SQL Shortcuts in Terminal Web UI * refactor some code --------- Signed-off-by: tcodehuber <tcodehuber@gmail.com> * [AMORO-2440] Fix the batch deletion of Change Store files for Mixed Format Table (#2439) fix remove change files * [AMORO-2418] Exclude kryo dependency from flink-optimizer (#2437) exclude kryo * [AMORO-2441] Fix `TableEntriesScan` when no file format is specified in the file name (#2442) * fix TableEntriesScan without format suffix * using the file format from entries * [Hotfix] Fix fetchLatestNonOptimizedSnapshotTime (#2396) * fix fetchLatestNonOptimizedSnapshotTime * fix * spotless * fix ut * rename data-expire.since * using UTC zone for snapshot timestamp * resolve conflict * rerview * spotless * review * review * [AMORO-2344] [Flink]: Support UnifiedCatalog to contain Iceberg format table in Flink Engine (#2427) * [AMORO-2330] Improve major plan (#2332) * [AMORO-2330][AMS] Improve major plan * [AMORO-2330][AMS] remove filter dataFileWith1Pos * Fix comments * Fix * Move the rollback logic of undersized segment to the split task stage * Rename * Fix * Rollback method name `fileShouldRewrite` * Rollback mixed table format full trigger condition & reuse `isUndersizedSegmentFile` * Rollback testSegmentFilesBase() * TreeNodeTaskSplitter logic keep same with bin-pack * Improve code duplicate --------- Co-authored-by: ZhouJinsong <zhoujinsong0505@163.com> * [AMORO-1810] Check the validity of the heatbeat interval when an opti… (#2432) * [AMORO-1810] Check the validity of the heatbeat interval when an optimizer start * adjust the import way * resolve some logic code * refactor code * refactor code * fix ut error * resolve ut error * fix ut error * fix ut error * fix ut error * fix ci error * refactor code * refactor code * refactor code * [AMORO-1812] support spark-based external optimizer * resolve code style error * refactor code * refactor code * bugfix * refactor code * refactor code * code style * bugfix * bugfix --------- Signed-off-by: tcodehuber <tcodehuber@gmail.com> Co-authored-by: JinMat <majin1102@gmail.com> Co-authored-by: majin1102 <majin1102@163.com> Co-authored-by: ZhouJinsong <zhoujinsong0505@163.com> Co-authored-by: wangzeyu <hameizi369@gmail.com> Co-authored-by: ConradJam <jam.gzczy@gmail.com> Co-authored-by: wangtaohz <103108928+wangtaohz@users.noreply.github.com> Co-authored-by: yeatsliao <liaoyt66066@gmail.com> Co-authored-by: Qishang Zhong <zhongqishang@gmail.com> Co-authored-by: chenyuzhi459 <553673833@qq.com> Co-authored-by: chenyuzhi <chenyuzhi@corp.netease.com> Co-authored-by: HuangFru <68625618+HuangFru@users.noreply.github.com> Co-authored-by: baiyangtx <xiangnebula@163.com> Co-authored-by: xujiangfeng001 <104614523+xujiangfeng001@users.noreply.github.com> Co-authored-by: Xianxun Ye <yesorno828423@gmail.com> Co-authored-by: liuweimin <minteliu.l@gmail.com> Co-authored-by: wangtao <wangtao3@corp.netease.com> Co-authored-by: Xavier Bai <xuba@cisco.com>

re add table filter

7fece64

github-actions bot added the module:core Core module label Nov 15, 2023

baiyangtx reviewed Nov 15, 2023

View reviewed changes

implement in external catalog

788570c

github-actions bot added module:ams-dashboard Ams dashboard module and removed module:core Core module labels Nov 16, 2023

add ut case

400c5b7

wangtaohz mentioned this pull request Nov 20, 2023

[AMORO-2326] Fixed the bug that some catalog configurations did not take effect #2327

Closed

3 tasks

huyuanfeng2018 mentioned this pull request Nov 20, 2023

[Bug]: Some catalog properties do not take effect #2326

Closed

2 tasks

shidayang reviewed Nov 21, 2023

View reviewed changes

ams/server/src/main/java/com/netease/arctic/server/catalog/ExternalCatalog.java Outdated Show resolved Hide resolved

fix comment

b5597d7

baiyangtx and others added 4 commits November 23, 2023 10:11

Merge branch 'master' into hotfix-table-filter

b141328

fix comment

825c31e

Merge remote-tracking branch 'origin/hotfix-table-filter' into hotfix…

1518790

…-table-filter # Conflicts: # core/src/main/java/com/netease/arctic/mixed/BasicMixedIcebergCatalog.java

fix comment

8915d44

github-actions bot added the module:core Core module label Nov 23, 2023

HuangFru and others added 3 commits November 23, 2023 15:59

fix ut

791bf36

fix update properties

3b79b57

Merge branch 'master' into hotfix-table-filter

beec6a6

roll back the engine side's filter

1ddf92f

github-actions bot removed the module:core Core module label Nov 29, 2023

wangtaohz reviewed Dec 6, 2023

View reviewed changes

ams/server/src/test/java/com/netease/arctic/server/catalog/TestServerCatalog.java Outdated Show resolved Hide resolved

HuangFru and others added 2 commits December 6, 2023 10:56

Merge branch 'master' into hotfix-table-filter

1c4e63a

resolve conflicts

ffd82bd

baiyangtx reviewed Dec 6, 2023

View reviewed changes

baiyangtx and others added 3 commits December 6, 2023 14:42

Merge branch 'master' into hotfix-table-filter

4ad23d5

Merge remote-tracking branch 'origin/hotfix-table-filter' into hotfix…

14afadd

…-table-filter

add ut

24a40a5

wangtaohz approved these changes Dec 6, 2023

View reviewed changes

baiyangtx approved these changes Dec 7, 2023

View reviewed changes

baiyangtx merged commit efe8ae7 into apache:master Dec 7, 2023
7 checks passed

zhoujinsong mentioned this pull request Dec 19, 2023

Release-0.6.1 roadmap #2448

Closed

33 tasks

zhoujinsong mentioned this pull request Jun 25, 2024

Release-0.7.0 roadmap #2176

Closed

66 tasks

zhoujinsong changed the title ~~[HotFix] Re-add table filter~~ [HotFix] Add table filter for catalog properties Jun 26, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[HotFix] Add table filter for catalog properties #2310

[HotFix] Add table filter for catalog properties #2310

HuangFru commented Nov 15, 2023

codecov bot commented Nov 15, 2023 •

edited

Loading

baiyangtx left a comment

wangtaohz commented Nov 15, 2023

HuangFru commented Nov 15, 2023

HuangFru commented Nov 15, 2023

wangtaohz commented Nov 15, 2023

shidayang commented Nov 16, 2023

HuangFru commented Nov 16, 2023

CLAassistant commented Nov 22, 2023 •

edited

Loading

baiyangtx commented Nov 22, 2023

baiyangtx commented Nov 22, 2023

HuangFru commented Nov 23, 2023

zhoujinsong commented Nov 29, 2023

HuangFru commented Nov 29, 2023

baiyangtx left a comment

wangtaohz left a comment

HuangFru commented Dec 6, 2023

zhongqishang commented Dec 6, 2023 •

edited

Loading

HuangFru commented Dec 6, 2023

wangtaohz commented Dec 6, 2023

baiyangtx left a comment

[HotFix] Add table filter for catalog properties #2310

[HotFix] Add table filter for catalog properties #2310

Conversation

HuangFru commented Nov 15, 2023

Why are the changes needed?

Brief change log

How was this patch tested?

Documentation

codecov bot commented Nov 15, 2023 • edited Loading

Codecov Report

baiyangtx left a comment

Choose a reason for hiding this comment

wangtaohz commented Nov 15, 2023

HuangFru commented Nov 15, 2023

HuangFru commented Nov 15, 2023

wangtaohz commented Nov 15, 2023

shidayang commented Nov 16, 2023

HuangFru commented Nov 16, 2023

CLAassistant commented Nov 22, 2023 • edited Loading

baiyangtx commented Nov 22, 2023

baiyangtx commented Nov 22, 2023

HuangFru commented Nov 23, 2023

zhoujinsong commented Nov 29, 2023

HuangFru commented Nov 29, 2023

baiyangtx left a comment

Choose a reason for hiding this comment

wangtaohz left a comment

Choose a reason for hiding this comment

HuangFru commented Dec 6, 2023

zhongqishang commented Dec 6, 2023 • edited Loading

HuangFru commented Dec 6, 2023

wangtaohz commented Dec 6, 2023

baiyangtx left a comment

Choose a reason for hiding this comment

codecov bot commented Nov 15, 2023 •

edited

Loading

CLAassistant commented Nov 22, 2023 •

edited

Loading

zhongqishang commented Dec 6, 2023 •

edited

Loading