[AMORO-1870]: Support table filters to make AMS ignore tables that are not needed

[celltobig]

Why are the changes needed?

Resolve #1870.
AMS has a database filter, but not a table filter.

Brief change log

• Add table filter configuration for catalog

How was this patch tested?

• Add some test cases that check the changes thoroughly including negative and positive cases if possible

• Add screenshots for manual tests if appropriate

• Run test locally before making a pull request

Documentation

• Does this pull request introduce a new feature? (yes )
• If yes, how is the feature documented? (documented)

[wangtaohz]

Thanks for your work! You can add resolve #1879 to relate to the issue.

[celltobig]

[wangtaohz]

The code looks good to me, but there are some check style issues to fix.
You could run mvn checkstyle:check locally to check the code before committing. @celltobig

[codecov]

Codecov Report

Patch coverage is 12.50% of modified lines.

Files Changed	Coverage
.../netease/arctic/catalog/IcebergCatalogWrapper.java	12.50%

📢 Thoughts on this report? Let us know!.

[baiyangtx]

Thank you for submitting your Pull Request.

I'm considering using two different configuration items to decide the database and table filters separately, is it a good style? Can we use one configuration item to solve this problem?
For example, using a package name filter instead of a regular expression, such as ods_.user_

@zhoujinsong What's your point of view?

[wangtaohz]

It must be acknowledged that using 2 configurations cannot solve the situation where there are tables with the same name in different databases.
For example:
There are 4 tables in 2 databases, a.A,a.B,b.A,b.B. If we want to keep a.A,b.B, it is impossible to achieve it with two configurations.

Therefore, I think the introduced parameter should consider both the database name and the table name.

BTW, for compatibility reasons, the previous database.filter-regular-expression should not be immediately deleted.

[zhoujinsong]

Yes, there are indeed configuration complexities and limitations when using two expressions to filter databases and tables.

My idea is to make the configuration table.filter-regular-expression apply to both the database name and table name, like:

• db1\..+ means tables under database db1
• a\.A|b\.B means a.A and b.B

If designed in this way, table.filter-regular-expression can replace the existing database.filter-regular-expression, but we can keep a major version to be compatible with the old configuration.

HDYT? @wangtaohz @baiyangtx @celltobig

[wangtaohz]

I think it's a very feasible way to do it. However, there are two small issues:

1. The /. may be \. ? And it should be like

• db1\..+ means tables under database db1
• a\.A|b\.B means a.A and b.B

2.How about changing the table.filter-regular-expression to database.table.filter-regular-expression?
It may be clearer to indicate matching both the database name and table name

[zhoujinsong]

1. The /. may be \. ? And it should be like

Sure, I have changed it in the comment.

2.How about changing the table.filter-regular-expression to database.table.filter-regular-expression? It may be clearer to indicate matching both the database name and table name

I do not like the configuration name database.table.filter-regular-expression and it may be clear enough if we expose the only configuration table.filter-regular-expression in docs and add some explanation.

[wangtaohz]

LGTM