[Feature]: Support managing tables with multiple formats under one Catalog

[baiyangtx]

Description

Currently, Table formats are strongly correlated with Catalog type, which makes the Arctic hard to use in some production situations.

We should support managing all types of table formats if the catalog type is able to create the table.

Use case/motivation

Catalog(type=hive)support TableTypes:

• Hive: native hive table
• Iceberg: native iceberg table which uses org.apache.iceberg.hive.HiveCatalog as implement.
• Mixed-Hive: Mixed Streaming format. the base store is a Hive table.
• Mixed-Iceberg: Mixed Streaming format, the base store and change store are iceberg tables created by HadoopTables.

Catalog(type=arctic) support TableTypes:

• Iceberg: native iceberg table which uses specific iceberg catalog class as implement.
• Mixed-Iceberg: Mixed Streaming format, the base store and change store are iceberg tables created by HadoopTables.

RoadMap

Milestone 0.5.0

Introduce the concept of the InternalCatalog and ExternalCatalog

• [Subtask]: Introdue the concept of the InternalCatalog and ExternalCatalog #1544

Implement the Mixed-Iceberg format above on the Iceberg format in any catalog type

• [Subtask]: Remove dir operation from ArcticFileIo in order to support S3 storage. #1362

Make AMS as an Iceberg Catalog provider via implement RestCatalogAPI

• [Subtask]: Make AMS as an Iceberg Catalog provider via implement RestCatalogAPI #1339

Milestone 0.5.0 later

support mixed-hive and mixed-iceberg in same catalog(Milestone 0.5.0 later)

• [Subtask]: Support creating mixed-iceberg format tables in any catalog which support iceberg. #1336
• [Subtask]: Do not store table metadata in AMS for MIXED-HIVE #1337
• [Subtask]: Support s3 for mixed-iceberg in InternalCatalog. #2156

Support all types of table formats in the same internal catalog.

• [Subtask]: Support iceberg format and mixed-iceberg format in InternalCatalog #1338

Make engines support unified catalog

• [Subtask][Spark]: Support UnifiedCatalog in Spark Engine. #2276
• [Subtask][Flink]: Support UnifiedCatalog to contain Mixed format table in Flink Engine  #1341
• [Subtask][Flink]: Support UnifiedCatalog to contain native Iceberg format table in Flink Engine #2344
• [Subtask][Flink]: Support UnifiedCatalog to contain Paimon format table in Flink Engine #2345
• [Subtask][Trino]: Support UnifiedCatalog in Trino Engine.  #1343

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[baiyangtx]

The design doc is here. https://docs.google.com/document/d/1Q5O2WAlHqGiWM43GbJoDvmoiejAy5DZUpXmIliYLP0k/edit#