Skip to content

Commit

Permalink
[deprecated](external) remove deprecated hudi and iceberg external ta…
Browse files Browse the repository at this point in the history
…ble (apache#27456)

The creation of hudi and iceberg table is disallowed since v1.2.
All these features are covered by hudi/iceberg catalog.
We should remove the code in v2.1

The PR mainly changes:
1. remove the code of hudi/iceberg external table.
2. remove code of iceberg database.
3. disallowed hive external table's creation.
4. disabled odbc,mysql,broker external table by default, and add FE config `disable_odbc_mysql_broker_table` to control it
  • Loading branch information
morningman authored Nov 24, 2023
1 parent 59efb1b commit 6142a53
Show file tree
Hide file tree
Showing 41 changed files with 48 additions and 2,508 deletions.
49 changes: 2 additions & 47 deletions docs/en/docs/admin-manual/config/fe-config.md
Original file line number Diff line number Diff line change
Expand Up @@ -2296,60 +2296,15 @@ MasterOnly:false

multi catalog concurrent file scan size

#### `enable_odbc_table`
#### `enable_odbc_mysql_broker_table`

Default:false

IsMutable:true

MasterOnly:true

Whether to enable the ODBC table, it is not enabled by default. You need to manually configure it when you use it.

This parameter can be set by: ADMIN SET FRONTEND CONFIG("key"="value")

**Note:** This parameter has been deleted in version 1.2. The ODBC External Table is enabled by default, and the ODBC External Table will be deleted in a later version. It is recommended to use the JDBC External Table

#### `disable_iceberg_hudi_table`

Default:true

IsMutable:true

MasterOnly:false

Starting from version 1.2, we no longer support create hudi and iceberg External Table. Please use the multi catalog.

#### `iceberg_table_creation_interval_second`

Default:10 (s)

IsMutable:true

MasterOnly:false

fe will create iceberg table every iceberg_table_creation_interval_second

#### `iceberg_table_creation_strict_mode`

Default:true

IsMutable:true

MasterOnly:true

If set to TRUE, the column definitions of iceberg table and the doris table must be consistent
If set to FALSE, Doris only creates columns of supported data types.

#### `max_iceberg_table_creation_record_size`

Default max number of recent iceberg database table creation record that can be stored in memory.

Default:2000

IsMutable:true

MasterOnly:true
Starting from version 2.1, we no longer support create odbc, jdbc and broker external table. For odbc and mysql external table, use jdbc table or jdbc catalog instead. For broker table, use table valued function instead.

#### `max_hive_partition_cache_num`

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -43,22 +43,6 @@ CREATE DATABASE [IF NOT EXISTS] db_name

`PROPERTIES` Additional information about the database, which can be defaulted.

- If you create an Iceberg database, you need to provide the following information in properties:

```sql
PROPERTIES (
"iceberg.database" = "iceberg_db_name",
"iceberg.hive.metastore.uris" = "thrift://127.0.0.1:9083",
"iceberg.catalog.type" = "HIVE_CATALOG"
)
````
illustrate:
- `ceberg.database` : the library name corresponding to Iceberg;
- `iceberg.hive.metastore.uris` : hive metastore service address;
- `iceberg.catalog.type`: The default is `HIVE_CATALOG`; currently only `HIVE_CATALOG` is supported, and more Iceberg catalog types will be supported in the future.
- If you want to specify the default replica distribution for tables in db, you need to specify `replication_allocation` (the `replication_allocation` attribute of table will have higher priority than db)

```sql
Expand All @@ -75,14 +59,12 @@ CREATE DATABASE [IF NOT EXISTS] db_name
CREATE DATABASE db_test;
````
2. Create a new Iceberg database iceberg_test
2. Create a new database with default replica distribution:
```sql
CREATE DATABASE `iceberg_test`
PROPERTIES (
"iceberg.database" = "doris",
"iceberg.hive.metastore.uris" = "thrift://127.0.0.1:9083",
"iceberg.catalog.type" = "HIVE_CATALOG"
"replication_allocation" = "tag.location.group_1:3"
);
````

Expand Down

This file was deleted.

3 changes: 1 addition & 2 deletions docs/sidebars.json
Original file line number Diff line number Diff line change
Expand Up @@ -909,7 +909,6 @@
"sql-manual/sql-reference/Data-Definition-Statements/Create/CREATE-TABLE",
"sql-manual/sql-reference/Data-Definition-Statements/Create/CREATE-TABLE-LIKE",
"sql-manual/sql-reference/Data-Definition-Statements/Create/CREATE-TABLE-AS-SELECT",
"sql-manual/sql-reference/Data-Definition-Statements/Create/CREATE-EXTERNAL-TABLE",
"sql-manual/sql-reference/Data-Definition-Statements/Create/CREATE-INDEX",
"sql-manual/sql-reference/Data-Definition-Statements/Create/CREATE-VIEW",
"sql-manual/sql-reference/Data-Definition-Statements/Create/CREATE-MATERIALIZED-VIEW",
Expand Down Expand Up @@ -1342,4 +1341,4 @@
]
}
]
}
}
Loading

0 comments on commit 6142a53

Please sign in to comment.