Skip to content

[Doc](partition) Refine some auto partition behaviours #488

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
luzhijing merged 1 commit into from
Mar 29, 2024
Merged

[Doc](partition) Refine some auto partition behaviours #488

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 10 additions & 27 deletions docs/advanced/partition/auto-partition.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -88,7 +88,7 @@ When building a table, use the following syntax to populate [CREATE-TABLE](../..
1. AUTO RANGE PARTITION:

```sql
AUTO PARTITION BY RANGE FUNC_CALL_EXPR
AUTO PARTITION BY RANGE (FUNC_CALL_EXPR)
(
)
```
Expand All @@ -97,6 +97,8 @@ When building a table, use the following syntax to populate [CREATE-TABLE](../..
FUNC_CALL_EXPR ::= date_trunc ( <partition_column>, '<interval>' )
```

Note: In version 2.1.0, `FUNC_CALL_EXPR` does not need to be surrounded by parentheses.

2. AUTO LIST PARTITION:

```sql
Expand All @@ -114,7 +116,7 @@ When building a table, use the following syntax to populate [CREATE-TABLE](../..
`TIME_STAMP` datev2 NOT NULL COMMENT 'Date of collection'
) ENGINE=OLAP
DUPLICATE KEY(`TIME_STAMP`)
AUTO PARTITION BY RANGE date_trunc(`TIME_STAMP`, 'month')
AUTO PARTITION BY RANGE (date_trunc(`TIME_STAMP`, 'month'))
(
)
DISTRIBUTED BY HASH(`TIME_STAMP`) BUCKETS 10
Expand Down Expand Up @@ -183,9 +185,7 @@ mysql> select * from auto_null_list partition(pX);
1 row in set (0.20 sec)
```

2. For AUTO RANGE PARTITION, the storage of NULL values requires the manual creation of specific partitions.

In Doris' RANGE PARTITION, we do not use a separate NULL PARTITION to contain NULL values, but rather group them in the **minium LESS THAN partition**. For logical clarity, the AUTO RANGE PARTITION does not automatically create such a partition when it encounters a NULL value. If necessary, you can create such a partition yourself to contain NULL values.
2. For AUTO RANGE PARTITION, **NULLABLE columns are not supported as partition columns**.

```sql
mysql> CREATE TABLE `range_table_nullable` (
Expand All @@ -194,31 +194,14 @@ mysql> CREATE TABLE `range_table_nullable` (
-> `k3` DATETIMEV2(6)
-> ) ENGINE=OLAP
-> DUPLICATE KEY(`k1`)
-> AUTO PARTITION BY RANGE date_trunc(`k2`, 'day')
-> AUTO PARTITION BY RANGE (date_trunc(`k2`, 'day'))
-> (
-> )
-> DISTRIBUTED BY HASH(`k1`) BUCKETS 16
-> PROPERTIES (
-> "replication_allocation" = "tag.location.default: 1"
-> );
Query OK, 0 rows affected (0.09 sec)

mysql> insert into range_table_nullable values (0, null, null);
ERROR 1105 (HY000): errCode = 2, detailMessage = [CANCELLED]TStatus: errCode = 2, detailMessage = Can't create partition for NULL Range

mysql> alter table range_table_nullable add partition pX VALUES LESS THAN ("1970-01-01");
Query OK, 0 rows affected (0.11 sec)

mysql> insert into range_table_nullable values (0, null, null);
Query OK, 1 row affected (0.18 sec)

mysql> select * from range_table_nullable;
+------+------+------+
| k1 | k2 | k3 |
+------+------+------+
| 0 | NULL | NULL |
+------+------+------+
1 row in set (0.18 sec)
ERROR 1105 (HY000): errCode = 2, detailMessage = AUTO RANGE PARTITION doesn't support NULL column
```

## Sample Scenarios
Expand All @@ -233,7 +216,7 @@ CREATE TABLE `DAILY_TRADE_VALUE`
......
)
UNIQUE KEY(`TRADE_DATE`, `TRADE_ID`)
AUTO PARTITION BY RANGE date_trunc(`TRADE_DATE`, 'year')
AUTO PARTITION BY RANGE (date_trunc(`TRADE_DATE`, 'year'))
(
)
DISTRIBUTED BY HASH(`TRADE_DATE`) BUCKETS 10
Expand Down Expand Up @@ -276,7 +259,7 @@ CREATE TABLE tbl3
k1 DATETIME NOT NULL,
col1 int
)
AUTO PARTITION BY RANGE date_trunc(`k1`, 'year') ()
AUTO PARTITION BY RANGE (date_trunc(`k1`, 'year') ())
DISTRIBUTED BY HASH(k1)
PROPERTIES
(
Expand Down Expand Up @@ -308,7 +291,7 @@ mysql > CREATE TABLE tbl3
k1 DATETIME NOT NULL,
col1 int
)
AUTO PARTITION BY RANGE date_trunc(`k1`, 'year') ()
AUTO PARTITION BY RANGE (date_trunc(`k1`, 'year') ())
DISTRIBUTED BY HASH(k1)
PROPERTIES
(
Expand Down
37 changes: 10 additions & 27 deletions ...-CN/docusaurus-plugin-content-docs/current/advanced/partition/auto-partition.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -88,7 +88,7 @@ PROPERTIES (
1. AUTO RANGE PARTITION:

```sql
AUTO PARTITION BY RANGE FUNC_CALL_EXPR
AUTO PARTITION BY RANGE (FUNC_CALL_EXPR)
(
)
```
Expand All @@ -97,6 +97,8 @@ PROPERTIES (
FUNC_CALL_EXPR ::= date_trunc ( <partition_column>, '<interval>' )
```

注意：在 2.1.0 版本，`FUNC_CALL_EXPR` 外围不需要被括号包围。

2. AUTO LIST PARTITION:

```sql
Expand All @@ -114,7 +116,7 @@ PROPERTIES (
`TIME_STAMP` datev2 NOT NULL COMMENT '采集日期'
) ENGINE=OLAP
DUPLICATE KEY(`TIME_STAMP`)
AUTO PARTITION BY RANGE date_trunc(`TIME_STAMP`, 'month')
AUTO PARTITION BY RANGE (date_trunc(`TIME_STAMP`, 'month'))
(
)
DISTRIBUTED BY HASH(`TIME_STAMP`) BUCKETS 10
Expand Down Expand Up @@ -183,9 +185,7 @@ mysql> select * from auto_null_list partition(pX);
1 row in set (0.20 sec)
```

2. 对于 AUTO RANGE PARTITION，NULL 值的存储需要手动创建特定的分区。

在 Doris 的 RANGE PARTITION 中，我们并不使用单独的 NULL PARTITION 来包含 NULL 值，而是将其归属于**最小的 LESS THAN 分区内**。为保持逻辑清晰，AUTO RANGE PARTITION 在遇到 NULL 值时不会自动创建这样的分区。如有需要，可以自行创建这样的分区，以包含 NULL 值。
2. 对于 AUTO RANGE PARTITION，**不支持 NULLABLE 列作为分区列**。

```sql
mysql> CREATE TABLE `range_table_nullable` (
Expand All @@ -194,31 +194,14 @@ mysql> CREATE TABLE `range_table_nullable` (
-> `k3` DATETIMEV2(6)
-> ) ENGINE=OLAP
-> DUPLICATE KEY(`k1`)
-> AUTO PARTITION BY RANGE date_trunc(`k2`, 'day')
-> AUTO PARTITION BY RANGE (date_trunc(`k2`, 'day'))
-> (
-> )
-> DISTRIBUTED BY HASH(`k1`) BUCKETS 16
-> PROPERTIES (
-> "replication_allocation" = "tag.location.default: 1"
-> );
Query OK, 0 rows affected (0.09 sec)

mysql> insert into range_table_nullable values (0, null, null);
ERROR 1105 (HY000): errCode = 2, detailMessage = [CANCELLED]TStatus: errCode = 2, detailMessage = Can't create partition for NULL Range

mysql> alter table range_table_nullable add partition pX VALUES LESS THAN ("1970-01-01");
Query OK, 0 rows affected (0.11 sec)

mysql> insert into range_table_nullable values (0, null, null);
Query OK, 1 row affected (0.18 sec)

mysql> select * from range_table_nullable;
+------+------+------+
| k1 | k2 | k3 |
+------+------+------+
| 0 | NULL | NULL |
+------+------+------+
1 row in set (0.18 sec)
ERROR 1105 (HY000): errCode = 2, detailMessage = AUTO RANGE PARTITION doesn't support NULL column
```

## 场景示例
Expand All @@ -233,7 +216,7 @@ CREATE TABLE `DAILY_TRADE_VALUE`
......
)
UNIQUE KEY(`TRADE_DATE`, `TRADE_ID`)
AUTO PARTITION BY RANGE date_trunc(`TRADE_DATE`, 'year')
AUTO PARTITION BY RANGE (date_trunc(`TRADE_DATE`, 'year'))
(
)
DISTRIBUTED BY HASH(`TRADE_DATE`) BUCKETS 10
Expand Down Expand Up @@ -276,7 +259,7 @@ CREATE TABLE tbl3
k1 DATETIME NOT NULL,
col1 int
)
AUTO PARTITION BY RANGE date_trunc(`k1`, 'year') ()
AUTO PARTITION BY RANGE (date_trunc(`k1`, 'year')) ()
DISTRIBUTED BY HASH(k1)
PROPERTIES
(
Expand Down Expand Up @@ -308,7 +291,7 @@ mysql > CREATE TABLE tbl3
k1 DATETIME NOT NULL,
col1 int
)
AUTO PARTITION BY RANGE date_trunc(`k1`, 'year') ()
AUTO PARTITION BY RANGE (date_trunc(`k1`, 'year')) ()
DISTRIBUTED BY HASH(k1)
PROPERTIES
(
Expand Down
39 changes: 10 additions & 29 deletions ...docusaurus-plugin-content-docs/version-2.1/advanced/partition/auto-partition.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -88,7 +88,7 @@ PROPERTIES (
1. AUTO RANGE PARTITION:

```sql
AUTO PARTITION BY RANGE FUNC_CALL_EXPR
AUTO PARTITION BY RANGE (FUNC_CALL_EXPR)
(
)
```
Expand All @@ -97,6 +97,8 @@ PROPERTIES (
FUNC_CALL_EXPR ::= date_trunc ( <partition_column>, '<interval>' )
```

注意：在 2.1.0 版本，`FUNC_CALL_EXPR` 外围不需要被括号包围。

2. AUTO LIST PARTITION:

```sql
Expand All @@ -114,7 +116,7 @@ PROPERTIES (
`TIME_STAMP` datev2 NOT NULL COMMENT '采集日期'
) ENGINE=OLAP
DUPLICATE KEY(`TIME_STAMP`)
AUTO PARTITION BY RANGE date_trunc(`TIME_STAMP`, 'month')
AUTO PARTITION BY RANGE (date_trunc(`TIME_STAMP`, 'month'))
(
)
DISTRIBUTED BY HASH(`TIME_STAMP`) BUCKETS 10
Expand Down Expand Up @@ -148,8 +150,6 @@ PROPERTIES (

### NULL 值分区

> 从 2.1.2 开始，Doris 支持以下功能。

当开启 session variable `allow_partition_column_nullable` 后，LIST 和 RANGE 分区都支持 NULL 列作为分区列。当分区列实际遇到 NULL 值的插入时：

1. 对于 AUTO LIST PARTITION，会自动创建对应的 NULL 值分区：
Expand Down Expand Up @@ -185,9 +185,7 @@ mysql> select * from auto_null_list partition(pX);
1 row in set (0.20 sec)
```

2. 对于 AUTO RANGE PARTITION，NULL 值的存储需要手动创建特定的分区。

在 Doris 的 RANGE PARTITION 中，我们并不使用单独的 NULL PARTITION 来包含 NULL 值，而是将其归属于**最小的 LESS THAN 分区内**。为保持逻辑清晰，AUTO RANGE PARTITION 在遇到 NULL 值时不会自动创建这样的分区。如有需要，可以自行创建这样的分区，以包含 NULL 值。
2. 对于 AUTO RANGE PARTITION，**不支持 NULLABLE 列作为分区列**。

```sql
mysql> CREATE TABLE `range_table_nullable` (
Expand All @@ -196,31 +194,14 @@ mysql> CREATE TABLE `range_table_nullable` (
-> `k3` DATETIMEV2(6)
-> ) ENGINE=OLAP
-> DUPLICATE KEY(`k1`)
-> AUTO PARTITION BY RANGE date_trunc(`k2`, 'day')
-> AUTO PARTITION BY RANGE (date_trunc(`k2`, 'day'))
-> (
-> )
-> DISTRIBUTED BY HASH(`k1`) BUCKETS 16
-> PROPERTIES (
-> "replication_allocation" = "tag.location.default: 1"
-> );
Query OK, 0 rows affected (0.09 sec)

mysql> insert into range_table_nullable values (0, null, null);
ERROR 1105 (HY000): errCode = 2, detailMessage = [CANCELLED]TStatus: errCode = 2, detailMessage = Can't create partition for NULL Range

mysql> alter table range_table_nullable add partition pX VALUES LESS THAN ("1970-01-01");
Query OK, 0 rows affected (0.11 sec)

mysql> insert into range_table_nullable values (0, null, null);
Query OK, 1 row affected (0.18 sec)

mysql> select * from range_table_nullable;
+------+------+------+
| k1 | k2 | k3 |
+------+------+------+
| 0 | NULL | NULL |
+------+------+------+
1 row in set (0.18 sec)
ERROR 1105 (HY000): errCode = 2, detailMessage = AUTO RANGE PARTITION doesn't support NULL column
```

## 场景示例
Expand All @@ -235,7 +216,7 @@ CREATE TABLE `DAILY_TRADE_VALUE`
......
)
UNIQUE KEY(`TRADE_DATE`, `TRADE_ID`)
AUTO PARTITION BY RANGE date_trunc(`TRADE_DATE`, 'year')
AUTO PARTITION BY RANGE (date_trunc(`TRADE_DATE`, 'year'))
(
)
DISTRIBUTED BY HASH(`TRADE_DATE`) BUCKETS 10
Expand Down Expand Up @@ -278,7 +259,7 @@ CREATE TABLE tbl3
k1 DATETIME NOT NULL,
col1 int
)
AUTO PARTITION BY RANGE date_trunc(`k1`, 'year') ()
AUTO PARTITION BY RANGE (date_trunc(`k1`, 'year')) ()
DISTRIBUTED BY HASH(k1)
PROPERTIES
(
Expand Down Expand Up @@ -310,7 +291,7 @@ mysql > CREATE TABLE tbl3
k1 DATETIME NOT NULL,
col1 int
)
AUTO PARTITION BY RANGE date_trunc(`k1`, 'year') ()
AUTO PARTITION BY RANGE (date_trunc(`k1`, 'year')) ()
DISTRIBUTED BY HASH(k1)
PROPERTIES
(
Expand Down
Loading