Skip to content

Forbidden creating table with dynamic partition when FE.config dynamic_partition_enable=false #5043

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
morningman merged 3 commits into from
Dec 16, 2020
Merged

Forbidden creating table with dynamic partition when FE.config dynamic_partition_enable=false #5043

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
ed62948
Forbidden creating table with dynamic partition when FE.config dynami…
EmmyMiao87 Dec 8, 2020
b0d9bd1
Forbidden date type of partition column when dynamic time unit is HOUR
EmmyMiao87 Dec 15, 2020
9799cca
Add unit test
EmmyMiao87 Dec 15, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 9 additions & 1 deletion docs/en/administrator-guide/dynamic-partition.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -83,7 +83,7 @@ The rules of dynamic partition are prefixed with `dynamic_partition.`:

The unit for dynamic partition scheduling. Can be specified as `HOUR`,`DAY`,` WEEK`, and `MONTH`, means to create or delete partitions by hour, day, week, and month, respectively.

When specified as `HOUR`, the suffix format of the dynamically created partition name is `yyyyMMddHH`, for example, `2020032501`.
When specified as `HOUR`, the suffix format of the dynamically created partition name is `yyyyMMddHH`, for example, `2020032501`. *When the time unit is HOUR, the data type of partition column cannot be DATE.*

When specified as `DAY`, the suffix format of the dynamically created partition name is `yyyyMMdd`, for example, `20200325`.

Expand Down Expand Up @@ -341,3 +341,11 @@ When dynamic partitioning feature is enabled, Doris will no longer allow users t
The dynamic partitioning feature can be disabled by executing `ALTER TABLE tbl_name SET ("dynamic_partition.enable" = "false") ` and converting it to a manual partition table.

When dynamic partitioning feature is disabled, Doris will no longer manage partitions automatically, and users will have to create or delete partitions manually by using `ALTER TABLE`.

## Common problem

1. After creating the dynamic partition table, it prompts ```Could not create table with dynamic partition when fe config dynamic_partition_enable is false```

Because the main switch of dynamic partition, that is, the configuration of FE ```dynamic_partition_enable``` is false, the dynamic partition table cannot be created.

At this time, please modify the FE configuration file, add a line ```dynamic_partition_enable=true```, and restart FE. Or execute the command ADMIN SET FRONTEND CONFIG ("dynamic_partition_enable" = "true") to turn on the dynamic partition switch.
5 changes: 3 additions & 2 deletions docs/en/sql-reference/sql-statements/Data Definition/CREATE TABLE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -253,7 +253,7 @@ Syntax:
```
PROPERTIES (
"dynamic_partition.enable" = "true|false",
"dynamic_partition.time_unit" = "DAY|WEEK|MONTH",
"dynamic_partition.time_unit" = "HOUR|DAY|WEEK|MONTH",
"dynamic_partitoin.end" = "${integer_value}",
"dynamic_partition.prefix" = "${string_value}",
"dynamic_partition.buckets" = "${integer_value}
Expand All @@ -262,7 +262,8 @@ Syntax:

Dynamic_partition. Enable: specifies whether dynamic partitioning at the table level is enabled

Dynamic_partition. Time_unit: used to specify the time unit for dynamically adding partitions, which can be selected as DAY, WEEK, and MONTH.
Dynamic_partition. Time_unit: used to specify the time unit for dynamically adding partitions, which can be selected as HOUR, DAY, WEEK, and MONTH.
Attention: When the time unit is HOUR, the data type of partition column cannot be DATE.

Dynamic_partition. End: used to specify the number of partitions created in advance

Expand Down
10 changes: 9 additions & 1 deletion docs/zh-CN/administrator-guide/dynamic-partition.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,7 +81,7 @@ under the License.

动态分区调度的单位。可指定为 `HOUR`、`DAY`、`WEEK`、`MONTH`。分别表示按天、按星期、按月进行分区创建或删除。

当指定为 `HOUR` 时，动态创建的分区名后缀格式为 `yyyyMMddHH`，例如`2020032501`。
当指定为 `HOUR` 时，动态创建的分区名后缀格式为 `yyyyMMddHH`，例如`2020032501`。小时为单位的分区列数据类型不能为 DATE。

当指定为 `DAY` 时，动态创建的分区名后缀格式为 `yyyyMMdd`，例如`20200325`。

Expand Down Expand Up @@ -339,3 +339,11 @@ mysql> SHOW DYNAMIC PARTITION TABLES;
通过执行 `ALTER TABLE tbl_name SET ("dynamic_partition.enable" = "false")` 即可关闭动态分区功能，将其转换为手动分区表。

关闭动态分区功能后，Doris 将不再自动管理分区，需要用户手动通过 `ALTER TABLE` 的方式创建或删除分区。

## 常见问题

1. 创建动态分区表后提示 ```Could not create table with dynamic partition when fe config dynamic_partition_enable is false```

由于动态分区的总开关，也就是 FE 的配置 ```dynamic_partition_enable``` 为 false，导致无法创建动态分区表。

这时候请修改 FE 的配置文件，增加一行 ```dynamic_partition_enable=true```，并重启 FE。或者执行命令 ADMIN SET FRONTEND CONFIG ("dynamic_partition_enable" = "true") 将动态分区开关打开即可。
5 changes: 3 additions & 2 deletions docs/zh-CN/sql-reference/sql-statements/Data Definition/CREATE TABLE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -276,14 +276,15 @@ under the License.
```
PROPERTIES (
"dynamic_partition.enable" = "true|false",
"dynamic_partition.time_unit" = "DAY|WEEK|MONTH",
"dynamic_partition.time_unit" = "HOUR|DAY|WEEK|MONTH",
"dynamic_partition.start" = "${integer_value}",
"dynamic_partitoin.end" = "${integer_value}",
"dynamic_partition.prefix" = "${string_value}",
"dynamic_partition.buckets" = "${integer_value}
```
dynamic_partition.enable: 用于指定表级别的动态分区功能是否开启。默认为 true。
dynamic_partition.time_unit: 用于指定动态添加分区的时间单位，可选择为DAY（天），WEEK(周)，MONTH（月）
dynamic_partition.time_unit: 用于指定动态添加分区的时间单位，可选择为HOUR（小时），DAY（天），WEEK(周)，MONTH（月）。
注意：以小时为单位的分区列，数据类型不能为 DATE。
dynamic_partition.start: 用于指定向前删除多少个分区。值必须小于0。默认为 Integer.MIN_VALUE。
dynamic_partition.end: 用于指定提前创建的分区数量。值必须大于0。
dynamic_partition.prefix: 用于指定创建的分区名前缀，例如分区名前缀为p，则自动创建分区名为p20200108
Expand Down
6 changes: 4 additions & 2 deletions fe/fe-core/src/main/java/org/apache/doris/catalog/Catalog.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5355,13 +5355,15 @@ public void replayRenameColumn(TableInfo tableInfo) throws DdlException {
throw new DdlException("not implemented");
}

public void modifyTableDynamicPartition(Database db, OlapTable table, Map<String, String> properties) throws DdlException {
public void modifyTableDynamicPartition(Database db, OlapTable table, Map<String, String> properties)
throws DdlException {
Map<String, String> logProperties = new HashMap<>(properties);
TableProperty tableProperty = table.getTableProperty();
if (tableProperty == null) {
DynamicPartitionUtil.checkAndSetDynamicPartitionProperty(table, properties);
} else {
Map<String, String> analyzedDynamicPartition = DynamicPartitionUtil.analyzeDynamicPartition(properties);
Map<String, String> analyzedDynamicPartition = DynamicPartitionUtil.
analyzeDynamicPartition(properties, table.getPartitionInfo());
tableProperty.modifyTableProperties(analyzedDynamicPartition);
tableProperty.buildDynamicProperty();
}
Expand Down
22 changes: 18 additions & 4 deletions fe/fe-core/src/main/java/org/apache/doris/catalog/TableProperty.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@

package org.apache.doris.catalog;

import org.apache.doris.common.Config;
import org.apache.doris.common.DdlException;
import org.apache.doris.common.FeConstants;
import org.apache.doris.common.io.Text;
import org.apache.doris.common.io.Writable;
Expand Down Expand Up @@ -56,7 +58,7 @@ public class TableProperty implements Writable {
* DEFAULT: depends on BE's config 'default_rowset_type'
* V1: alpha rowset
* V2: beta rowset
*
*
* This property should be set when creating the table, and can only be changed to V2 using Alter Table stmt.
*/
private TStorageFormat storageFormat = TStorageFormat.DEFAULT;
Expand All @@ -77,7 +79,7 @@ public static boolean isSamePrefixProperties(Map<String, String> properties, Str
public TableProperty buildProperty(short opCode) {
switch (opCode) {
case OperationType.OP_DYNAMIC_PARTITION:
buildDynamicProperty();
executeBuildDynamicProperty();
break;
case OperationType.OP_MODIFY_REPLICATION_NUM:
buildReplicationNum();
Expand All @@ -90,7 +92,19 @@ public TableProperty buildProperty(short opCode) {
}
return this;
}
public TableProperty buildDynamicProperty() {
public TableProperty buildDynamicProperty() throws DdlException {
if (properties.containsKey(DynamicPartitionProperty.ENABLE)
&& Boolean.valueOf(properties.get(DynamicPartitionProperty.ENABLE))
&& !Config.dynamic_partition_enable) {
throw new DdlException("Could not create table with dynamic partition "
+ "when fe config dynamic_partition_enable is false. "
+ "Please ADMIN SET FRONTEND CONFIG (\"dynamic_partition_enable\" = \"true\") firstly.");
}
executeBuildDynamicProperty();
return this;
}

private TableProperty executeBuildDynamicProperty() {
HashMap<String, String> dynamicPartitionProperties = new HashMap<>();
for (Map.Entry<String, String> entry : properties.entrySet()) {
if (entry.getKey().startsWith(DYNAMIC_PARTITION_PROPERTY_PREFIX)) {
Expand Down Expand Up @@ -153,7 +167,7 @@ public void write(DataOutput out) throws IOException {

public static TableProperty read(DataInput in) throws IOException {
return GsonUtils.GSON.fromJson(Text.readString(in), TableProperty.class)
.buildDynamicProperty()
.executeBuildDynamicProperty()
.buildReplicationNum()
.buildInMemory()
.buildStorageFormat();
Expand Down
68 changes: 40 additions & 28 deletions fe/fe-core/src/main/java/org/apache/doris/common/util/DynamicPartitionUtil.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,14 +60,24 @@ public class DynamicPartitionUtil {
private static final String DATE_FORMAT = "yyyy-MM-dd";
private static final String DATETIME_FORMAT = "yyyy-MM-dd HH:mm:ss";

public static void checkTimeUnit(String timeUnit) throws DdlException {
public static void checkTimeUnit(String timeUnit, PartitionInfo partitionInfo) throws DdlException {
if (Strings.isNullOrEmpty(timeUnit)
|| !(timeUnit.equalsIgnoreCase(TimeUnit.DAY.toString())
|| timeUnit.equalsIgnoreCase(TimeUnit.HOUR.toString())
|| timeUnit.equalsIgnoreCase(TimeUnit.WEEK.toString())
|| timeUnit.equalsIgnoreCase(TimeUnit.MONTH.toString()))) {
ErrorReport.reportDdlException(ErrorCode.ERROR_DYNAMIC_PARTITION_TIME_UNIT, timeUnit);
}
Preconditions.checkState(partitionInfo instanceof RangePartitionInfo);
RangePartitionInfo rangePartitionInfo = (RangePartitionInfo) partitionInfo;
Preconditions.checkState(!rangePartitionInfo.isMultiColumnPartition());
Column partitionColumn = rangePartitionInfo.getPartitionColumns().get(0);
if ((partitionColumn.getDataType() == PrimitiveType.DATE)
&& (timeUnit.equalsIgnoreCase(TimeUnit.HOUR.toString()))) {
ErrorReport.reportDdlException(DynamicPartitionProperty.TIME_UNIT + " could not be "
+ TimeUnit.HOUR.toString() + " when type of partition column "
+ partitionColumn.getDisplayName() + " is " + PrimitiveType.DATE.toString());
}
}

private static void checkPrefix(String prefix) throws DdlException {
Expand Down Expand Up @@ -142,8 +152,8 @@ private static void checkStartDayOfWeek(String val) throws DdlException {
throw new DdlException("Invalid properties: " + DynamicPartitionProperty.START_DAY_OF_WEEK);
}
try {
int dayOfWeek= Integer.parseInt(val);
if (dayOfWeek< 1 || dayOfWeek > 7) {
int dayOfWeek = Integer.parseInt(val);
if (dayOfWeek < 1 || dayOfWeek > 7) {
throw new DdlException(DynamicPartitionProperty.START_DAY_OF_WEEK + " should between 1 and 7");
}
} catch (NumberFormatException e) {
Expand Down Expand Up @@ -180,7 +190,7 @@ public static boolean checkDynamicPartitionPropertiesExist(Map<String, String> p
properties.containsKey(DynamicPartitionProperty.START_DAY_OF_MONTH);
}

public static boolean checkInputDynamicPartitionProperties(Map<String, String> properties, PartitionInfo partitionInfo) throws DdlException{
public static boolean checkInputDynamicPartitionProperties(Map<String, String> properties, PartitionInfo partitionInfo) throws DdlException {
if (properties == null || properties.isEmpty()) {
return false;
}
Expand All @@ -194,13 +204,13 @@ public static boolean checkInputDynamicPartitionProperties(Map<String, String> p
String end = properties.get(DynamicPartitionProperty.END);
String buckets = properties.get(DynamicPartitionProperty.BUCKETS);
String enable = properties.get(DynamicPartitionProperty.ENABLE);
if (!((Strings.isNullOrEmpty(enable) &&
if (!(Strings.isNullOrEmpty(enable) &&
Strings.isNullOrEmpty(timeUnit) &&
Strings.isNullOrEmpty(timeZone) &&
Strings.isNullOrEmpty(prefix) &&
Strings.isNullOrEmpty(start) &&
Strings.isNullOrEmpty(end) &&
Strings.isNullOrEmpty(buckets)))) {
Strings.isNullOrEmpty(buckets))) {
if (Strings.isNullOrEmpty(enable)) {
properties.put(DynamicPartitionProperty.ENABLE, "true");
}
Expand Down Expand Up @@ -237,12 +247,12 @@ public static void registerOrRemoveDynamicPartitionTable(long dbId, OlapTable ol
}
}

public static Map<String, String> analyzeDynamicPartition(Map<String, String> properties) throws DdlException {
public static Map<String, String> analyzeDynamicPartition(Map<String, String> properties, PartitionInfo partitionInfo) throws DdlException {
// properties should not be empty, check properties before call this function
Map<String, String> analyzedProperties = new HashMap<>();
if (properties.containsKey(DynamicPartitionProperty.TIME_UNIT)) {
String timeUnitValue = properties.get(DynamicPartitionProperty.TIME_UNIT);
checkTimeUnit(timeUnitValue);
checkTimeUnit(timeUnitValue, partitionInfo);
properties.remove(DynamicPartitionProperty.TIME_UNIT);
analyzedProperties.put(DynamicPartitionProperty.TIME_UNIT, timeUnitValue);
}
Expand Down Expand Up @@ -277,14 +287,14 @@ public static Map<String, String> analyzeDynamicPartition(Map<String, String> pr
properties.remove(DynamicPartitionProperty.START);
analyzedProperties.put(DynamicPartitionProperty.START, startValue);
}

if (properties.containsKey(DynamicPartitionProperty.START_DAY_OF_MONTH)) {
String val = properties.get(DynamicPartitionProperty.START_DAY_OF_MONTH);
checkStartDayOfMonth(val);
properties.remove(DynamicPartitionProperty.START_DAY_OF_MONTH);
analyzedProperties.put(DynamicPartitionProperty.START_DAY_OF_MONTH, val);
}

if (properties.containsKey(DynamicPartitionProperty.START_DAY_OF_WEEK)) {
String val = properties.get(DynamicPartitionProperty.START_DAY_OF_WEEK);
checkStartDayOfWeek(val);
Expand Down Expand Up @@ -334,9 +344,11 @@ public static boolean isDynamicPartitionTable(Table table) {
/**
* properties should be checked before call this method
*/
public static void checkAndSetDynamicPartitionProperty(OlapTable olapTable, Map<String, String> properties) throws DdlException {
public static void checkAndSetDynamicPartitionProperty(OlapTable olapTable, Map<String, String> properties)
throws DdlException {
if (DynamicPartitionUtil.checkInputDynamicPartitionProperties(properties, olapTable.getPartitionInfo())) {
Map<String, String> dynamicPartitionProperties = DynamicPartitionUtil.analyzeDynamicPartition(properties);
Map<String, String> dynamicPartitionProperties =
DynamicPartitionUtil.analyzeDynamicPartition(properties, olapTable.getPartitionInfo());
TableProperty tableProperty = olapTable.getTableProperty();
if (tableProperty != null) {
tableProperty.modifyTableProperties(dynamicPartitionProperties);
Expand Down Expand Up @@ -391,7 +403,7 @@ public static String getFormattedPartitionName(TimeZone tz, String formattedDate
// add support: HOUR by caoyang10
// TODO: support YEAR
public static String getPartitionRangeString(DynamicPartitionProperty property, ZonedDateTime current,
int offset, String format) {
int offset, String format) {
String timeUnit = property.getTimeUnit();
if (timeUnit.equalsIgnoreCase(TimeUnit.DAY.toString())) {
return getPartitionRangeOfDay(current, offset, format);
Expand All @@ -408,25 +420,25 @@ public static String getPartitionRangeString(DynamicPartitionProperty property,
* return formatted string of partition range in HOUR granularity.
* offset: The offset from the current hour. 0 means current hour, 1 means next hour, -1 last hour.
* format: the format of the return date string.
*
* <p>
* Eg:
* Today is 2020-05-24 00:12:34, offset = -1
* It will return 2020-05-23 23:00:00
* Today is 2020-05-24 00, offset = 1
* It will return 2020-05-24 01:00:00
* Today is 2020-05-24 00:12:34, offset = -1
* It will return 2020-05-23 23:00:00
* Today is 2020-05-24 00, offset = 1
* It will return 2020-05-24 01:00:00
*/
private static String getPartitionRangeOfHour(ZonedDateTime current, int offset, String format) {
return getFormattedTimeWithoutMinuteSecond(current.plusHours(offset), format);
}

/**
* return formatted string of partition range in DAY granularity.
* offset: The offset from the current day. 0 means current day, 1 means tomorrow, -1 means yesterday.
* format: the format of the return date string.
*
* <p>
* Eg:
* Today is 2020-05-24, offset = -1
* It will return 2020-05-23
* Today is 2020-05-24, offset = -1
* It will return 2020-05-23
*/
private static String getPartitionRangeOfDay(ZonedDateTime current, int offset, String format) {
return getFormattedTimeWithoutHourMinuteSecond(current.plusDays(offset), format);
Expand All @@ -437,10 +449,10 @@ private static String getPartitionRangeOfDay(ZonedDateTime current, int offset,
* offset: The offset from the current week. 0 means current week, 1 means next week, -1 means last week.
* startOf: Define the start day of each week. 1 means MONDAY, 7 means SUNDAY.
* format: the format of the return date string.
*
* <p>
* Eg:
* Today is 2020-05-24, offset = -1, startOf.dayOfWeek = 3
* It will return 2020-05-20 (Wednesday of last week)
* Today is 2020-05-24, offset = -1, startOf.dayOfWeek = 3
* It will return 2020-05-20 (Wednesday of last week)
*/
private static String getPartitionRangeOfWeek(ZonedDateTime current, int offset, StartOfDate startOf, String format) {
Preconditions.checkArgument(startOf.isStartOfWeek());
Expand All @@ -457,10 +469,10 @@ private static String getPartitionRangeOfWeek(ZonedDateTime current, int offset,
* offset: The offset from the current month. 0 means current month, 1 means next month, -1 means last month.
* startOf: Define the start date of each month. 1 means start on the 1st of every month.
* format: the format of the return date string.
*
* <p>
* Eg:
* Today is 2020-05-24, offset = 1, startOf.month = 3
* It will return 2020-06-03
* Today is 2020-05-24, offset = 1, startOf.month = 3
* It will return 2020-06-03
*/
private static String getPartitionRangeOfMonth(ZonedDateTime current, int offset, StartOfDate startOf, String format) {
Preconditions.checkArgument(startOf.isStartOfMonth());
Expand Down
Loading