Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[feature](cold-hot) support s3 resource #8808

Merged
merged 6 commits into from
Apr 13, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions docs/.vuepress/sidebar/en.js
Original file line number Diff line number Diff line change
Expand Up @@ -621,6 +621,7 @@ module.exports = [
initialOpenGroupIndex: -1,
children: [
"ALTER DATABASE",
"ALTER RESOURCE",
"ALTER TABLE",
"ALTER VIEW",
"BACKUP",
Expand All @@ -632,6 +633,7 @@ module.exports = [
"CREATE INDEX",
"CREATE MATERIALIZED VIEW",
"CREATE REPOSITORY",
"CREATE RESOURCE",
"CREATE TABLE LIKE",
"CREATE TABLE",
"CREATE VIEW",
Expand All @@ -641,6 +643,7 @@ module.exports = [
"DROP INDEX",
"DROP MATERIALIZED VIEW",
"DROP REPOSITORY",
"DROP RESOURCE",
"DROP TABLE",
"DROP VIEW",
"HLL",
Expand All @@ -649,6 +652,7 @@ module.exports = [
"REFRESH TABLE",
"RESTORE",
"SHOW ENCRYPTKEYS",
"SHOW RESOURCES",
"TRUNCATE TABLE",
"create-function",
"drop-function",
Expand Down
1 change: 1 addition & 0 deletions docs/.vuepress/sidebar/zh-CN.js
Original file line number Diff line number Diff line change
Expand Up @@ -634,6 +634,7 @@ module.exports = [
initialOpenGroupIndex: -1,
children: [
"ALTER DATABASE",
"ALTER RESOURCE",
"ALTER TABLE",
"ALTER VIEW",
"BACKUP",
Expand Down
217 changes: 110 additions & 107 deletions docs/en/sql-reference/sql-statements/Administration/ALTER SYSTEM.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,114 +25,117 @@ under the License.
-->

# ALTER SYSTEM
## Description

This statement is used to operate on nodes in a system. (Administrator only!)
Grammar:
1) Adding nodes (without multi-tenant functionality, add in this way)
ALTER SYSTEM ADD BACKEND "host:heartbeat_port"[,"host:heartbeat_port"...];
2) Adding idle nodes (that is, adding BACKEND that does not belong to any cluster)
ALTER SYSTEM ADD FREE BACKEND "host:heartbeat_port"[,"host:heartbeat_port"...];
3) Adding nodes to a cluster
ALTER SYSTEM ADD BACKEND TO cluster_name "host:heartbeat_port"[,"host:heartbeat_port"...];
4) Delete nodes
ALTER SYSTEM DROP BACKEND "host:heartbeat_port"[,"host:heartbeat_port"...];
5) Node offline
ALTER SYSTEM DECOMMISSION BACKEND "host:heartbeat_port"[,"host:heartbeat_port"...];
6)226;- 21152;-Broker
ALTER SYSTEM ADD BROKER broker_name "host:port"[,"host:port"...];
(7) 20943;"23569;" Broker
ALTER SYSTEM DROP BROKER broker_name "host:port"[,"host:port"...];
8) Delete all Brokers
ALTER SYSTEM DROP ALL BROKER broker_name
9) Set up a Load error hub for centralized display of import error information
ALTER SYSTEM SET LOAD ERRORS HUB PROPERTIES ("key" = "value"[, ...]);
10) Modify property of BE
ALTER SYSTEM MODIFY BACKEND "host:heartbeat_port" SET ("key" = "value"[, ...]);

Explain:
1) Host can be hostname or IP address
2) heartbeat_port is the heartbeat port of the node
3) Adding and deleting nodes are synchronous operations. These two operations do not take into account the existing data on the node, the node is directly deleted from the metadata, please use cautiously.
4) Node offline operations are used to secure offline nodes. This operation is asynchronous. If successful, the node will eventually be removed from the metadata. If it fails, the offline will not be completed.
5) The offline operation of the node can be cancelled manually. See CANCEL DECOMMISSION for details
6) Load error hub:
Currently, two types of Hub are supported: Mysql and Broker. You need to specify "type" = "mysql" or "type" = "broker" in PROPERTIES.
If you need to delete the current load error hub, you can set type to null.
1) When using the Mysql type, the error information generated when importing will be inserted into the specified MySQL library table, and then the error information can be viewed directly through the show load warnings statement.

Hub of Mysql type needs to specify the following parameters:
host: mysql host
port: mysql port
user: mysql user
password: mysql password
database mysql database
table: mysql table

2) When the Broker type is used, the error information generated when importing will form a file and be written to the designated remote storage system through the broker. Make sure that the corresponding broker is deployed
Hub of Broker type needs to specify the following parameters:
Broker: Name of broker
Path: Remote Storage Path
Other properties: Other information necessary to access remote storage, such as authentication information.

7) Modify BE node attributes currently supports the following attributes:
1. tag.location:Resource tag
2. disable_query: Query disabled attribute
3. disable_load: Load disabled attribute

## example

1. Add a node
ALTER SYSTEM ADD BACKEND "host:port";

2. Adding an idle node
ALTER SYSTEM ADD FREE BACKEND "host:port";

3. Delete two nodes
ALTER SYSTEM DROP BACKEND "host1:port", "host2:port";

4. offline two nodes
ALTER SYSTEM DECOMMISSION BACKEND "host1:port", "host2:port";

5. Add two Hdfs Broker
ALTER SYSTEM ADD BROKER hdfs "host1:port", "host2:port";

6. Add a load error hub of Mysql type
ALTER SYSTEM SET LOAD ERRORS HUB PROPERTIES
("type"= "mysql",
"host" = "192.168.1.17"
"port" = "3306",
"User" = "my" name,
"password" = "my_passwd",
"database" = "doris_load",
"table" = "load_errors"
);

7. 添加一个 Broker 类型的 load error hub
ALTER SYSTEM SET LOAD ERRORS HUB PROPERTIES
("type"= "broker",
"Name" = BOS,
"path" = "bos://backup-cmy/logs",
"bos_endpoint" ="http://gz.bcebos.com",
"bos_accesskey" = "069fc278xxxxxx24ddb522",
"bos_secret_accesskey"="700adb0c6xxxxxx74d59eaa980a"
);

8. Delete the current load error hub
ALTER SYSTEM SET LOAD ERRORS HUB PROPERTIES
("type"= "null");

9. Modify BE resource tag

ALTER SYSTEM MODIFY BACKEND "host1:9050" SET ("tag.location" = "group_a");

10. Modify the query disabled attribute of BE

ALTER SYSTEM MODIFY BACKEND "host1:9050" SET ("disable_query" = "true");
## Description

11. Modify the load disabled attribute of BE

ALTER SYSTEM MODIFY BACKEND "host1:9050" SET ("disable_load" = "true");
This statement is used to operate on nodes in a system. (Administrator only!)

Syntax:
1) Adding nodes (without multi-tenant functionality, add in this way)
ALTER SYSTEM ADD BACKEND "host:heartbeat_port"[,"host:heartbeat_port"...];
2) Adding idle nodes (that is, adding BACKEND that does not belong to any cluster)
ALTER SYSTEM ADD FREE BACKEND "host:heartbeat_port"[,"host:heartbeat_port"...];
3) Adding nodes to a cluster
ALTER SYSTEM ADD BACKEND TO cluster_name "host:heartbeat_port"[,"host:heartbeat_port"...];
4) Delete nodes
ALTER SYSTEM DROP BACKEND "host:heartbeat_port"[,"host:heartbeat_port"...];
5) Node offline
ALTER SYSTEM DECOMMISSION BACKEND "host:heartbeat_port"[,"host:heartbeat_port"...];
6) Add Broker
ALTER SYSTEM ADD BROKER broker_name "host:port"[,"host:port"...];
7) Drop Broker
ALTER SYSTEM DROP BROKER broker_name "host:port"[,"host:port"...];
8) Delete all Brokers
ALTER SYSTEM DROP ALL BROKER broker_name
9) Set up a Load error hub for centralized display of import error information
ALTER SYSTEM SET LOAD ERRORS HUB PROPERTIES ("key" = "value"[, ...]);
10) Modify property of BE
ALTER SYSTEM MODIFY BACKEND "host:heartbeat_port" SET ("key" = "value"[, ...]);

Explain:
1) Host can be hostname or IP address
2) heartbeat_port is the heartbeat port of the node
3) Adding and deleting nodes are synchronous operations. These two operations do not take into account the existing data on the node, the node is directly deleted from the metadata, please use cautiously.
4) Node offline operations are used to secure offline nodes. This operation is asynchronous. If successful, the node will eventually be removed from the metadata. If it fails, the offline will not be completed.
5) The offline operation of the node can be cancelled manually. See CANCEL DECOMMISSION for details
6) Load error hub:
Currently, two types of Hub are supported: Mysql and Broker. You need to specify "type" = "mysql" or "type" = "broker" in PROPERTIES.
If you need to delete the current load error hub, you can set type to null.
1) When using the Mysql type, the error information generated when importing will be inserted into the specified MySQL library table, and then the error information can be viewed directly through the show load warnings statement.

Hub of Mysql type needs to specify the following parameters:
host: mysql host
port: mysql port
user: mysql user
password: mysql password
database mysql database
table: mysql table

2) When the Broker type is used, the error information generated when importing will form a file and be written to the designated remote storage system through the broker. Make sure that the corresponding broker is deployed
Hub of Broker type needs to specify the following parameters:
Broker: Name of broker
Path: Remote Storage Path
Other properties: Other information necessary to access remote storage, such as authentication information.

7) Modify BE node attributes currently supports the following attributes:
1. tag.location:Resource tag
2. disable_query: Query disabled attribute
3. disable_load: Load disabled attribute

## Example

1. Add a node
ALTER SYSTEM ADD BACKEND "host:port";

2. Adding an idle node
ALTER SYSTEM ADD FREE BACKEND "host:port";

3. Delete two nodes
ALTER SYSTEM DROP BACKEND "host1:port", "host2:port";

4. offline two nodes
ALTER SYSTEM DECOMMISSION BACKEND "host1:port", "host2:port";

5. Add two Hdfs Broker
ALTER SYSTEM ADD BROKER hdfs "host1:port", "host2:port";

6. Add a load error hub of Mysql type
ALTER SYSTEM SET LOAD ERRORS HUB PROPERTIES
("type"= "mysql",
"host" = "192.168.1.17"
"port" = "3306",
"User" = "my" name,
"password" = "my_passwd",
"database" = "doris_load",
"table" = "load_errors"
);

7. 添加一个 Broker 类型的 load error hub
ALTER SYSTEM SET LOAD ERRORS HUB PROPERTIES
("type"= "broker",
"Name" = BOS,
"path" = "bos://backup-cmy/logs",
"bos_endpoint" ="http://gz.bcebos.com",
"bos_accesskey" = "069fc278xxxxxx24ddb522",
"bos_secret_accesskey"="700adb0c6xxxxxx74d59eaa980a"
);

8. Delete the current load error hub
ALTER SYSTEM SET LOAD ERRORS HUB PROPERTIES
("type"= "null");

9. Modify BE resource tag

ALTER SYSTEM MODIFY BACKEND "host1:9050" SET ("tag.location" = "group_a");

10. Modify the query disabled attribute of BE

ALTER SYSTEM MODIFY BACKEND "host1:9050" SET ("disable_query" = "true");

11. Modify the load disabled attribute of BE

ALTER SYSTEM MODIFY BACKEND "host1:9050" SET ("disable_load" = "true");

## keyword
AGE,SYSTEM,BACKGROUND,BROKER,FREE

AGE, SYSTEM, BACKGROUND, BROKER, FREE
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
---
{
"title": "ALTER RESOURCE",
"language": "en"
}
---

<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->

# ALTER RESOURCE

## Description

This statement is used to modify an existing resource. Only the root or admin user can modify resources.
Syntax:
ALTER RESOURCE 'resource_name'
PROPERTIES ("key"="value", ...);

Note: The resource type does not support modification.

## Example

1. Modify the working directory of the Spark resource named spark0:
ALTER RESOURCE 'spark0' PROPERTIES ("working_dir" = "hdfs://127.0.0.1:10000/tmp/doris_new");

2. Modify the maximum number of connections to the S3 resource named remote_s3:
ALTER RESOURCE 'remote_s3' PROPERTIES ("s3_max_connections" = "100");

## keyword

ALTER, RESOURCE
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,7 @@ under the License.
1) The following attributes of the modified partition are currently supported.
- storage_medium
- storage_cooldown_time
- remote_storage_cooldown_time
- replication_num
— in_memory
2) For single-partition tables, partition_name is the same as the table name.
Expand Down
Loading