-
Notifications
You must be signed in to change notification settings - Fork 760
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
feat(cluster): support system-managed cluster (#17051)
* feat(cluster): support custom management cluster * feat(cluster): support custom management cluster * feat(cluster): support custom management cluster * feat(cluster): support custom management cluster * feat(cluster): support custom management cluster * feat(cluster): support custom management cluster * feat(cluster): support custom management cluster * feat(cluster): support custom management cluster * feat(cluster): support custom management cluster * feat(cluster): support custom management cluster * feat(cluster): support custom management cluster * feat(cluster): support custom management cluster * feat(cluster): support custom management cluster * feat(cluster): support custom management cluster * feat(cluster): support custom management cluster * feat(cluster): support custom management cluster * feat(cluster): support custom management cluster * feat(cluster): support custom management cluster * feat(cluster): support custom management cluster * feat(cluster): support custom management cluster * feat(cluster): support custom management cluster * feat(cluster): support custom management cluster * feat(cluster): support custom management cluster * feat(cluster): support custom management cluster * feat(cluster): support custom management cluster * feat(cluster): support custom management cluster * feat(cluster): support custom management cluster * feat(cluster): support custom management cluster * feat(cluster): support custom management cluster * feat(cluster): support custom management cluster * feat(cluster): support custom management cluster * feat(cluster): support custom management cluster * feat(cluster): support custom management cluster * feat(cluster): support custom management cluster * refactor: improve and clean up `warehouse_mgr::upsert_self_managed()` ### Improvements: - Added retry mechanism with a fallback in the retry loop. - Return an error if an unexpected response is received when `TxnGetResponse` is expected. - Refined quit-retry condition: now only triggered when the seq of `NodeInfo` changes. ### Refactoring: - Simplified and decoupled nested branching for better readability and maintainability. - Consolidated related logic, e.g., building `txn if_then` operations in a single place. - Differentiated `NodeInfo` with and without warehouse-related information. ### Documentation: - Added details explaining behavioral differences between insert and update modes. * feat(cluster): clean code * feat(cluster): clean code * feat(cluster): clean code * feat(cluster): clean code * feat(cluster): clean code * feat(cluster): clean code * feat(cluster): clean code * feat(cluster): clean code * feat(cluster): clean code * feat(cluster): clean code * feat(cluster): clean code * feat(cluster): clean code * feat(cluster): clean code * feat(cluster): clean code * feat(cluster): add concurrent unit test * feat(cluster): add concurrent unit test * feat(cluster): clean code * feat(cluster): clean code --------- Co-authored-by: 张炎泼 <drdr.xp@gmail.com>
- Loading branch information
1 parent
783d155
commit 6362ff4
Showing
83 changed files
with
7,228 additions
and
627 deletions.
There are no files selected for viewing
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
96 changes: 96 additions & 0 deletions
96
scripts/ci/deploy/config/databend-query-node-system-managed.toml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,96 @@ | ||
# Usage: | ||
# databend-query -c databend_query_config_spec.toml | ||
|
||
[query] | ||
max_active_sessions = 256 | ||
shutdown_wait_timeout_ms = 5000 | ||
|
||
# For flight rpc. | ||
flight_api_address = "0.0.0.0:flight_port" | ||
|
||
# Databend Query http address. | ||
# For admin RESET API. | ||
admin_api_address = "0.0.0.0:admin_api_port" | ||
|
||
# Databend Query metrics RESET API. | ||
metric_api_address = "0.0.0.0:metric_api_port" | ||
|
||
# Databend Query MySQL Handler. | ||
mysql_handler_host = "0.0.0.0" | ||
mysql_handler_port = mysql_port | ||
|
||
# Databend Query ClickHouse Handler. | ||
clickhouse_http_handler_host = "0.0.0.0" | ||
clickhouse_http_handler_port = clickhouse_port | ||
|
||
# Databend Query HTTP Handler. | ||
http_handler_host = "0.0.0.0" | ||
http_handler_port = http_port | ||
|
||
# Databend Query FlightSQL Handler. | ||
flight_sql_handler_host = "0.0.0.0" | ||
flight_sql_handler_port = flight_sql_port | ||
|
||
tenant_id = "test_tenant" | ||
|
||
table_engine_memory_enabled = true | ||
default_storage_format = 'parquet' | ||
default_compression = 'zstd' | ||
|
||
[[query.users]] | ||
name = "root" | ||
auth_type = "no_password" | ||
|
||
[[query.users]] | ||
name = "default" | ||
auth_type = "no_password" | ||
|
||
# This for test | ||
[[query.udfs]] | ||
name = "ping" | ||
definition = "CREATE FUNCTION ping(STRING) RETURNS STRING LANGUAGE python HANDLER = 'ping' ADDRESS = 'http://0.0.0.0:8815'" | ||
|
||
[query.resources_management] | ||
type = "system_managed" | ||
node_group | ||
|
||
[log] | ||
|
||
[log.file] | ||
level = "INFO" | ||
format = "text" | ||
dir = "./.databend/query_logs" | ||
prefix_filter = "" | ||
|
||
[meta] | ||
# It is a list of `grpc_api_advertise_host:<grpc-api-port>` of databend-meta config | ||
endpoints = ["0.0.0.0:9191"] | ||
username = "root" | ||
password = "root" | ||
client_timeout_in_second = 60 | ||
auto_sync_interval = 60 | ||
|
||
# Storage config. | ||
[storage] | ||
# fs | s3 | azblob | obs | oss | ||
type = "fs" | ||
|
||
# Set a local folder to store your data. | ||
# Comment out this block if you're NOT using local file system as storage. | ||
[storage.fs] | ||
data_path = "./.databend/stateless_test_data" | ||
|
||
# Cache config. | ||
[cache] | ||
# Type of storage to keep the table data cache | ||
# | ||
# available options: [none|disk] | ||
# default is "none", which disable table data cache | ||
# use "disk" to enabled disk cache | ||
data_cache_storage = "none" | ||
|
||
[cache.disk] | ||
# cache path | ||
path = "./.databend/_cache" | ||
# max bytes of cached data 20G | ||
max_bytes = 21474836480 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,135 @@ | ||
#!/bin/bash | ||
# Copyright 2022 The Databend Authors. | ||
# SPDX-License-Identifier: Apache-2.0. | ||
|
||
set -e | ||
|
||
SCRIPT_PATH="$(cd "$(dirname "$0")" >/dev/null 2>&1 && pwd)" | ||
cd "$SCRIPT_PATH/../../.." || exit | ||
BUILD_PROFILE=${BUILD_PROFILE:-debug} | ||
|
||
if [ $# -eq 1 ]; then | ||
num=$1 | ||
node_group="" | ||
elif [ $# -eq 2 ]; then | ||
num=$1 | ||
node_group=$2 | ||
else | ||
echo "Usage: $0 <number> - Start number of databend-query with system-managed mode" | ||
exit 1 | ||
fi | ||
|
||
if ! [[ "$num" =~ ^[0-9]*$ ]]; then | ||
echo "Error: Argument must be an integer." | ||
exit 1 | ||
fi | ||
|
||
# Caveat: has to kill query first. | ||
# `query` tries to remove its liveness record from meta before shutting down. | ||
# If meta is stopped, `query` will receive an error that hangs graceful | ||
# shutdown. | ||
killall databend-query || true | ||
sleep 3 | ||
|
||
killall databend-meta || true | ||
sleep 3 | ||
|
||
for bin in databend-query databend-meta; do | ||
if test -n "$(pgrep $bin)"; then | ||
echo "The $bin is not killed. force killing." | ||
killall -9 $bin || true | ||
fi | ||
done | ||
|
||
# Wait for killed process to cleanup resources | ||
sleep 1 | ||
|
||
echo 'Start Meta service HA cluster(3 nodes)...' | ||
|
||
mkdir -p ./.databend/ | ||
|
||
nohup ./target/${BUILD_PROFILE}/databend-meta -c scripts/ci/deploy/config/databend-meta-node-1.toml >./.databend/meta-1.out 2>&1 & | ||
python3 scripts/ci/wait_tcp.py --timeout 30 --port 9191 | ||
|
||
# wait for cluster formation to complete. | ||
sleep 1 | ||
|
||
nohup ./target/${BUILD_PROFILE}/databend-meta -c scripts/ci/deploy/config/databend-meta-node-2.toml >./.databend/meta-2.out 2>&1 & | ||
python3 scripts/ci/wait_tcp.py --timeout 30 --port 28202 | ||
|
||
# wait for cluster formation to complete. | ||
sleep 1 | ||
|
||
nohup ./target/${BUILD_PROFILE}/databend-meta -c scripts/ci/deploy/config/databend-meta-node-3.toml >./.databend/meta-3.out 2>&1 & | ||
python3 scripts/ci/wait_tcp.py --timeout 30 --port 28302 | ||
|
||
# wait for cluster formation to complete. | ||
sleep 1 | ||
|
||
find_available_port() { | ||
local base_port=20000 | ||
local max_port=65535 | ||
local attempts=10 | ||
|
||
for ((i=0; i<attempts; i++)); do | ||
port=$(( RANDOM % (max_port - base_port + 1) + base_port )) | ||
if ! lsof -i :$port >/dev/null 2>&1; then | ||
echo $port | ||
return | ||
fi | ||
done | ||
|
||
echo "Unable to find an available port after $attempts attempts" >&2 | ||
exit 1 | ||
} | ||
|
||
|
||
start_databend_query() { | ||
local http_port=$1 | ||
local mysql_port=$2 | ||
local log_dir=$3 | ||
local node_group=$4 | ||
system_managed_config="./scripts/ci/deploy/config/databend-query-node-system-managed.toml" | ||
|
||
temp_file=$(mktemp) | ||
|
||
if [ -f "$system_managed_config" ]; then | ||
sed -e "s/flight_port/$(find_available_port)/g" \ | ||
-e "s/admin_api_port/$(find_available_port)/g" \ | ||
-e "s/metric_api_port/$(find_available_port)/g" \ | ||
-e "s/mysql_port/${mysql_port}/g" \ | ||
-e "s/clickhouse_port/$(find_available_port)/g" \ | ||
-e "s/http_port/${http_port}/g" \ | ||
-e "s/flight_sql_port/$(find_available_port)/g" \ | ||
-e "s/query_logs/${log_dir}/g" \ | ||
-e "s/node_group/node_group=\"${node_group}\"/g" \ | ||
"$system_managed_config" > "$temp_file" | ||
|
||
if [ $? -eq 0 ]; then | ||
echo "Start databend-query on port $http_port..." | ||
nohup target/${BUILD_PROFILE}/databend-query -c $temp_file --internal-enable-sandbox-tenant & | ||
|
||
echo "Waiting on databend-query 10 seconds..." | ||
python3 scripts/ci/wait_tcp.py --timeout 30 --port $http_port | ||
else | ||
echo "Error occurred during port replacement." | ||
rm -f "$temp_file" | ||
exit 1 | ||
fi | ||
else | ||
echo "Error: system-managed config file is not exists." | ||
exit 1 | ||
fi | ||
} | ||
|
||
if ! lsof -i :8000 >/dev/null 2>&1; then | ||
start_databend_query 8000 3307 "logs_1" $node_group | ||
num=$(( num - 1 )) | ||
fi | ||
|
||
for (( i=0; i<$num; i++ )) | ||
do | ||
http_port=$(find_available_port) | ||
mysql_port=$(find_available_port) | ||
start_databend_query $http_port $mysql_port "logs_$http_port" $node_group | ||
done |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.