Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pass parameters freely to scripts/run_assistant_server.sh #492

Merged
merged 4 commits into from
Jun 20, 2024
Merged

Pass parameters freely to scripts/run_assistant_server.sh #492

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
298e3f3
首次适配Qwen2-72B-Instruct-GPTQ-Int4
monk-after-90s Jun 14, 2024
dd9cbc2
Revert "首次适配Qwen2-72B-Instruct-GPTQ-Int4"
monk-after-90s Jun 14, 2024
6cce569
手动超时而不是自动
monk-after-90s Jun 14, 2024
27efd7a
更加优雅传递命令行参数
monk-after-90s Jun 14, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 7 additions & 5 deletions scripts/run_assistant_server.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,9 @@ MODEL_DIR=""
MODEL_SERVER=""
MODEL_NAME=""

# Save all command line arguments
ALL_ARGS="$@"

# Loop through arguments and process them
while [[ $# -gt 0 ]]; do
case $1 in
Expand Down Expand Up @@ -38,15 +41,14 @@ echo "Model name: $MODEL_NAME"
echo "Model directory: $MODEL_DIR"
echo "Model server: $MODEL_SERVER"


# running
echo "Running fastapi assistant server at port 31512."
export PYTHONPATH=$PYTHONPATH:modelscope_agent_servers

if [ "$MODEL_DIR" != "" ]; then
echo "Running vllm server, please make sure install vllm"
# Start the first server in the background on port 8000
python -m vllm.entrypoints.openai.api_server --served-model-name $MODEL_NAME --model $MODEL_DIR & SERVER_1_PID=$!
python -m vllm.entrypoints.openai.api_server $ALL_ARGS & SERVER_1_PID=$!
export MODEL_SERVER=vllm-server
export OPENAI_API_BASE=http://localhost:8000/v1
echo "Model server: $MODEL_SERVER"
Expand All @@ -55,7 +57,7 @@ if [ "$MODEL_DIR" != "" ]; then
# Function to check if the first server is up
check_first_server() {
echo "Checking if Server 1 is up..."
for i in {1..10}; do # try up to 10 times
for i in {1..100000000}; do # try up to 100000000 times
curl -s http://localhost:8000 > /dev/null
if [ $? -eq 0 ]; then
echo "Server 1 is up and running."
Expand All @@ -70,9 +72,9 @@ if [ "$MODEL_DIR" != "" ]; then

# Wait for the first server to be up
if check_first_server; then
# Start the second server on port 8001
# Start the second server on port 31512
echo "Starting Server 2..."
uvicorn modelscope_agent_servers.assistant_server.api:app --host 0.0.0.0 --port 31512 & $SERVER_2_PID
uvicorn modelscope_agent_servers.assistant_server.api:app --host 0.0.0.0 --port 31512 & SERVER_2_PID=$!
else
echo "Failed to start Server 1."
exit 1
Expand Down
Loading