Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add env_run.sh and correct packages version #733

Merged
merged 3 commits into from
Oct 10, 2024
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 29 additions & 13 deletions README.md
Original file line number Diff line number Diff line change
@@ -48,30 +48,46 @@ PaddleMIX是基于飞桨的多模态大模型开发套件,聚合图像、文
https://github.com/PaddlePaddle/PaddleMIX/assets/29787866/8d32722a-e307-46cb-a8c0-be8acd93d2c8


## 安装
## 安装步骤
### 1. 克隆PaddleMIX仓库
```
git clone https://github.com/PaddlePaddle/PaddleMIX
cd PaddleMIX
```

1. 环境依赖
### 2. 创建虚拟环境
```
conda create -n paddlemix python=3.10 -y
conda activate paddlemix
```
pip install -r requirements.txt
### 3. 安装PaddlePaddle
#### 方法 1: 一键安装(推荐)
- CUDA 11.x或12.3
- PaddlePaddle 3.0.0b1
```
sh build_paddle_env.sh
```

#### 方法 2: 手动安装
关于PaddlePaddle安装的详细教程请查看[Installation](https://www.paddlepaddle.org.cn/install/quick?docurl=/documentation/docs/zh/develop/install/pip/linux-pip.html)。

> 注:ppdiffusers部分模型需要依赖 CUDA 11.2 及以上版本,如果本地机器不符合要求,建议前往 [AI Studio](https://aistudio.baidu.com/index) 进行模型训练、推理任务。

> 如果希望使用**bf16**训练推理,请使用支持**bf16**的GPU,如A100。
### 4. 安装依赖

2. 手动安装
```
git clone https://github.com/PaddlePaddle/PaddleMIX
cd PaddleMIX
pip install -e .
#### 方法 1: 一键安装(推荐)

#ppdiffusers 安装
cd ppdiffusers
pip install -e .
运行以下命令来自动安装所有必要的依赖:
```
sh build_env.sh
```

#### 方法 2: 手动安装(请参考 build_env.sh)

> 注:ppdiffusers部分模型需要依赖 CUDA 11.2 及以上版本,如果本地机器不符合要求,建议前往 [AI Studio](https://aistudio.baidu.com/index) 进行模型训练、推理任务。

> 如果希望使用**bf16**训练推理,请使用支持**bf16**的GPU,如A100。


## 教程

- [快速开始](applications/README.md/#快速开始)
44 changes: 31 additions & 13 deletions README_EN.md
Original file line number Diff line number Diff line change
@@ -47,29 +47,47 @@ https://github.com/PaddlePaddle/PaddleMIX/assets/29787866/8d32722a-e307-46cb-a8c



## Installation
## Installation
### 1. Clone the PaddleMIX repository
```
git clone https://github.com/PaddlePaddle/PaddleMIX
cd PaddleMIX
```

1. Environment Dependencies
### 2. Create a virtual environment
```
conda create -n paddlemix python=3.10 -y
conda activate paddlemix
```
pip install -r requirements.txt
### 3. Install PaddlePaddle
#### Method 1: One-click installation (recommended)
- CUDA 11.x or 12.3
- PaddlePaddle 3.0.0b1
```
sh build_paddle_env.sh
```

#### Method 2: Manual installation
For detailed instructions on installing PaddlePaddle, please refer to [Installation](https://www.paddlepaddle.org.cn/install/quick?docurl=/documentation/docs/zh/develop/install/pip/linux-pip.html).


### 4. Install dependencies

#### Method 1: One-click installation (recommended)

Run the following command to automatically install all necessary dependencies:
```
sh build_env.sh
```

#### Method 2: Manual installation (Please refer to build_env.sh)

Detailed [installation]((https://www.paddlepaddle.org.cn/install/quick?docurl=/documentation/docs/zh/develop/install/pip/linux-pip.html)) tutorials for PaddlePaddle

> Note: parts of Some models in ppdiffusers require CUDA 11.2 or higher. If your local machine does not meet the requirements, it is recommended to go to [AI Studio](https://aistudio.baidu.com/index) for model training and inference tasks.

> If you wish to train and infer using **bf16**, please use a GPU that supports **bf16**, such as the A100.

2. Manual Installation
```
git clone https://github.com/PaddlePaddle/PaddleMIX
cd PaddleMIX
pip install -e .

#ppdiffusers 安装
cd ppdiffusers
pip install -e .
```

## Tutorial

28 changes: 28 additions & 0 deletions build_env.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
#!/bin/bash

# 设置错误时退出
set -e

echo "开始安装 PaddleMIX 及其依赖..."

# 安装 PaddleMIX
echo "安装 PaddleMIX..."
pip install -e .

# 安装 ppdiffusers
echo "安装 ppdiffusers..."
cd ppdiffusers
pip install -e .
cd ..

# 安装依赖包
echo "安装依赖包..."
pip install -r requirements.txt

# 安装自定义算子
echo "安装自定义算子..."
cd paddlemix/external_ops
python setup.py install
cd ../../

echo "安装完成!"
34 changes: 34 additions & 0 deletions build_paddle_env.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
#!/bin/bash

# 设置错误时退出
set -e

echo "开始安装 paddlepaddle ..."
# 检测 CUDA 版本并安装相应的 paddlepaddle
if command -v nvcc &> /dev/null; then
cuda_version=$(nvcc --version | grep "release" | sed -n 's/.*release \([0-9]\+\.[0-9]\+\).*/\1/p')
echo "检测到 CUDA 版本: $cuda_version"
if [[ "$cuda_version" == "11.2" ]]; then
echo "安装 CUDA 11.2 版本的 paddlepaddle..."
python -m pip install paddlepaddle-gpu==3.0.0b1 -i https://www.paddlepaddle.org.cn/packages/stable/cu112/
elif [[ "$cuda_version" == "11.6" ]]; then
echo "安装 CUDA 11.6 版本的 paddlepaddle..."
python -m pip install paddlepaddle-gpu==3.0.0b1 -i https://www.paddlepaddle.org.cn/packages/stable/cu116/
elif [[ "$cuda_version" == "11.7" ]]; then
echo "安装 CUDA 11.7 版本的 paddlepaddle..."
python -m pip install paddlepaddle-gpu==3.0.0b1 -i https://www.paddlepaddle.org.cn/packages/stable/cu117/
elif [[ "$cuda_version" == "11.8" ]]; then
echo "安装 CUDA 11.8 版本的 paddlepaddle..."
python -m pip install paddlepaddle-gpu==3.0.0b1 -i https://www.paddlepaddle.org.cn/packages/stable/cu118/
elif [[ "$cuda_version" == "12.3" ]]; then
echo "安装 CUDA 12.3 版本的 paddlepaddle..."
python -m pip install paddlepaddle-gpu==3.0.0b1 -i https://www.paddlepaddle.org.cn/packages/stable/cu123/
else
echo "警告: 不支持的 CUDA 版本。请手动安装适合您系统的 paddlepaddle 版本。"
fi
else
echo "未检测到 CUDA。安装 CPU 版本的 paddlepaddle..."
pip install paddlepaddle-gpu==3.0.0b1
fi

echo "安装完成!"
1 change: 1 addition & 0 deletions paddlemix/appflow/requirements.txt
Original file line number Diff line number Diff line change
@@ -4,3 +4,4 @@ paddlenlp>=2.6.0rc0
https://bj.bcebos.com/v1/paddlenlp/models/community/paddlemix/appflow/paddlespeech/paddlespeech-0.0.1-py3-none-any.whl
opencc==1.1.6
matplotlib==3.7.5
scipy==1.12.0
2 changes: 1 addition & 1 deletion paddlemix/datasets/__init__.py
Original file line number Diff line number Diff line change
@@ -31,7 +31,7 @@
if version.startswith('3'):
from .internvl_dataset import *
else:
print(f"paddlenlp version {version} is not 3.x, skipping import.")
print(f"paddlenlp version {version} is not 3.x, skipping import internvl2 datasets.")

except ImportError:
print("paddlenlp is not installed.")
8 changes: 7 additions & 1 deletion paddlemix/examples/qwen2_vl/README.md
Original file line number Diff line number Diff line change
@@ -10,7 +10,13 @@
- **python >= 3.10**
- tiktoken
> 注:tiktoken 要求python >= 3.8
- paddlepaddle-gpu >= 2.6.1
- **paddlepaddle-gpu 要求版本develop**
```
# 安装示例
python -m pip install paddlepaddle-gpu==0.0.0.post118 -f https://www.paddlepaddle.org.cn/whl/linux/gpu/develop.html
```


- paddlenlp >= 3.0.0(默认开启flash_attn,推荐源码编译安装)

> 注:
6 changes: 4 additions & 2 deletions paddlemix/models/__init__.py
Original file line number Diff line number Diff line change
@@ -26,6 +26,7 @@
from .minigpt4.modeling import *
from .visualglm.configuration import *
from .visualglm.modeling import *
from .qwen_vl import *


import pkg_resources
@@ -34,9 +35,10 @@
try:
if version.startswith('3'):
from .internvl2 import *
from .qwen_vl import *
from .qwen2_vl import *

else:
print(f"paddlenlp version {version} is not 3.x, skipping import.")
print(f"paddlenlp version {version} is not 3.x, skipping import internvl2 and qwen2_vl.")

except ImportError:
print("paddlenlp is not installed.")
2 changes: 1 addition & 1 deletion paddlemix/models/llava/__init__.py
Original file line number Diff line number Diff line change
@@ -28,7 +28,7 @@
from .modeling_qwen import *
from .configuration_qwen import *
else:
print(f"paddlenlp version {version} is not 3.x, skipping import.")
print(f"paddlenlp version {version} is not 3.x, skipping import Qwen2Model for llava-next.")

except ImportError:
print("paddlenlp is not installed.")
5 changes: 5 additions & 0 deletions ppdiffusers/README.md
Original file line number Diff line number Diff line change
@@ -88,6 +88,11 @@ git clone https://github.com/PaddlePaddle/PaddleMIX
cd PaddleMIX/ppdiffusers
python setup.py install
```
### 设置代理
```shell
export HF_HUB_ENABLE_HF_TRANSFER=1
export HF_ENDPOINT=https://hf-mirror.com
```

## 快速开始
我们将以扩散模型的典型代表**Stable Diffusion**为例,带你快速了解PPDiffusers。
5 changes: 3 additions & 2 deletions ppdiffusers/requirements.txt
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
paddlenlp==2.7.2
paddlenlp>=2.7.2
safetensors>=0.3.1
ftfy
regex
@@ -14,4 +14,5 @@ urllib3<=2.0.0
einops>=0.6.1
paddlesde
ligo-segments
huggingface_hub==0.23
huggingface_hub>=0.22.0
hf_transfer