Skip to content
This repository was archived by the owner on Jul 4, 2025. It is now read-only.

Commit 7918935

Browse files
nguyenhoangthuan99hiento09
nguyenhoangthuan99
and
hiento09
authored
Feat/python package ci (#1792)
* feat: add ci for python package * feat: add ci for python package * feat: add ci for python package * test * test * test * test * test * test * test * test * test * test * test * test * test * test * test package unix * test package mac arm * test package using miniconda * test package using miniconda * test package using miniconda * test package using miniconda * test package using miniconda * test package using miniconda * test package using miniconda * test package using miniconda * test package using miniconda * test package using miniconda * test package using miniconda * Add upload artifact linux * feat: add codesign for macos * Test CI window * Test CI window * Test CI window * Test CI window * test CI windows * test CI windows * test CI windows include hidden file * test CI macos include hidden file * test CI macos include hidden file * test CI macos include hidden file * chore: add package pipeline for 4 os * chore: add package pipeline for 4 os change compression level * chore: add package pipeline for 4 os optimize linux size * chore: add package pipeline for 4 os optimize linux size and fix windows * chore: add package pipeline fix windows * Feat python package codesign (#1780) * feat: add codesign for macos * feat: add codesign for macos * fix: notary python zipped folder --------- Co-authored-by: Hien To <tominhhien97@gmail.com> * Update python-package.yml * Update python-package.yml * test: package fish speech * test: package fish speech * test: rerun windows * feat: package env for ichigo-wrapper * feat: package env for ichigo-wrapper * feat: package env for ichigo-wrapper * feat: package env for whispervq * feat: package env for fish-speech * feat: package env for fish-speech mac * Fix: increase timeout for macos notarize * Update python-package.yml * Fix: upload venv to hf instead of github releas * Update: test run new CI for package python * Update: test run new CI for package python * Update: test run new CI for package python * Update: test run new CI for package python windows * Update: test run new CI for package python windows * Finished: venv package * feat: init CI for upload python script to huggingface * Finished CI for upload python scripts --------- Co-authored-by: Hien To <tominhhien97@gmail.com> Co-authored-by: hiento09 <136591877+hiento09@users.noreply.github.com>
1 parent c4b370f commit 7918935

File tree

2 files changed

+347
-0
lines changed

2 files changed

+347
-0
lines changed

‎.github/workflows/python-script-package.yml‎

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
name: Build and Package Python Code
2+
3+
on:
4+
workflow_dispatch:
5+
inputs:
6+
model_dir:
7+
description: "Path to model directory in github repo"
8+
required: true
9+
repo_name:
10+
description: "name of repo to be checked out"
11+
required: true
12+
branch_name:
13+
description: "name of branch to be checked out"
14+
required: true
15+
default: main
16+
hf_repo:
17+
description: "name of huggingface repo to be pushed"
18+
required: true
19+
hf_prefix_branch:
20+
description: "prefix of hf branch"
21+
required: false
22+
23+
env:
24+
MODEL_DIR: ${{ inputs.model_dir }}
25+
REPO_NAME: ${{ inputs.repo_name}}
26+
BRANCH_NAME: ${{ inputs.branch_name }}
27+
HF_REPO: ${{ inputs.hf_repo }}
28+
HF_PREFIX_BRANCH: ${{ inputs.hf_prefix_branch }}
29+
30+
jobs:
31+
build-and-test:
32+
runs-on: ${{ matrix.runs-on }}
33+
timeout-minutes: 3600
34+
strategy:
35+
fail-fast: false
36+
matrix:
37+
include:
38+
- os: "linux"
39+
name: "amd64"
40+
runs-on: "ubuntu-20-04-cuda-12-0"
41+
- os: "mac"
42+
name: "amd64"
43+
runs-on: "macos-selfhosted-12"
44+
- os: "mac"
45+
name: "arm64"
46+
runs-on: "macos-selfhosted-12-arm64"
47+
- os: "windows"
48+
name: "amd64"
49+
runs-on: "windows-cuda-12-0"
50+
steps:
51+
- name: Clone
52+
id: checkout
53+
uses: actions/checkout@v3
54+
with:
55+
submodules: recursive
56+
repository: ${{env.REPO_NAME}}
57+
ref: ${{env.BRANCH_NAME}}
58+
- name: use python
59+
uses: actions/setup-python@v5
60+
with:
61+
python-version: "3.10"
62+
63+
- name: Install dependencies
64+
run: |
65+
python -m pip install --upgrade pip
66+
python -m pip install hf-transfer huggingface_hub
67+
68+
- name: Upload Artifact
69+
run: |
70+
huggingface-cli login --token ${{ secrets.HUGGINGFACE_TOKEN_WRITE }} --add-to-git-credential
71+
cd ${{env.MODEL_DIR}} && huggingface-cli upload ${{env.HF_REPO}} . . --revision ${{env.HF_PREFIX_BRANCH}}-${{ matrix.os }}-${{ matrix.name }}
72+
huggingface-cli logout

‎.github/workflows/python-venv-package.yml‎

Lines changed: 275 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,275 @@
1+
name: Build and Package Python Virtual Environment
2+
3+
on:
4+
workflow_dispatch:
5+
inputs:
6+
model_dir:
7+
description: "Path to model directory in github repo"
8+
required: true
9+
model_name:
10+
description: "name of model to be release"
11+
required: true
12+
repo_name:
13+
description: "name of repo to be checked out"
14+
required: true
15+
branch_name:
16+
description: "name of branch to be checked out"
17+
required: true
18+
default: main
19+
hf_repo:
20+
description: "name of huggingface repo to be pushed"
21+
required: true
22+
hf_prefix_branch:
23+
description: "prefix of hf branch"
24+
required: false
25+
26+
27+
28+
env:
29+
MODEL_DIR: ${{ inputs.model_dir }}
30+
MODEL_NAME: ${{ inputs.model_name }}
31+
REPO_NAME: ${{ inputs.repo_name }}
32+
BRANCH_NAME: ${{ inputs.branch_name }}
33+
HF_REPO: ${{ inputs.hf_repo }}
34+
HF_PREFIX_BRANCH: ${{ inputs.hf_prefix_branch }}
35+
36+
jobs:
37+
build-and-test:
38+
runs-on: ${{ matrix.runs-on }}
39+
timeout-minutes: 3600
40+
strategy:
41+
fail-fast: false
42+
matrix:
43+
include:
44+
- os: "linux"
45+
name: "amd64"
46+
runs-on: "ubuntu-20-04-cuda-12-0"
47+
- os: "mac"
48+
name: "amd64"
49+
runs-on: "macos-selfhosted-12"
50+
- os: "mac"
51+
name: "arm64"
52+
runs-on: "macos-selfhosted-12-arm64"
53+
- os: "windows"
54+
name: "amd64"
55+
runs-on: "windows-cuda-12-0"
56+
steps:
57+
- name: Clone
58+
id: checkout
59+
uses: actions/checkout@v3
60+
with:
61+
submodules: recursive
62+
repository: ${{env.REPO_NAME}}
63+
ref: ${{env.BRANCH_NAME}}
64+
- uses: conda-incubator/setup-miniconda@v3
65+
if: runner.os != 'windows'
66+
with:
67+
auto-update-conda: true
68+
python-version: 3.11
69+
- name: use python
70+
if : runner.os == 'windows'
71+
uses: actions/setup-python@v5
72+
with:
73+
python-version: "3.11"
74+
75+
- name: Get Cer for code signing
76+
if: runner.os == 'macOS'
77+
run: base64 -d <<< "$CODE_SIGN_P12_BASE64" > /tmp/codesign.p12
78+
shell: bash
79+
env:
80+
CODE_SIGN_P12_BASE64: ${{ secrets.CODE_SIGN_P12_BASE64 }}
81+
82+
- uses: apple-actions/import-codesign-certs@v2
83+
continue-on-error: true
84+
if: runner.os == 'macOS'
85+
with:
86+
p12-file-base64: ${{ secrets.CODE_SIGN_P12_BASE64 }}
87+
p12-password: ${{ secrets.CODE_SIGN_P12_PASSWORD }}
88+
89+
- name: Get Cer for code signing
90+
if: runner.os == 'macOS'
91+
run: base64 -d <<< "$NOTARIZE_P8_BASE64" > /tmp/notary-key.p8
92+
shell: bash
93+
env:
94+
NOTARIZE_P8_BASE64: ${{ secrets.NOTARIZE_P8_BASE64 }}
95+
96+
- name: Install dependencies Windows
97+
if: runner.os == 'windows'
98+
shell: pwsh
99+
run: |
100+
python3 -m pip install fastapi
101+
python3 -m pip freeze | % { python3 -m pip uninstall -y $_ }
102+
python3 -m pip install --upgrade pip
103+
python3 -m pip install -I -r ${{env.MODEL_DIR}}/requirements.cuda.txt
104+
python3 -m pip install python-dotenv
105+
- name: Install dependencies Linux
106+
if: runner.os == 'linux'
107+
run: |
108+
conda create -y -n ${{env.MODEL_NAME}} python=3.11
109+
source $HOME/miniconda3/bin/activate base
110+
conda init
111+
conda activate ${{env.MODEL_NAME}}
112+
python -m pip install fastapi
113+
python -m pip freeze | xargs python -m pip uninstall -y
114+
python -m pip install --upgrade pip
115+
python -m pip install -r ${{env.MODEL_DIR}}/requirements.cuda.txt
116+
python -m pip install python-dotenv
117+
- name: Install dependencies Mac
118+
if: runner.os == 'macOS'
119+
run: |
120+
conda create -y -n ${{env.MODEL_NAME}} python=3.11
121+
source $HOME/miniconda3/bin/activate base
122+
conda init
123+
conda activate ${{env.MODEL_NAME}}
124+
python -m pip install fastapi
125+
python -m pip freeze | xargs python -m pip uninstall -y
126+
python -m pip install --upgrade pip
127+
python -m pip install -r ${{env.MODEL_DIR}}/requirements.txt
128+
python -m pip install python-dotenv
129+
130+
- name: prepare python package windows
131+
if : runner.os == 'windows'
132+
shell: pwsh
133+
run: |
134+
$pythonPath = where.exe python
135+
echo "Python path (where.exe): $pythonPath"
136+
$pythonFolder = Split-Path -Path "$pythonPath" -Parent
137+
echo "PYTHON_FOLDER=$pythonFolder" >> $env:GITHUB_ENV
138+
copy "$pythonFolder\python*.*" "$pythonFolder\Scripts\"
139+
140+
- name: prepare python package macos
141+
if : runner.os == 'macOs'
142+
run: |
143+
source $HOME/miniconda3/bin/activate base
144+
conda init
145+
conda activate ${{env.MODEL_NAME}}
146+
PYTHON_PATH=$(which python)
147+
echo $PYTHON_PATH
148+
PYTHON_FOLDER=$(dirname $(dirname "$PYTHON_PATH"))
149+
echo "PYTHON_FOLDER=$PYTHON_FOLDER" >> $GITHUB_ENV
150+
echo "github end PYTHON_FOLDER: ${{env.PYTHON_FOLDER}}"
151+
- name: prepare python package linux
152+
if : runner.os == 'linux'
153+
run: |
154+
source $HOME/miniconda3/bin/activate base
155+
conda init
156+
conda activate ${{env.MODEL_NAME}}
157+
PYTHON_PATH=$(which python)
158+
echo $PYTHON_PATH
159+
PYTHON_FOLDER=$(dirname $(dirname "$PYTHON_PATH"))
160+
rm -rf $PYTHON_FOLDER/lib/python3.1
161+
echo "PYTHON_FOLDER=$PYTHON_FOLDER" >> $GITHUB_ENV
162+
echo "github end PYTHON_FOLDER: ${{env.PYTHON_FOLDER}}"
163+
164+
- name: create plist file
165+
if: runner.os == 'macOS'
166+
run: |
167+
cat << EOF > /tmp/entitlements.plist
168+
<?xml version="1.0" encoding="UTF-8"?>
169+
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
170+
<plist version="1.0">
171+
<dict>
172+
<!-- These are required for binaries built by PyInstaller -->
173+
<key>com.apple.security.cs.allow-jit</key>
174+
<true/>
175+
<key>com.apple.security.cs.allow-unsigned-executable-memory</key>
176+
<true/>
177+
178+
<!-- Add these for additional permissions -->
179+
<key>com.apple.security.app-sandbox</key>
180+
<false/>
181+
<key>com.apple.security.network.client</key>
182+
<true/>
183+
<key>com.apple.security.network.server</key>
184+
<true/>
185+
<key>com.apple.security.device.audio-input</key>
186+
<true/>
187+
<key>com.apple.security.device.microphone</key>
188+
<true/>
189+
<key>com.apple.security.device.camera</key>
190+
<true/>
191+
<key>com.apple.security.files.user-selected.read-write</key>
192+
<true/>
193+
<key>com.apple.security.cs.disable-library-validation</key>
194+
<true/>
195+
<key>com.apple.security.cs.allow-dyld-environment-variables</key>
196+
<true/>
197+
<key>com.apple.security.cs.allow-executable-memory</key>
198+
<true/>
199+
</dict>
200+
</plist>
201+
EOF
202+
203+
- name: Notary macOS Binary
204+
if: runner.os == 'macOS'
205+
run: |
206+
codesign --force --entitlements="/tmp/entitlements.plist" -s "${{ secrets.DEVELOPER_ID }}" --options=runtime ${{env.PYTHON_FOLDER}}/bin/python
207+
codesign --force --entitlements="/tmp/entitlements.plist" -s "${{ secrets.DEVELOPER_ID }}" --options=runtime ${{env.PYTHON_FOLDER}}/bin/python3
208+
# Code sign all .so files and .dylib files
209+
210+
find ${{env.PYTHON_FOLDER}} -type f \( -name "*.so" -o -name "*.dylib" \) -exec codesign --force --entitlements="/tmp/entitlements.plist" -s "${{ secrets.DEVELOPER_ID }}" --options=runtime {} \;
211+
212+
curl -sSfL https://raw.githubusercontent.com/anchore/quill/main/install.sh | sudo sh -s -- -b /usr/local/bin
213+
# Notarize the binary
214+
quill notarize ${{env.PYTHON_FOLDER}}/bin/python
215+
quill notarize ${{env.PYTHON_FOLDER}}/bin/python3
216+
find ${{env.PYTHON_FOLDER}} -type f \( -name "*.so" -o -name "*.dylib" \) -exec quill notarize {} \;
217+
env:
218+
QUILL_NOTARY_KEY_ID: ${{ secrets.NOTARY_KEY_ID }}
219+
QUILL_NOTARY_ISSUER: ${{ secrets.NOTARY_ISSUER }}
220+
QUILL_NOTARY_KEY: "/tmp/notary-key.p8"
221+
222+
223+
- name: Upload Artifact MacOS
224+
if : runner.os == 'macOS'
225+
run: |
226+
brew install zip
227+
cd ${{env.PYTHON_FOLDER}} && zip -r venv.zip *
228+
conda create -y -n hf-upload python=3.11
229+
source $HOME/miniconda3/bin/activate base
230+
conda init
231+
conda activate hf-upload
232+
python -m pip install hf-transfer huggingface_hub
233+
huggingface-cli login --token ${{ secrets.HUGGINGFACE_TOKEN_WRITE }} --add-to-git-credential
234+
huggingface-cli upload ${{env.HF_REPO}} venv.zip --revision ${{env.HF_PREFIX_BRANCH}}-${{ matrix.os }}-${{ matrix.name }}
235+
rm -rf venv.zip
236+
huggingface-cli logout
237+
238+
- name: Upload Artifact Linux
239+
if : runner.os == 'linux'
240+
run: |
241+
sudo apt-get install -y zip
242+
cd ${{env.PYTHON_FOLDER}} && zip -r venv.zip *
243+
conda create -y -n hf-upload python=3.11
244+
source $HOME/miniconda3/bin/activate base
245+
conda init
246+
conda activate hf-upload
247+
python -m pip install hf-transfer huggingface_hub
248+
huggingface-cli login --token ${{ secrets.HUGGINGFACE_TOKEN_WRITE }} --add-to-git-credential
249+
huggingface-cli upload ${{env.HF_REPO}} venv.zip --revision ${{env.HF_PREFIX_BRANCH}}-${{ matrix.os }}-${{ matrix.name }}
250+
rm -rf venv.zip
251+
huggingface-cli logout
252+
253+
254+
- name: Upload Artifact Windows
255+
if : runner.os == 'windows'
256+
shell: pwsh
257+
run: |
258+
Compress-Archive -Path ${{env.PYTHON_FOLDER}}/* -DestinationPath venv.zip
259+
python -m pip install hf-transfer huggingface_hub
260+
huggingface-cli login --token ${{ secrets.HUGGINGFACE_TOKEN_WRITE }} --add-to-git-credential
261+
huggingface-cli upload ${{env.HF_REPO}} venv.zip --revision ${{env.HF_PREFIX_BRANCH}}-${{ matrix.os }}-${{ matrix.name }}
262+
rm venv.zip
263+
huggingface-cli logout
264+
265+
266+
- name: Post Upload windows
267+
if : runner.os == 'windows'
268+
run: |
269+
rm ${{env.PYTHON_FOLDER}}/Scripts/python*.*
270+
271+
- name: Remove Keychain
272+
continue-on-error: true
273+
if: always() && runner.os == 'macOS'
274+
run: |
275+
security delete-keychain signing_temp.keychain

0 commit comments

Comments
 (0)