Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

macOS用自動ビルド: ONNX版コアに対応 #257

Merged
merged 15 commits into from
Dec 31, 2021
Merged

macOS用自動ビルド: ONNX版コアに対応 #257

Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
f5afca5
build-macos: download onnxruntime instead of libtorch
aoirint Dec 30, 2021
d48c565
build-macos: rename Hiroshiba -> VOICEVOX
aoirint Dec 30, 2021
599bb94
build-macos: copy onnxruntime dylib on nuitka build
aoirint Dec 30, 2021
77e271b
build-macos: specify dylib name to copy
aoirint Dec 30, 2021
2d7c45d
build-macos: use libcore.dylib in 0.10.preview.0
aoirint Dec 30, 2021
fd34f13
build-macos: fix core extracted path
aoirint Dec 30, 2021
db09b76
build-macos: revert core directory
aoirint Dec 30, 2021
7af9c7d
build-mac: fix cd & copy order
aoirint Dec 30, 2021
90f6123
add set -eux
aoirint Dec 30, 2021
8c6a46a
fix matrix.os (invalid cache key)
aoirint Dec 30, 2021
3e4192f
build-mac: copy libonnxruntime explicitly
aoirint Dec 30, 2021
acd7d55
fix comment location
aoirint Dec 30, 2021
8703565
add set -eux
aoirint Dec 30, 2021
101ac8d
build-mac: onnxruntime extract only dylibs
aoirint Dec 30, 2021
2762680
fix argument order
aoirint Dec 30, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
114 changes: 66 additions & 48 deletions .github/workflows/build.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,14 +17,15 @@ jobs:
strategy:
matrix:
include:
- os: [macos-11]
- os: macos-11
python: '3.8'
python_architecture: 'x64'
pip_cache_path: ~/Library/Caches/pip
voicevox_resource_version: '0.10.preview.1'
voicevox_core_version: '0.9.0'
voicevox_core_example_version: '0.9.0'
libtorch_url: https://download.pytorch.org/libtorch/cpu/libtorch-macos-1.9.0.zip
voicevox_core_version: '0.10.preview.0'
voicevox_core_source_version: '0.10.preview.0'
voicevox_core_library_name: libcore_cpu_x64.dylib
onnxruntime_url: https://github.com/microsoft/onnxruntime/releases/download/v1.9.0/onnxruntime-osx-x64-1.9.0.tgz
artifact_name: macos-x64

runs-on: ${{ matrix.os }}
Expand Down Expand Up @@ -60,7 +61,7 @@ jobs:
id: venv-build-voicevox-cache
with:
path: ${{ env.pythonLocation }}
key: ${{ runner.os }}-libs-build_voicevox-${{ matrix.python }}-${{ matrix.python_architecture }}-${{ matrix.voicevox_core_version }}-${{ matrix.voicevox_core_example_version }}-${{ hashFiles('**/requirements.txt', '**/requirements-dev.txt') }}
key: ${{ runner.os }}-libs-build_voicevox-${{ matrix.python }}-${{ matrix.python_architecture }}-${{ matrix.voicevox_core_version }}-${{ matrix.voicevox_core_source_version }}-${{ hashFiles('**/requirements.txt', '**/requirements-dev.txt') }}

- name: Prepare Python venv for licenses.json generation cache
uses: actions/cache@v2
Expand Down Expand Up @@ -89,26 +90,33 @@ jobs:
python generate_licenses.py > licenses.json
deactivate

# Download LibTorch
- name: Export LibTorch url to calc hash
# Download ONNX Runtime
- name: Export ONNX Runtime url to calc hash
shell: bash
run: echo "${{ matrix.libtorch_url }}" > download/libtorch_url.txt
run: echo "${{ matrix.onnxruntime_url }}" > download/onnxruntime_url.txt

- name: Prepare LibTorch cache
- name: Prepare ONNX Runtime cache
uses: actions/cache@v2
id: libtorch-dylib-cache
id: onnxruntime-dylib-cache
with:
key: ${{ matrix.os }}-libtorch-dylib-${{ hashFiles('download/libtorch_url.txt') }}
path: download/libtorch
key: ${{ matrix.os }}-onnxruntime-dylib-${{ hashFiles('download/onnxruntime_url.txt') }}-v1
path: download/onnxruntime

- name: Download LibTorch
if: steps.libtorch-dylib-cache.outputs.cache-hit != 'true'
- name: Download ONNX Runtime
if: steps.onnxruntime-dylib-cache.outputs.cache-hit != 'true'
shell: bash
run: |
curl -L "${{ matrix.libtorch_url }}" > download/libtorch.zip
curl -L "${{ matrix.onnxruntime_url }}" > download/onnxruntime.tgz

# extract only dylibs
unzip download/libtorch.zip "libtorch/lib/*.dylib" -d download/
rm download/libtorch.zip
mkdir -p download/onnxruntime
tar xf "download/onnxruntime.tgz" -C "download/onnxruntime" --strip-components 1 'onnxruntime-*/lib/*.dylib'
rm download/onnxruntime.tgz

- name: Show disk space (debug info)
shell: bash
run: |
df -h

# Download VOICEVOX RESOURCE
- name: Prepare VOICEVOX RESOURCE cache
Expand All @@ -122,7 +130,7 @@ jobs:
if: steps.voicevox-resource-cache.outputs.cache-hit != 'true'
uses: actions/checkout@v2
with:
repository: Hiroshiba/voicevox_resource
repository: VOICEVOX/voicevox_resource
ref: ${{ matrix.voicevox_resource_version }}
path: download/resource

Expand All @@ -134,58 +142,64 @@ jobs:
rm -r speaker_info
cp -r download/resource/character_info speaker_info

- name: Prepare VOICEVOX Core cache
- name: Prepare VOICEVOX Core release cache
uses: actions/cache@v2
id: voicevox-core-cache
with:
key: ${{ matrix.os }}-voicevox-core-${{ matrix.voicevox_core_version }}
path: download/core

- name: Download VOICEVOX Core
- name: Download VOICEVOX Core release
if: steps.voicevox-core-cache.outputs.cache-hit != 'true'
shell: bash
run: |
curl -L "https://github.com/Hiroshiba/voicevox_core/releases/download/${{ matrix.voicevox_core_version }}/core.zip" > download/core.zip
# extracted like download/core/metas.json
curl -L "https://github.com/VOICEVOX/voicevox_core/releases/download/${{ matrix.voicevox_core_version }}/core.zip" > download/core.zip
ditto -x -k --sequesterRsrc --rsrc download/core.zip download/
rm download/core.zip

- name: Prepare VOICEVOX Core file
if: steps.voicevox-core-cache.outputs.cache-hit != 'true'
shell: bash
run: mv libcore_cpu.dylib libcore.dylib
working-directory: download/core

- name: Export VOICEVOX Core path to LD_LIBRARY_PATH and LIBRARY_PATH
run: |
echo "LD_LIBRARY_PATH=$(pwd):${{ env.LD_LIBRARY_PATH }}" >> $GITHUB_ENV
echo "LIBRARY_PATH=$(pwd):${{ env.LIBRARY_PATH }}" >> $GITHUB_ENV
working-directory: download/core

- name: Prepare VOICEVOX Core example cache
# Install VOICEVOX Core Python package
- name: Prepare VOICEVOX Core source cache
uses: actions/cache@v2
id: voicevox-core-example-cache
id: voicevox-core-source-cache
with:
key: ${{ matrix.os }}-voicevox-core-example-${{ matrix.voicevox_core_example_version }}
path: download/voicevox_core_example
key: ${{ matrix.os }}-voicevox-core-source-${{ matrix.voicevox_core_source_version }}
path: download/voicevox_core_source

- name: Clone VOICEVOX Core example
if: steps.voicevox-core-example-cache.outputs.cache-hit != 'true'
shell: bash
run: |
git clone -b "${{ matrix.voicevox_core_example_version }}" --depth 1 "https://github.com/Hiroshiba/voicevox_core.git" download/voicevox_core_example
- name: Checkout VOICEVOX Core source
if: steps.voicevox-core-source-cache.outputs.cache-hit != 'true'
uses: actions/checkout@v2
with:
repository: VOICEVOX/voicevox_core
ref: ${{ matrix.voicevox_core_source_version }}
path: download/voicevox_core_source

- name: Install dependencies for building VOICEVOX Core Python package
shell: bash
run: |
pip install -r download/voicevox_core_example/example/python/requirements.txt
pip install -r download/voicevox_core_source/requirements.txt

- name: Install VOICEVOX Core Python package
shell: bash
run: |
NUMPY_INCLUDE=`python -c "import numpy; print(numpy.get_include())"`
cp download/voicevox_core_example/core.h download/voicevox_core_example/example/python/
set -eux

mkdir -p download/voicevox_core_source/core/lib

CPATH="$NUMPY_INCLUDE:${CPATH:-}" pip install download/voicevox_core_example/example/python/
# Copy VOICEVOX Core dylib to core/lib
cp download/core/${{ matrix.voicevox_core_library_name }} download/voicevox_core_source/core/lib/libcore.dylib

cd download/voicevox_core_source
cp core/src/core.h core/lib/
cd -

# Copy ONNX Runtime dylib
cp download/onnxruntime/lib/libonnxruntime.*.dylib download/voicevox_core_source/core/lib/

# Install VOICEVOX Core Python package with libcore.dylib & libonnxruntime.*.dylib
cd download/voicevox_core_source
NUMPY_INCLUDE=`python -c "import numpy; print(numpy.get_include())"`
CPATH="$NUMPY_INCLUDE:${CPATH:-}" pip install .

- name: Install dependencies for building VOICEVOX engine
shell: bash
Expand All @@ -195,6 +209,8 @@ jobs:
- name: Download PyOpenJTalk dictionary
shell: bash
run: |
set -eux

# Download pyopenjtalk dictionary
# try 5 times, sleep 5 seconds before retry
for i in $(seq 5); do
Expand All @@ -214,6 +230,9 @@ jobs:

- name: Build run.py
run: |
set -eux

# Nuitka does not copy grandchild dependencies. Explicitly copy libonnxruntime
python -m nuitka \
--output-dir=./ \
--standalone \
Expand All @@ -227,10 +246,9 @@ jobs:
--include-data-file=../VERSION.txt=./ \
--include-data-file=../licenses.json=./ \
--include-data-file=../presets.yaml=./ \
--include-data-file=../download/libtorch/lib/*.dylib=./ \
--include-data-file=../download/core/*.dylib=./ \
--include-data-file=../download/core/*.bin=./ \
--include-data-file=../download/core/metas.json=./ \
--include-data-file=../download/onnxruntime/lib/libonnxruntime.*.dylib=./ \
--include-data-file=${{ env.pythonLocation }}/lib/python*/site-packages/scipy/.dylibs/*.dylib=./scipy/.dylibs/ \
--include-data-file=${{ env.pythonLocation }}/lib/python*/site-packages/_soundfile_data/*=./_soundfile_data/ \
--include-data-dir=../speaker_info=./speaker_info \
Expand Down