Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding tagged builds for M1 #6140

Merged
merged 7 commits into from
Jun 9, 2022
Merged

Adding tagged builds for M1 #6140

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
5eda8fd
Adding tagged builds
atalman Jun 8, 2022
3fa5c3f
Testing
atalman Jun 8, 2022
015e8db
Testing
atalman Jun 8, 2022
33c2dc2
Testing
atalman Jun 8, 2022
1f0f7f4
Testing
atalman Jun 8, 2022
42abc4a
Adding conda builds
atalman Jun 9, 2022
dc37d5e
Merge branch 'main' into tagged_build_m1
atalman Jun 9, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 38 additions & 9 deletions .github/workflows/build-m1-binaries.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,13 @@ on:
branches:
- nightly
- main
tags:
# NOTE: Binary build pipelines should only get triggered on release candidate builds
# Release candidate tags look like: v1.11.0-rc1
- v[0-9]+.[0-9]+.[0-9]+-rc[0-9]+
workflow_dispatch:
env:
CHANNEL: "nightly"
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here is where we set default CHANNEL to nightly

jobs:
build_wheels:
name: "Build TorchVision M1 wheels"
Expand All @@ -18,6 +24,13 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@v2
- name: Set CHANNEL (only for tagged pushes)
if: ${{ github.event_name == 'push' && startsWith(github.event.ref, 'refs/tags/') }}
run: |
# reference ends with an RC suffix
if [[ ${GITHUB_REF_NAME} = *-rc[0-9]* ]]; then
echo "CHANNEL=test" >> "$GITHUB_ENV"
fi
- name: Build TorchVision M1 wheel
shell: arch -arch arm64 bash {0}
env:
Expand All @@ -29,10 +42,16 @@ jobs:
export PATH=~/miniconda3/bin:$PATH
set -ex
. packaging/pkg_helpers.bash
setup_build_version
# if we are uploading to test channell, our version consist only of the base: 0.x.x - no date string or suffix added
if [[ $CHANNEL == "test" ]]; then
setup_base_build_version
else
setup_build_version
fi

WHL_NAME=torchvision-${BUILD_VERSION}-cp${PY_VERS/.}-cp${PY_VERS/.}-macosx_11_0_arm64.whl
conda create -yp ${ENV_NAME} python=${PY_VERS} numpy libpng jpeg wheel pkg-config
conda run -p ${ENV_NAME} python3 -mpip install torch --pre --extra-index-url=https://download.pytorch.org/whl/nightly
conda run -p ${ENV_NAME} python3 -mpip install torch --pre --extra-index-url=https://download.pytorch.org/whl/${CHANNEL}
conda run -p ${ENV_NAME} python3 -mpip install delocate
conda run -p ${ENV_NAME} python3 setup.py bdist_wheel
export PYTORCH_VERSION="$(conda run -p ${ENV_NAME} python3 -mpip show torch | grep ^Version: | sed 's/Version: *//')"
Expand All @@ -47,7 +66,7 @@ jobs:
. ~/miniconda3/etc/profile.d/conda.sh
set -ex
conda create -yp ${ENV_NAME} python=${PY_VERS} numpy
conda run -p ${ENV_NAME} python3 -mpip install torch --pre --extra-index-url=https://download.pytorch.org/whl/nightly
conda run -p ${ENV_NAME} python3 -mpip install torch --pre --extra-index-url=https://download.pytorch.org/whl/${CHANNEL}
conda run -p ${ENV_NAME} python3 -mpip install dist/*.whl
# Test torch is importable, by changing cwd and running import commands
conda run --cwd /tmp -p ${ENV_NAME} python3 -c "import torchvision;print('torchvision version is ', torchvision.__version__)"
Expand All @@ -60,12 +79,11 @@ jobs:
name: torchvision-py${{ matrix.py_vers }}-macos11-m1
path: dist/
- name: Upload wheel to S3
if: ${{ github.event_name == 'push' && steps.extract_branch.outputs.branch == 'nightly' }}
if: ${{ github.event_name == 'push' && (steps.extract_branch.outputs.branch == 'nightly' || startsWith(github.event.ref, 'refs/tags/')) }}
shell: arch -arch arm64 bash {0}
env:
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_PYTORCH_UPLOADER_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_PYTORCH_UPLOADER_SECRET_ACCESS_KEY }}
CHANNEL: nightly
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Err, this will fail for nightly builds, isn't it (I.e. Set CHANNEL (only for tagged pushes) would only set it to test, but what will set it to nightly?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please see comment below we do set CHANNEL nightly by default

run: |
for pkg in dist/*; do
aws s3 cp "$pkg" "s3://pytorch/whl/${CHANNEL}/cpu/" --acl public-read
Expand All @@ -79,7 +97,13 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@v2

- name: Set CHANNEL (only for tagged pushes)
if: ${{ github.event_name == 'push' && startsWith(github.event.ref, 'refs/tags/') }}
run: |
# reference ends with an RC suffix
if [[ ${GITHUB_REF_NAME} = *-rc[0-9]* ]]; then
echo "CHANNEL=test" >> "$GITHUB_ENV"
fi
- name: Install conda-build and purge previous artifacts
shell: arch -arch arm64 bash {0}
run: |
Expand All @@ -97,7 +121,13 @@ jobs:
. ~/miniconda3/etc/profile.d/conda.sh
set -ex
. packaging/pkg_helpers.bash
setup_build_version

if [[ $CHANNEL == "test" ]]; then
setup_base_build_version
else
setup_build_version
fi

setup_conda_pytorch_constraint
export SOURCE_ROOT_DIR=$(pwd)
conda build -c defaults $CONDA_CHANNEL_FLAGS --no-anaconda-upload --python "$PYTHON_VERSION" packaging/torchvision
Expand All @@ -113,9 +143,8 @@ jobs:
shell: arch -arch arm64 bash {0}
env:
CONDA_PYTORCHBOT_TOKEN: ${{ secrets.CONDA_PYTORCHBOT_TOKEN }}
UPLOAD_CHANNEL: nightly
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above, what will set a channel environment variable to nightly for regular builds?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please see comment below we do set CHANNEL nightly by default

run: |
. ~/miniconda3/etc/profile.d/conda.sh
conda install -yq anaconda-client
set -x
anaconda -t "${CONDA_PYTORCHBOT_TOKEN}" upload ~/miniconda3/conda-bld/osx-arm64/*.tar.bz2 -u "pytorch-${UPLOAD_CHANNEL}" --label main --no-progress --force
anaconda -t "${CONDA_PYTORCHBOT_TOKEN}" upload ~/miniconda3/conda-bld/osx-arm64/*.tar.bz2 -u "pytorch-${CHANNEL}" --label main --no-progress --force
23 changes: 15 additions & 8 deletions packaging/pkg_helpers.bash
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -101,23 +101,30 @@ setup_cuda() {
setup_build_version() {
if [[ -z "$BUILD_VERSION" ]]; then
if [[ -z "$1" ]]; then
SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
# version.txt for some reason has `a` character after major.minor.rev
# command below yields 0.10.0 from version.txt containing 0.10.0a0
_VERSION_BASE=$( cut -f 1 -d a "$SCRIPT_DIR/../version.txt" )
setup_base_build_version
else
_VERSION_BASE="$1"
BUILD_VERSION="$1"
fi
export BUILD_VERSION="$_VERSION_BASE.dev$(date "+%Y%m%d")$VERSION_SUFFIX"
BUILD_VERSION="$BUILD_VERSION.dev$(date "+%Y%m%d")$VERSION_SUFFIX"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not a bash expert, doesn't one need to re-export it here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes totally its re-exported at line 119 below

else
export BUILD_VERSION="$BUILD_VERSION$VERSION_SUFFIX"
BUILD_VERSION="$BUILD_VERSION$VERSION_SUFFIX"
fi

# Set build version based on tag if on tag
if [[ -n "${CIRCLE_TAG}" ]]; then
# Strip tag
export BUILD_VERSION="$(echo "${CIRCLE_TAG}" | sed -e 's/^v//' -e 's/-.*$//')${VERSION_SUFFIX}"
BUILD_VERSION="$(echo "${CIRCLE_TAG}" | sed -e 's/^v//' -e 's/-.*$//')${VERSION_SUFFIX}"
fi

export BUILD_VERSION
}

setup_base_build_version() {
SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
# version.txt for some reason has `a` character after major.minor.rev
# command below yields 0.10.0 from version.txt containing 0.10.0a0
BUILD_VERSION=$( cut -f 1 -d a "$SCRIPT_DIR/../version.txt" )
export BUILD_VERSION
}

# Set some useful variables for OS X, if applicable
Expand Down