Rewrite our packaging from scratch #217

ezyang · 2019-08-07T01:09:32Z

Structure of the PR:

New entry points are packaging/build_wheel.sh and packaging/build_conda.sh. The only mandatory environment variable you have to set is PYTHON_VERSION
CircleCI configuration uses 2.1-style parametrized builds to let you toggle python version, etc. as you do builds. We create a separate job per build configuration for maximum parallelism
build_tools/packaging got moved to packaging, to be in-line with directory structure in torchvision
The build_conda.sh and build_wheel.sh delegate most of the heavy lifting to pkg_helpers.bash, which defines a number of bash functions for performing common operations. The intent is that I'll copy-paste this file between other domain API projects.
TORCHAUDIO_ prefix removed from envvars, so that I can more easily share packaging scripts between projects
BUILD_VERSION is completely gone; just change the version number if you need to rebuild
No more logic for cloning and checking out a fresh copy of torchaudio

This reverts commit 0e64ab6.

This reverts commit 5ad6d2e.

Signed-off-by: Edward Z. Yang <ezyang@fb.com>

soumith · 2019-08-07T03:04:32Z

packaging/pkg_helpers.bash

+setup_python() {
+  if [[ "$(uname)" == Darwin ]]; then
+    eval "$(conda shell.bash hook)"
+    conda create -yn "env$PYTHON_VERSION" python="$PYTHON_VERSION"


what if such an env is already created, and potentially corrupted?

conda env remove -n "env$PYTHON_VERSION" || true

:)

soumith · 2019-08-07T03:07:06Z

packaging/torchaudio/meta.yaml


 build:
-  number: {{ environ.get('TORCHAUDIO_BUILD_NUMBER') }}
-  string: py{{py}}_{{environ.get('TORCHAUDIO_BUILD_NUMBER')}}
+  string: py{{py}}


you dont have a BUILD_NUMBER, are you sure you will never botch a binary build ever?

My thinking is, if I botch build of 1.2.0, then I will next build 1.2.0.post2 (because I don't get build number in pip wheels). Now the version has changed and I don't need it. But I could be wrong!

Signed-off-by: Edward Z. Yang <ezyang@fb.com>

Following a similar line of inquiry to pytorch/audio#217

* New entry points are packaging/build_wheel.sh and packaging/build_conda.sh. The only mandatory environment variable you have to set is PYTHON_VERSION * CircleCI configuration uses 2.1-style parametrized builds to let you toggle python version, etc. as you do builds. We create a separate job per build configuration for maximum parallelism * build_tools/packaging got moved to packaging, to be in-line with directory structure in torchvision * The build_conda.sh and build_wheel.sh delegate most of the heavy lifting to pkg_helpers.bash, which defines a number of bash functions for performing common operations. The intent is that I'll copy-paste this file between other domain API projects. * TORCHAUDIO_ prefix removed from envvars, so that I can more easily share packaging scripts between projects * BUILD_VERSION is completely gone; just change the version number if you need to rebuild * No more logic for cloning and checking out a fresh copy of torchaudio

* Rewrite torchvision packaging (#1209) Following a similar line of inquiry to pytorch/audio#217 * Packaging fixes (#1214) Add uploading support, make CUDA builds actually work. * 0.4.0 parameters Signed-off-by: Edward Z. Yang <ezyang@fb.com> * Actually upload wheels (please port to master) Signed-off-by: Edward Z. Yang <ezyang@fb.com> * Put macos binaries in the right place Signed-off-by: Edward Z. Yang <ezyang@fb.com> * Propagate more environment variables. Signed-off-by: Edward Z. Yang <ezyang@fb.com> * Change the version number Signed-off-by: Edward Z. Yang <ezyang@fb.com> * Go time Signed-off-by: Edward Z. Yang <ezyang@fb.com>

* Rewrite our packaging from scratch (#217) * New entry points are packaging/build_wheel.sh and packaging/build_conda.sh. The only mandatory environment variable you have to set is PYTHON_VERSION * CircleCI configuration uses 2.1-style parametrized builds to let you toggle python version, etc. as you do builds. We create a separate job per build configuration for maximum parallelism * build_tools/packaging got moved to packaging, to be in-line with directory structure in torchvision * The build_conda.sh and build_wheel.sh delegate most of the heavy lifting to pkg_helpers.bash, which defines a number of bash functions for performing common operations. The intent is that I'll copy-paste this file between other domain API projects. * TORCHAUDIO_ prefix removed from envvars, so that I can more easily share packaging scripts between projects * BUILD_VERSION is completely gone; just change the version number if you need to rebuild * No more logic for cloning and checking out a fresh copy of torchaudio * Setup cuda_suffix (#218) Signed-off-by: Edward Z. Yang <ezyang@fb.com> * Add uploading support (#220) Also bugfixes and updates to the packaging scripts. * Finish up and set parameters Signed-off-by: Edward Z. Yang <ezyang@fb.com> * Updates from vision Signed-off-by: Edward Z. Yang <ezyang@fb.com> * Go time Signed-off-by: Edward Z. Yang <ezyang@fb.com>

ezyang added 22 commits August 6, 2019 13:39

Revert "Revert "Attempt OS X binaries""

97dd37e

This reverts commit 0e64ab6.

Revert "Revert "Add to workflow""

9bd3fee

This reverts commit 5ad6d2e.

Put in portability thing for realpath

bb30476

Signed-off-by: Edward Z. Yang <ezyang@fb.com>

Replace wget with portable curl, add missing conda install

b25eccc

Signed-off-by: Edward Z. Yang <ezyang@fb.com>

Fix bad pwd assumption

f208db7

Signed-off-by: Edward Z. Yang <ezyang@fb.com>

Follow URLs in curl

1e4f097

Signed-off-by: Edward Z. Yang <ezyang@fb.com>

Expunge more realpath use

28c62b7

Signed-off-by: Edward Z. Yang <ezyang@fb.com>

Disable coreaudio by default

fc30f58

Signed-off-by: Edward Z. Yang <ezyang@fb.com>

Get started on release bits

4e7f966

Signed-off-by: Edward Z. Yang <ezyang@fb.com>

Refactor scripts

81696fd

Signed-off-by: Edward Z. Yang <ezyang@fb.com>

Improve

d60f728

Signed-off-by: Edward Z. Yang <ezyang@fb.com>

OS X build fix

196ed19

Signed-off-by: Edward Z. Yang <ezyang@fb.com>

Disable release builds for now

fd297e1

Signed-off-by: Edward Z. Yang <ezyang@fb.com>

Minor improvements

5deb45f

Signed-off-by: Edward Z. Yang <ezyang@fb.com>

More severe rewrite

f76fa18

Signed-off-by: Edward Z. Yang <ezyang@fb.com>

bugfix

72c8ebd

Signed-off-by: Edward Z. Yang <ezyang@fb.com>

Properly activate conda

d7ba812

Signed-off-by: Edward Z. Yang <ezyang@fb.com>

conda stuff

5bb4b16

Signed-off-by: Edward Z. Yang <ezyang@fb.com>

Conda online

848792e

Signed-off-by: Edward Z. Yang <ezyang@fb.com>

cpu suffix bugfix on osx

c8f5d09

Signed-off-by: Edward Z. Yang <ezyang@fb.com>

Update README

10b3240

Signed-off-by: Edward Z. Yang <ezyang@fb.com>

Remove dead file

8dcab8f

Signed-off-by: Edward Z. Yang <ezyang@fb.com>

soumith approved these changes Aug 7, 2019

View reviewed changes

ezyang added 5 commits August 6, 2019 20:14

Nuke old copy of env if it already exists

212445f

Signed-off-by: Edward Z. Yang <ezyang@fb.com>

a little jinja

6fa39e9

Signed-off-by: Edward Z. Yang <ezyang@fb.com>

Rename

63078cd

Signed-off-by: Edward Z. Yang <ezyang@fb.com>

Fix correctly

8c771d2

Signed-off-by: Edward Z. Yang <ezyang@fb.com>

fixup

e47cadd

Signed-off-by: Edward Z. Yang <ezyang@fb.com>

ezyang merged commit cd185d1 into master Aug 7, 2019

ezyang mentioned this pull request Aug 7, 2019

Rewrite torchvision packaging pytorch/vision#1209

Merged

ezyang added a commit to pytorch/vision that referenced this pull request Aug 7, 2019

Rewrite torchvision packaging (#1209)

64ccfe3

Following a similar line of inquiry to pytorch/audio#217

ezyang added a commit to pytorch/vision that referenced this pull request Aug 7, 2019

Rewrite torchvision packaging (#1209)

39f2679

Following a similar line of inquiry to pytorch/audio#217

jamarshon deleted the pr/packaging-rewrite branch August 8, 2019 02:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rewrite our packaging from scratch #217

Rewrite our packaging from scratch #217

ezyang commented Aug 7, 2019 •

edited

Loading

soumith Aug 7, 2019

ezyang Aug 7, 2019

soumith Aug 7, 2019

ezyang Aug 7, 2019

Rewrite our packaging from scratch #217

Rewrite our packaging from scratch #217

Conversation

ezyang commented Aug 7, 2019 • edited Loading

soumith Aug 7, 2019

Choose a reason for hiding this comment

ezyang Aug 7, 2019

Choose a reason for hiding this comment

soumith Aug 7, 2019

Choose a reason for hiding this comment

ezyang Aug 7, 2019

Choose a reason for hiding this comment

ezyang commented Aug 7, 2019 •

edited

Loading