[Cherrypick 0.13] Add M1 build and test jobs #6112

YosuaMichael · 2022-05-31T09:04:33Z

Cherrypick from main: #5948 and #6110

This change will add a file .github/workflows/build-m1-binaries.yml which build m1 on the CI.

EDIT: this more generally picks all M1-related stuff, including #6122 and #6132 (and a lot more stuff tracked in #6081 (comment))

This would package libpng and libjpeg.dylib into wheel files Add a very simple test step, copied from https://github.com/pytorch/pytorch.github.io/blob/1eaa33a3d3f1b83b64c5031c6dd04dbb238d6105/scripts/test_install.py#L78 Cherry-picked from https://github.com/pytorch/builder/blob/d0bc74cc363a9da5a8b6a40e883d40d25d050036/build_m1_domains.sh#L22

NicolasHug

Thanks Yosua

YosuaMichael · 2022-05-31T09:07:20Z

Thanks Yosua

Fast! Thanks @NicolasHug !

datumbox · 2022-05-31T10:24:10Z

FYI we will need to also cherrypick #6111

YosuaMichael · 2022-05-31T14:53:50Z

#6111

@datumbox once #6111 is merged on main, we can put it together on this PR

malfet

It needs some more changes to be usable on release branches, please hold until they are done. (For example, one need to change version from 0.14 back to 0.13)

datumbox

@NicolasHug grabbed the latest differences from numerous M1 related PRs merged on main.

I understand that this is a straight copy-paste of the final configs from main. They all LGTM.

NicolasHug · 2022-06-06T12:41:49Z

Before merging we will need @malfet 's input on #6117 (comment).

#6117 has made changes to .github/workflows/build-m1-binaries.yml and I wonder if the env variables are correctly set now.

datumbox · 2022-06-06T12:45:45Z

@NicolasHug I think his PR at #6117 does solve the issue and if it's not already cherrypicked, needs to be included here. I agree, let's wait to see what he thinks.

* [BE] Unify version computation Instead of hardcoding dev version in various script, use one from `version.txt` if `setup_build_version` is called without arguments Also, pass `--pre` option to M1 build/test pip install commands to build TorchVision against nightly pytorch * Pin torchvision dependency to a specific pytorch version

Clean up Conda build folder before every run Enable artifact upload to GitHub for every workflow run, but upload to Conda/S3 only on nightly pushes Test plan: `conda install -c pytorch-nightly torchvision; python -c "import torchvision;print(torchvision.io.read_image('hummingbird.jpg').shape)"`

NicolasHug · 2022-06-09T08:18:25Z

.github/workflows/build-m1-binaries.yml

+          setup_build_version
+          WHL_NAME=torchvision-${BUILD_VERSION}-cp${PY_VERS/.}-cp${PY_VERS/.}-macosx_11_0_arm64.whl
+          conda create -yp ${ENV_NAME} python=${PY_VERS} numpy libpng jpeg wheel pkg-config
+          conda run -p ${ENV_NAME} python3 -mpip install torch --pre -f https://download.pytorch.org/whl/test/cpu/torch_test.html


I have edited this line to rely on the test repo instead of the nightlies, but there are still various references to "nightly" in this script and I'm not sure what to do with them.

@malfet would you mind double checking this whole file?
(perhaps you'd like to wait after #6140?)

Note: This ended up being overwritten in #6140.

…ick013/m1-build-setup

* Adding tagged builds * Testing * Testing * Testing * Testing * Adding conda builds

Replace `steps.extract_branch.outputs.branch` (which were probably taken from https://stackoverflow.com/questions/58033366/how-to-get-the-current-branch-within-github-actions ) with straightforward `github.event.ref` Tested in https://github.com/malfet/deleteme/runs/6822647720?check_suite_focus=true and https://github.com/malfet/deleteme/runs/6822691158?check_suite_focus=true

NicolasHug · 2022-06-13T09:46:17Z

.github/workflows/build-m1-binaries.yml

+        run: |
+          # reference ends with an RC suffix
+          if [[ ${GITHUB_REF_NAME} = *-rc[0-9]* ]]; then
+            echo "CHANNEL=test" >> "$GITHUB_ENV"
+          fi


@malfet @atalman

It looks like the builds are using the nightly channel instead of the test channel, see e.g. https://github.com/pytorch/vision/runs/6858724488?check_suite_focus=true#step:4:133

Should we update the tag to also consider branches like the current release/0.13?

This is a nice followup(cc: @atalman), but for the time being feel free to just change CHANNEL environment variable on line 16 to test...

Done merged fix for this

malfet · 2022-06-13T14:38:17Z

.github/workflows/build-m1-binaries.yml

+        run: |
+          # reference ends with an RC suffix
+          if [[ ${GITHUB_REF_NAME} = *-rc[0-9]* ]]; then
+            echo "CHANNEL=test" >> "$GITHUB_ENV"
+          fi


This is a nice followup(cc: @atalman), but for the time being feel free to just change CHANNEL environment variable on line 16 to test...

malfet · 2022-06-13T14:39:26Z

.github/workflows/build-m1-binaries.yml

+        run: |
+          # reference ends with an RC suffix
+          if [[ ${GITHUB_REF_NAME} = *-rc[0-9]* ]]; then
+            echo "CHANNEL=test" >> "$GITHUB_ENV"


@atalman : Hmm, don't we need to setup channel for build_conda workflow as well?

Done! Fix is commited: fb9a761

datumbox · 2022-06-13T14:57:02Z

More things need to be cherrypicked. For example #6158

At this point, with so many changes, it might be worth checking that this PR contains all possible improvements done on main related to M1s.

.github/workflows/build-m1-binaries.yml

old

Co-authored-by: Vasilis Vryniotis <datumbox@users.noreply.github.com>

NicolasHug · 2022-06-13T15:28:00Z

At this point, with so many changes, it might be worth checking that this PR contains all possible improvements done on main related to M1s.

I tried to keep track of everything and updated #6081 (comment), but yes. There are so many PRs that this will require a very careful review.

datumbox · 2022-06-13T15:39:55Z

@NicolasHug Agreed. Happy to lend a pair of eyes at the last PR if you want. Ping me when ready.

NicolasHug

The build jobs are currently failing with a dependency resolution conflict https://github.com/pytorch/vision/runs/6864443482?check_suite_focus=true

Marking red so we don't merge by mistake.

…ick013/m1-build-setup

NicolasHug · 2022-06-14T15:31:50Z

@malfet @atalman would you mind advising on the errors above?

) * Make sure we are building against test chanell for release * Cleanup * Cleanup * Refactoring logic * Remove tagged tests

datumbox · 2022-06-15T07:45:30Z

@NicolasHug @YosuaMichael There is another PR that adds M1 support by @atalman at #6167 that passes all the tests. Might be worth merging that one instead.

YosuaMichael · 2022-06-15T08:07:26Z

@NicolasHug @YosuaMichael There is another PR that adds M1 support by @atalman at #6167 that passes all the tests. Might be worth merging that one instead.

Agree, lets use #6167 for the cherry pick and close this PR, I have edited on the release tracker, what do you think @NicolasHug ?

malfet added 2 commits May 31, 2022 10:00

Add M1 wheels binary builds (pytorch#5948)

cd7639f

YosuaMichael self-assigned this May 31, 2022

facebook-github-bot added the cla signed label May 31, 2022

YosuaMichael added topic: build topic: binaries labels May 31, 2022

NicolasHug approved these changes May 31, 2022

View reviewed changes

YosuaMichael mentioned this pull request May 31, 2022

[v0.13] Release Tracker #6081

Closed

malfet requested changes May 31, 2022

View reviewed changes

NicolasHug added 2 commits June 1, 2022 09:13

0.14 -> 0.13

56eb2b6

Cherry pick pytorch#6122 and pytorch#6132 which supersedes pytorch#6111

d3b1d3c

NicolasHug changed the title ~~[Cherrypick 0.13] Add M1 CI build setup~~ [Cherrypick 0.13] Add M1 build and test jobs Jun 6, 2022

datumbox requested a review from malfet June 6, 2022 12:31

datumbox previously approved these changes Jun 6, 2022

View reviewed changes

malfet and others added 4 commits June 8, 2022 17:31

Update version.txt

489f03f

rely on test instead of nightly

b46fd47

NicolasHug reviewed Jun 9, 2022

View reviewed changes

NicolasHug and others added 3 commits June 13, 2022 10:31

Merge branch 'release/0.13' of github.com:pytorch/vision into cherryp…

1e3d3ab

…ick013/m1-build-setup

Adding tagged builds for M1 (pytorch#6140)

434e02a

* Adding tagged builds * Testing * Testing * Testing * Testing * Adding conda builds

NicolasHug reviewed Jun 13, 2022

View reviewed changes

datumbox requested review from NicolasHug and datumbox June 13, 2022 13:54

malfet approved these changes Jun 13, 2022

View reviewed changes

NicolasHug added 2 commits June 13, 2022 15:44

Force CHANNEL=test

efc569f

Better way

9123343

datumbox reviewed Jun 13, 2022

View reviewed changes

.github/workflows/build-m1-binaries.yml Outdated Show resolved Hide resolved

.github/workflows/build-m1-binaries.yml Outdated Show resolved Hide resolved

Cherry-picking pytorch#6158

98257ec

Co-authored-by: Vasilis Vryniotis <datumbox@users.noreply.github.com>

NicolasHug requested changes Jun 13, 2022

View reviewed changes

Merge branch 'release/0.13' of github.com:pytorch/vision into cherryp…

26b3d94

…ick013/m1-build-setup

datumbox mentioned this pull request Jun 14, 2022

M1 fixes cherry #6167

Merged

Make sure we are building against test channel for release (pytorch#6168

fb9a761

) * Make sure we are building against test chanell for release * Cleanup * Cleanup * Refactoring logic * Remove tagged tests

atalman force-pushed the cherrypick013/m1-build-setup branch from cd32d58 to fb9a761 Compare June 15, 2022 00:34

YosuaMichael closed this Jun 15, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Cherrypick 0.13] Add M1 build and test jobs #6112

[Cherrypick 0.13] Add M1 build and test jobs #6112

YosuaMichael commented May 31, 2022 •

edited by NicolasHug

Loading

NicolasHug left a comment

YosuaMichael commented May 31, 2022 •

edited

Loading

datumbox commented May 31, 2022

YosuaMichael commented May 31, 2022

malfet left a comment

datumbox left a comment

NicolasHug commented Jun 6, 2022

datumbox commented Jun 6, 2022

NicolasHug Jun 9, 2022

NicolasHug Jun 13, 2022

NicolasHug Jun 13, 2022

malfet Jun 13, 2022

atalman Jun 15, 2022

malfet Jun 13, 2022

malfet Jun 13, 2022

atalman Jun 15, 2022

datumbox commented Jun 13, 2022

NicolasHug commented Jun 13, 2022

datumbox commented Jun 13, 2022

NicolasHug left a comment

NicolasHug commented Jun 14, 2022

datumbox commented Jun 15, 2022 •

edited

Loading

YosuaMichael commented Jun 15, 2022 •

edited

Loading

[Cherrypick 0.13] Add M1 build and test jobs #6112

[Cherrypick 0.13] Add M1 build and test jobs #6112

Conversation

YosuaMichael commented May 31, 2022 • edited by NicolasHug Loading

NicolasHug left a comment

Choose a reason for hiding this comment

YosuaMichael commented May 31, 2022 • edited Loading

datumbox commented May 31, 2022

YosuaMichael commented May 31, 2022

malfet left a comment

Choose a reason for hiding this comment

datumbox left a comment

Choose a reason for hiding this comment

NicolasHug commented Jun 6, 2022

datumbox commented Jun 6, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

datumbox commented Jun 13, 2022

NicolasHug commented Jun 13, 2022

datumbox commented Jun 13, 2022

NicolasHug left a comment

Choose a reason for hiding this comment

NicolasHug commented Jun 14, 2022

datumbox commented Jun 15, 2022 • edited Loading

YosuaMichael commented Jun 15, 2022 • edited Loading

YosuaMichael commented May 31, 2022 •

edited by NicolasHug

Loading

YosuaMichael commented May 31, 2022 •

edited

Loading

datumbox commented Jun 15, 2022 •

edited

Loading

YosuaMichael commented Jun 15, 2022 •

edited

Loading