Using Nova Reusable Build Workflows

Creating your own pipelines for building and distributing binaries from scratch is hard. Monitoring, alerting, and verification are even harder. Let us help you! With Nova Reusable Workflows, you have a comprehensive set of shared components that you can simply call to get binary builds and distribution working across your entire build matrix out-of-the-box. For more involved build processes, we have a number of hooks for domain-specific customizability. No need to copy dozens of bash scripts between repos any longer!

Quick Start

Let's say you want to distribute Python Wheels for Linux systems for your package. Create a file called .github/workflows/build-wheels-linux.yml in your repo. Add the following to the file, and change the fields marked TODO for your use-case:

name: Build Linux Wheels

on:
  pull_request:
  push:
    branches:
      - nightly
  workflow_dispatch:

jobs:
  generate-matrix:
    uses: pytorch/test-infra/.github/workflows/generate_binary_build_matrix.yml@main
    with:
      package-type: wheel
      os: linux
      test-infra-repository: pytorch/test-infra
      test-infra-ref: main
      # TODO #1: Want to build with CUDA support? Enter "enable" or "disable".
      with-cuda: disable
      # TODO #2: Want to build with RoCM support? Enter "enable" or "disable".
      with-rocm: disable
      # TODO #3: Want to build with CPU support? Enter "enable" or "disable".
      with-cpu: disable
  build:
    needs: generate-matrix
    name: ${{ matrix.repository }}
    uses: pytorch/test-infra/.github/workflows/build_wheels_linux.yml@main
    with:
      # TODO #4: Enter the name of your repo
      repository: <Your_Organization_Name/Your_Repo_Name>
      ref: ""
      # TODO #5: Have custom build steps before building the wheel? Pass the path to the shell script with those steps below.
      pre-script: <path/to/your/Custom_Pre_Build_Script.sh>
      # TODO #6: Same as above, but post-build steps. If you have none, just use "".
      post-script: <path/to/your/Custom_Post_Build_Script.sh>
      # TODO #7: Want to verify the correctness of your binaries before distributing them? Add custom python smoke tests below.
      smoke-test-script: <path/to/your/Smoke_Tests.py>
      # TODO #8: Enter the name of your python package below. (for example, torchvision)
      package-name: <Your_Python_Package_Name>
      test-infra-repository: pytorch/test-infra
      test-infra-ref: main
      build-matrix: ${{ needs.generate-matrix.outputs.matrix }}
      trigger-event: ${{ github.event_name }}
    secrets:
      AWS_PYTORCH_UPLOADER_ACCESS_KEY_ID: ${{ secrets.AWS_PYTORCH_UPLOADER_ACCESS_KEY_ID }}
      AWS_PYTORCH_UPLOADER_SECRET_ACCESS_KEY: ${{ secrets.AWS_PYTORCH_UPLOADER_SECRET_ACCESS_KEY }}

And that's it! Every time a push is done to your repo's nightly branch, this will build, verify, and upload nightlies to download.pytorch.org. Every time a push is done to your release candidate branches, the same workflow will be run. And official releases will also be done via this workflow.

How It Works

We have 6 "base" workflows in pytorch/test-infra that take care of most of the logic for creating and uploading binaries. The workflows cover the set {Linux, MacOS, Windows} x {Wheels, Conda}:

• build_wheels_linux.yml
• build_wheels_macos.yml
• build_wheels_windows.yml
• build_conda_linux.yml
• build_conda_macos.yml
• build_conda_windows.yml

Note that Linux/Mac workflows are used to publish nightlies whereas Windows workflows are still under development.

To run these workflows, you need to write a "caller" workflow in the .github/workflows/ directory of your repo. It is best to have a separate caller workflow for each base workflow you want to call. Each caller workflow does 3 things:

1. Configure Triggers (when should build workflow should be run?)
2. Generate the Build Matrix
3. Calls the base workflow

Configure Triggers

The fields you define in the on: section of the yaml config determine which triggers your workflow runs after. This part is vanilla GitHub Actions, so this should be a good reference: Triggering a Workflow. Generally you want your workflows to run on pushes to nightly and release candidate branches, so you can list those branches under the push: field. Additionally, you may want these jobs triggered on PR's, so add the pull_request: field. Lastly, it's nice to be able to manually trigger your workflow from the GitHub UI. To enable this, add the workflow_dispatch: trigger. Your on: section may end up looking something like this:

on:
  pull_request:
  push:
    branches:
      - nightly
  workflow_dispatch:

Generate the Build Matrix

The pytorch/test-infra repo contains a workflow called generate_binary_build_matrix.yml that generates a build matrix based on provided inputs. For example, you'll typically want to support various Python Versions, GPU architectures, etc. This build matrix enumerates all of these configurations that you want your binaries to be built with. You can click on the link above to view the full list of inputs to the build matrix generation job, but these are the main inputs you'll likely need to set:

• os: the Operating System you want these binaries to support
• package-type: either wheel or conda (the package you are building)
• with-cuda: toggles whether you need to build separate binaries with GPU support. Default is enable, set to disable if dedicated CUDA builds are not required.

The call to generate the build matrix should be its own job in the jobs: section of your caller. It may look something like this:

jobs:
  generate-matrix:
    uses: pytorch/test-infra/.github/workflows/generate_binary_build_matrix.yml@main
    with:
      package-type: wheel
      os: linux
      test-infra-repository: pytorch/test-infra
      test-infra-ref: main
      with-cuda: disable

Calling the base workflow

In this step, we call one of the 6 base workflows with some inputs. The exact inputs you need to provide will vary by the exact base workflow you're calling. You can click on the links to those workflows above to view the detailed list of expected inputs, default values, and explanations. Here are a few of the key inputs to keep in mind:

• repository: the name of your repository
• pre-script: a bash script that defines steps that you want to run pre-build. For example, installing a set of dependencies could go here
• post-script: a bash script that defines steps that you want to run post-build
• smoke-test-script: a Python script that tests the health of your built binary. If no script is provided, we will just check if the import works
• package-name: the name of your base python module. For example, torchvision or torchtext
• conda-package-directory: (Conda-only) the directory where your meta.yaml file lives
• runner-type: (Mac-only) the type of Mac runner to use. Use macos-12 for x86 MacOS or macos-m1-12 for Arm64 MacOS

To upload you binaries to appropriate distribution channels, you must provide secrets as well. For conda builds, you will need to pass CONDA_PYTORCHBOT_TOKEN, while wheels builds will need AWS_PYTORCH_UPLOADER_ACCESS_KEY_ID and AWS_PYTORCH_UPLOADER_SECRET_ACCESS_KEY to push binaries to download.pytorch.org.

You must add the call to the base workflow as another job after the generate-matrix job in the jobs: section. It may look as follows:

  build:
    needs: generate-matrix
    name: ${{ matrix.repository }}
    uses: pytorch/test-infra/.github/workflows/build_wheels_linux.yml@main
    with:
      repository: pytorch/vision
      ref: ""
      pre-script: packaging/pre_build_script.sh
      post-script: packaging/post_build_script.sh
      smoke-test-script: tests/smoke_test.py
      package-name: torchvision
      test-infra-repository: pytorch/test-infra
      test-infra-ref: main
      build-matrix: ${{ needs.generate-matrix.outputs.matrix }}
      trigger-event: ${{ github.event_name }}
    secrets:
      AWS_PYTORCH_UPLOADER_ACCESS_KEY_ID: ${{ secrets.AWS_PYTORCH_UPLOADER_ACCESS_KEY_ID }}
      AWS_PYTORCH_UPLOADER_SECRET_ACCESS_KEY: ${{ secrets.AWS_PYTORCH_UPLOADER_SECRET_ACCESS_KEY }}

Enable document preview on the HUD

To enable docs preview on the HUD, you need to do two things:

1. Use the default linux job to compile the documentation. This part can be done in different ways, depending on the project.
2. Copy the docs preview to the specific RUNNER_DOCS_DIR directory. The S3 path will look like this: pytorch/{REPO_NAME}/{PULL_REQUEST_NUMBER}. Here is an example of the workflow upload job:

upload-docs:
    uses: pytorch/test-infra/.github/workflows/linux_job.yml@main
    with:
      runner: linux.2xlarge
      script: |
        echo "hello" > "${RUNNER_DOCS_DIR}/index.html"

Examples

Nova Workflows are now used by torchvision, torchtext, torchrec, (WIP) torchaudio for binary builds. For reference, all of torchvision's caller workflows can be found here: https://github.com/pytorch/vision/tree/main/.github/workflows. The Nova binary build workflows are named build-*.yml.

Pull request builds

Pull request builds are limited, by default, to only one version of Python since our CI data shows that failures in one version of Python typically strongly correlate with failures in other versions of Python.

In order to bypass this limitation and test for all versions of Python you can add the ciflow/binaries/all label to your pull request and push a new commit in order to get testing for all versions of Python.

Setup nightly trigger and permission to upload the binaries

After having the build workflows in place, the last step is to release the wheel starting with nightly releases so that people can start downloading it. The recommend approach is to leverage PyTorch nightly channel to install the package, for example pip install --pre YOUR_PACKAGE --index-url https://download.pytorch.org/whl/nightly/cpu. To achieve this, we need to do the following steps:

1. Add pytorchbot account to your repo as a collaborator with write permission to the repo. This is the one responsible to update the nightly branch everyday.
2. Create an empty nightly branch on the repo if you haven't done that already. Please remember to setup branch protection rule for nightly with write permission to the branch granted only to the pytorchbot from the previous step.

git switch --orphan nightly
git commit --allow-empty -m "Initial commit on nightly"
git push origin nightly

3. Reach out to PyTorch Dev Infra to help you setup a pytorchbot-env on your repo with the necessary credentials.
4. Submit a PR to create a new nightly trigger on test-infra. For example, #5003. Note that you could create nightly release from main or from your stable branch. For example, TorchVision uses main while PyTorch uses its stable branch viable/strict.
5. [Internal] Submit a diff to grant the new workflow the permission to upload their artifacts, i.e. D54874599. Once land, the permission will be granted after a few hours.