Skip to content

Commit 995cde8

Browse files
authored
Update build pipeline to use 1ES pools and manylinux (#343)
1 parent 33392b8 commit 995cde8

10 files changed

+489
-313
lines changed

.gitignore

-1
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,5 @@ Pods
4545
xcuserdata
4646

4747
# DML
48-
/dml_build
4948
/tfdml_test
5049
!third_party/dml/ci/build

BUILD.md

+160-18
Original file line numberDiff line numberDiff line change
@@ -1,14 +1,37 @@
1-
# Building TensorFlow-DirectML
1+
# Building TensorFlow-DirectML <!-- omit in toc -->
22

33
This document contains instructions for producing private builds of tensorflow-directml.
44

5-
## DirectML Branch
5+
- [DirectML Branch](#directml-branch)
6+
- [Developer Mode](#developer-mode)
7+
- [DirectX Development Files](#directx-development-files)
8+
- [Recommended Build Environment](#recommended-build-environment)
9+
- [Build Script](#build-script)
10+
- [CI/PyPI Builds](#cipypi-builds)
11+
- [Linux Wheels (Manylinux) and DirectX Libraries](#linux-wheels-manylinux-and-directx-libraries)
12+
- [Detailed Instructions: Windows](#detailed-instructions-windows)
13+
- [Install Visual Studio 2017](#install-visual-studio-2017)
14+
- [Install MSYS2](#install-msys2)
15+
- [Install Bazel](#install-bazel)
16+
- [Install Miniconda](#install-miniconda)
17+
- [Create Conda Build Environment](#create-conda-build-environment)
18+
- [Clone](#clone)
19+
- [Build](#build)
20+
- [Detailed Instructions: Linux](#detailed-instructions-linux)
21+
- [Install Development Tools](#install-development-tools)
22+
- [Install Bazel](#install-bazel-1)
23+
- [Install Miniconda](#install-miniconda-1)
24+
- [Create Conda Build Environment](#create-conda-build-environment-1)
25+
- [Clone](#clone-1)
26+
- [Build](#build-1)
27+
28+
# DirectML Branch
629

730
This project is a fork of the official [tensorflow](https://github.com/tensorflow/tensorflow) repository that targets TensorFlow v1.15. This fork will not merge upstream, since TensorFlow does not accept new features for previous releases, so the `master` branch is based off v1.15.0 from the upstream repository. DirectML changes do not appear in the `master` branch.
831

932
The `directml` branch is considered the main branch for development in this repository. This branch contains changes related to DirectML as well as security fixes in the upstream repository (e.g. changes in 1.15.1, 1.15.2, 1.15.3., etc.). Since this project is still in early development, the `directml` branch may be unstable and reference a "preview" version of DirectML that is subject to change. We recommend testing pre-built packages of [tensorflow-directml on PyPI](https://pypi.org/project/tensorflow-directml/); these builds are produced from the `directml` branch, but **only** when a stable version of DirectML is referenced. Alternatively, we recommend building against release tags associated with PyPI releases.
1033

11-
## Developer Mode
34+
# Developer Mode
1235

1336
This repository may periodically reference in-development versions of DirectML for testing new features. For example, experimental APIs are added to `DirectMLPreview.h`, which may have breaking changes; once an API appears in `DirectML.h` it is immutable. A preview build of DirectML requires *developer mode* to be enabled or it will fail to load. This restriction is intended to avoid a long-term dependency on the preview library. **Packages on PyPI will only be released when the repository depends on a stable version of DirectML.**
1437

@@ -32,24 +55,24 @@ Developer mode is, as the name indicates, only intended to be used for developme
3255
- **WSL**
3356
- Create a plain-text file in your Linux home directory, `~/.directml.conf`, with the contents `devmode = 1`.
3457

35-
## DirectX Development Files
58+
# DirectX Development Files
3659

3760
The development headers and libraries used to build DirectX-based applications are included in the Windows SDK; however, this SDK is not available when building for Linux, and some of required APIs may not yet exist in public versions of the SDK. For these reasons, the DirectX development files are integrated a little differently in this project.
3861

3962
The header files for Direct3D, DXCore, and DirectML are downloaded automatically. This project does not use any DirectX headers included in the Windows 10 SDK *except* for dxgi1_6.h, which is part of Windows 10 SDK 10.0.17763.0 or newer.
4063

4164
The use of the redistributable DirectML library is governed by a separate license that is found as part of the package (found in `tensorflow_core/python/DirectML_LICENSE.txt` when extracted).
4265

43-
## Build Environment
66+
# Recommended Build Environment
4467

45-
We've tested the following build environments, and we recommend using a Python environment manager like [Miniconda](https://docs.conda.io/en/latest/miniconda.html) to sandbox your builds.
68+
We've tested the following build environments, and we recommend using a Python environment manager like [Miniconda](https://docs.conda.io/en/latest/miniconda.html) to sandbox your builds. It may be possible to build in other environments that we have not tested.
4669

4770
- **Windows**
4871
- Visual Studio 2017 (15.x)
4972
- Windows 10 SDK 10.0.17763.0 or newer
5073
- MSYS2 20190524 or newer with the *git*, *unzip*, and *patch* packages installed
51-
- Bazel 0.24.1
52-
- Python 3.5, 3.6, or 3.7 environment with the following packages installed:
74+
- Bazel 0.26.1
75+
- Python 3.6, or 3.7 environment with the following packages installed:
5376
- six
5477
- numpy
5578
- wheel
@@ -58,19 +81,31 @@ We've tested the following build environments, and we recommend using a Python e
5881

5982
- **Linux**
6083
- Any glibc-based distro (we've tested Ubuntu 18.04+) with gcc/g++ packages
61-
- Bazel 0.24.1
62-
- Python 3.5, 3.6, or 3.7 environment with the following packages installed:
84+
- Bazel 0.26.1
85+
- Python 3.6, or 3.7 environment with the following packages installed:
6386
- six
6487
- numpy
6588
- wheel
6689
- keras_applications==1.0.6
6790
- keras_preprocessing==1.0.5
6891

69-
## Build Script
92+
# Build Script
7093

7194
A helper script, build.py, can be used to build this repository with support for DirectML. This script is a thin wrapper around the bazel commands for configuring and build TensorFlow; you may use bazel directly if you prefer, but make sure to include `--config=dml`. Run `build.py --help` for a full list of options, or inspect this file to get a full list of the bazel commands it executes.
7295

73-
For example, to build the repository and produce a Python wheel use `build.py --package` in a Python 3.5-3.8 environment with `bazel` on the PATH.
96+
# CI/PyPI Builds
97+
98+
The wheels published to [tensorflow-directml PyPI](https://pypi.org/project/tensorflow-directml/) are produced using the Azure Pipelines scripts found under [third_party/dml/ci/pipeline](third_party/dml/ci/pipeline).
99+
100+
# Linux Wheels (Manylinux) and DirectX Libraries
101+
102+
Building *portable* Linux binaries is tricky in comparison to Windows or MacOS. The [Manylinux](https://github.com/pypa/manylinux) project attempts to wrangle some of this complexity by providing standards for build environments to produce binaries that are widely usable across GLIBC-based Linux distros. Manylinux also provides container images that implement the various manylinux standards.
103+
104+
Unfortunately, the official manylinux2010 image does not support pre-built versions of bazel (the build tool for TensorFlow); it's necessary to build bazel from source or set up a manylinux2010-compliant toolchain on a different host OS. The official TensorFlow CI builds use Ubuntu 16.04 as the base OS for its container-based builds, and TFDML CI builds take a similar approach by leveraging Google's [custom-op-ubuntu16](https://hub.docker.com/layers/tensorflow/tensorflow/custom-op-ubuntu16/images/sha256-f0676af86cb61665ae20c7935340b4073e325ccbad1f2bc7b904577dd6d511c0?context=explore) image.
105+
106+
The DirectX libraries bundled into the TFDML wheel are built using a different cross-compiling technique intended to support both glibc and musl, so they may not be recognized as manylinux2010 compliant. The DX libraries only run in WSL2 distros, so this should not have any real-world consequence, but the auditwheel tool may report issues with these binaries when scanning the TFDML Linux wheels.
107+
108+
You will need to build this repository from source if the manylinux wheels on PyPI do not work for your target WSL2 distro (e.g. Alpine Linux uses musl instead of glibc). PyPI does not support uploading Linux binary wheels unless they conform to the manylinux standards.
74109

75110
# Detailed Instructions: Windows
76111

@@ -96,8 +131,8 @@ PS> C:\msys64\usr\bin\pacman.exe -S git patch unzip
96131

97132
[Bazel](https://bazel.build/) is the build tool for TensorFlow. Bazel is distributed as an executable and there is no installer step.
98133

99-
1. Download [Bazel 0.24.1](https://github.com/bazelbuild/bazel/releases/download/0.24.1/bazel-0.24.1-windows-x86_64.exe).
100-
2. Copy/rename `bazel-0.24.1-windows-x86_64.exe` to a versioned path, such as `C:\bazel\0.24.1\bazel.exe`. TensorFlow will expect bazel.exe on the `%PATH%`, so renaming the executable while retaining the version within the path is useful.
134+
1. Download [Bazel 0.26.1](https://github.com/bazelbuild/bazel/releases/download/0.26.1/bazel-0.26.1-windows-x86_64.exe).
135+
2. Copy/rename `bazel-0.26.1-windows-x86_64.exe` to a versioned path, such as `C:\bazel\0.26.1\bazel.exe`. TensorFlow will expect bazel.exe on the `%PATH%`, so renaming the executable while retaining the version within the path is useful.
101136

102137
## Install Miniconda
103138

@@ -111,7 +146,7 @@ Launch a Miniconda prompt (appears as "Anaconda PowerShell Prompt (Miniconda3)"
111146

112147
After launching the prompt, create a new environment. The examples here use the name `tfdml`, but you can choose any name you like. The idea is to sandbox any packages and dependencies in this environment. This will create a separate copy of Python and its packages in a directory `C:\miniconda3\envs\tfdml`.
113148

114-
**IMPORTANT**: make sure to use Python 3.7, 3.6, or 3.5. TensorFlow 1.15 is not supported on newer versions of Python.
149+
**IMPORTANT**: make sure to use Python 3.7 or 3.6. TensorFlow 1.15 is not supported on newer versions of Python.
115150

116151
```
117152
(base) PS> conda create --name tfdml python=3.7
@@ -150,7 +185,7 @@ Set the contents of the activation script (`%CONDA_PREFIX%\etc\conda\activate.d\
150185

151186
```PowerShell
152187
$env:TFDML_PATH_RESTORE = $env:PATH
153-
$env:PATH = "C:\bazel\0.24.1;$env:PATH"
188+
$env:PATH = "C:\bazel\0.26.1;$env:PATH"
154189
$env:PATH = "C:\msys64\usr\bin;$env:PATH"
155190
```
156191

@@ -167,7 +202,7 @@ Restart your conda environment (launch the Miniconda prompt again and activate `
167202
168203
CommandType Name Version Source
169204
----------- ---- ------- ------
170-
Application bazel.exe 0.0.0.0 C:\bazel\0.24.1\bazel.exe
205+
Application bazel.exe 0.0.0.0 C:\bazel\0.26.1\bazel.exe
171206
172207
173208
(tfdev) PS> get-command git
@@ -201,7 +236,7 @@ To produce the Python package run the following:
201236
(tfdev) PS> python build.py -p -c release
202237
```
203238

204-
After the package is built you will find a wheel package under `<PATH_TO_CLONE>\..\dml_build\python_package` (e.g. `C:\src\dml_build\python_package` in these examples). You can run `pip install` on the output .whl file to install your locally built copy of TensorFlow-DirectML.
239+
After the package is built you will find a wheel package under `<PATH_TO_CLONE>\..\tfdml_build\python_package` (e.g. `C:\src\tfdml_build\python_package` in these examples). You can run `pip install` on the output .whl file to install your locally built copy of TensorFlow-DirectML.
205240

206241
The build script has additional options you can experiment with. To see more details:
207242

@@ -210,3 +245,110 @@ The build script has additional options you can experiment with. To see more det
210245
```
211246

212247
Note that the `config` parameter accepts debug or release as an argument, but these are largely the same: debug builds are effectively just "release with debug symbols" since the output PDBs for TensorFlow without optimizations are prohibitively large.
248+
249+
# Detailed Instructions: Linux
250+
251+
These instructions use Miniconda to sandbox your build environment. This isn't strictly necessary, and there are other ways to do this (e.g. virtual machines, containers), but for the purpose of this walk-through you will use a Python virtual environment to build TFDML.
252+
253+
## Install Development Tools
254+
255+
```bash
256+
sudo apt install unzip gcc g++
257+
```
258+
259+
## Install Bazel
260+
261+
```bash
262+
wget https://github.com/bazelbuild/bazel/releases/download/0.26.1/bazel-0.26.1-installer-linux-x86_64.sh
263+
bash bazel-0.26.1-installer-linux-x86_64.sh --bin=$HOME/bin/bazel/0.26.1 --base=$HOME/.bazel
264+
```
265+
266+
## Install Miniconda
267+
268+
Miniconda is a bundle that includes [Python](https://www.python.org/), a package and environment manager named [Conda](https://docs.conda.io/projects/conda/en/latest/), and a very small set of Python packages. It is a lighter-weight alternative to Anaconda, which contains hundreds of Python packages that aren't necessary for building. The rest of this document applies equally to Anaconda if you prefer.
269+
270+
```bash
271+
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
272+
bash Miniconda3-latest-Linux-x86_64.sh
273+
```
274+
275+
## Create Conda Build Environment
276+
277+
Restart your shell if you configured conda to init in the .bashrc file. Otherwise, you can launch the base environment as follows:
278+
279+
```bash
280+
eval "$($HOME/miniconda3/bin/conda shell.bash hook)"
281+
```
282+
283+
After launching the prompt, create a new environment. The examples here use the name `tfdml`, but you can choose any name you like. The idea is to sandbox any packages and dependencies in this environment. This will create a separate copy of Python and its packages.
284+
285+
**IMPORTANT**: make sure to use Python 3.7 or 3.6. TensorFlow 1.15 is not supported on newer versions of Python.
286+
287+
```
288+
(base) ~$ conda create --name tfdml python=3.7
289+
```
290+
291+
Next, activate the environment. Activating an environment will set up the `PATH` to use the correct version of Python and its packages.
292+
293+
```
294+
(base) ~$ conda activate tfdml
295+
```
296+
297+
Within the activated environment, install the following Python packages required for TensorFlow:
298+
299+
```
300+
(tfdml) ~$ pip install six numpy wheel
301+
(tfdml) ~$ pip install keras_applications==1.0.6 --no-deps
302+
(tfdml) ~$ pip install keras_preprocessing==1.0.5 --no-deps
303+
```
304+
305+
Finally, we need to augment the `PATH` of the conda environment to include Bazel.
306+
307+
```
308+
(tfdml) :~$ mkdir -p $CONDA_PREFIX/etc/conda/activate.d
309+
(tfdml) :~$ mkdir -p $CONDA_PREFIX/etc/conda/deactivate.d
310+
(tfdml) :~$ touch $CONDA_PREFIX/etc/conda/activate.d/path.sh
311+
(tfdml) :~$ touch $CONDA_PREFIX/etc/conda/deactivate.d/path.sh
312+
```
313+
314+
Set the contents of the activation script (`$CONDA_PREFIX/etc/conda/activate.d/path.sh`). Make sure to use the correct paths where you installed Bazel:
315+
316+
```bash
317+
export TFDML_PATH_RESTORE=$PATH
318+
export PATH=$HOME/bin/bazel/0.26.1:$PATH
319+
```
320+
321+
Set the contents of the deactivation script (`$CONDA_PREFIX/etc/conda/deactivate.d/path.sh`):
322+
323+
```bash
324+
export PATH=$TFDML_PATH_RESTORE
325+
unset TFDML_PATH_RESTORE
326+
```
327+
328+
## Clone
329+
330+
Clone the repository to a location of your choosing. The examples here will assume you clone to `~/src/tensorflow-directml` for the sake of brevity, but you may clone wherever you like.
331+
332+
```
333+
(tfdml) :~$ cd ~/src
334+
(tfdml) :~$ git clone https://github.com/microsoft/tensorflow-directml.git
335+
```
336+
337+
## Build
338+
339+
Remember to activate your build environment whenever you need to build. To produce the Python package run the following:
340+
341+
```
342+
(tfdml) :~$ cd ~/src/tensorflow-directml
343+
(tfdml) :~$ python build.py -p -c release
344+
```
345+
346+
After the package is built you will find a wheel package under `~/tfdml_build/python_package`. You can run `pip install` on the output .whl file to install your locally built copy of TensorFlow-DirectML.
347+
348+
The build script has additional options you can experiment with. To see more details:
349+
350+
```
351+
(tfdml) :~$ python build.py --help
352+
```
353+
354+
Note that the `config` parameter accepts debug or release as an argument, but these are largely the same: debug builds are effectively just "release with debug symbols" since the symbols for TensorFlow without optimizations are prohibitively large.

0 commit comments

Comments
 (0)