You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This project is a fork of the official [tensorflow](https://github.com/tensorflow/tensorflow) repository that targets TensorFlow v1.15. This fork will not merge upstream, since TensorFlow does not accept new features for previous releases, so the `master` branch is based off v1.15.0 from the upstream repository. DirectML changes do not appear in the `master` branch.
8
31
9
32
The `directml` branch is considered the main branch for development in this repository. This branch contains changes related to DirectML as well as security fixes in the upstream repository (e.g. changes in 1.15.1, 1.15.2, 1.15.3., etc.). Since this project is still in early development, the `directml` branch may be unstable and reference a "preview" version of DirectML that is subject to change. We recommend testing pre-built packages of [tensorflow-directml on PyPI](https://pypi.org/project/tensorflow-directml/); these builds are produced from the `directml` branch, but **only** when a stable version of DirectML is referenced. Alternatively, we recommend building against release tags associated with PyPI releases.
10
33
11
-
##Developer Mode
34
+
# Developer Mode
12
35
13
36
This repository may periodically reference in-development versions of DirectML for testing new features. For example, experimental APIs are added to `DirectMLPreview.h`, which may have breaking changes; once an API appears in `DirectML.h` it is immutable. A preview build of DirectML requires *developer mode* to be enabled or it will fail to load. This restriction is intended to avoid a long-term dependency on the preview library. **Packages on PyPI will only be released when the repository depends on a stable version of DirectML.**
14
37
@@ -32,24 +55,24 @@ Developer mode is, as the name indicates, only intended to be used for developme
32
55
-**WSL**
33
56
- Create a plain-text file in your Linux home directory, `~/.directml.conf`, with the contents `devmode = 1`.
34
57
35
-
##DirectX Development Files
58
+
# DirectX Development Files
36
59
37
60
The development headers and libraries used to build DirectX-based applications are included in the Windows SDK; however, this SDK is not available when building for Linux, and some of required APIs may not yet exist in public versions of the SDK. For these reasons, the DirectX development files are integrated a little differently in this project.
38
61
39
62
The header files for Direct3D, DXCore, and DirectML are downloaded automatically. This project does not use any DirectX headers included in the Windows 10 SDK *except* for dxgi1_6.h, which is part of Windows 10 SDK 10.0.17763.0 or newer.
40
63
41
64
The use of the redistributable DirectML library is governed by a separate license that is found as part of the package (found in `tensorflow_core/python/DirectML_LICENSE.txt` when extracted).
42
65
43
-
##Build Environment
66
+
#Recommended Build Environment
44
67
45
-
We've tested the following build environments, and we recommend using a Python environment manager like [Miniconda](https://docs.conda.io/en/latest/miniconda.html) to sandbox your builds.
68
+
We've tested the following build environments, and we recommend using a Python environment manager like [Miniconda](https://docs.conda.io/en/latest/miniconda.html) to sandbox your builds. It may be possible to build in other environments that we have not tested.
46
69
47
70
-**Windows**
48
71
- Visual Studio 2017 (15.x)
49
72
- Windows 10 SDK 10.0.17763.0 or newer
50
73
- MSYS2 20190524 or newer with the *git*, *unzip*, and *patch* packages installed
51
-
- Bazel 0.24.1
52
-
- Python 3.5, 3.6, or 3.7 environment with the following packages installed:
74
+
- Bazel 0.26.1
75
+
- Python 3.6, or 3.7 environment with the following packages installed:
53
76
- six
54
77
- numpy
55
78
- wheel
@@ -58,19 +81,31 @@ We've tested the following build environments, and we recommend using a Python e
58
81
59
82
-**Linux**
60
83
- Any glibc-based distro (we've tested Ubuntu 18.04+) with gcc/g++ packages
61
-
- Bazel 0.24.1
62
-
- Python 3.5, 3.6, or 3.7 environment with the following packages installed:
84
+
- Bazel 0.26.1
85
+
- Python 3.6, or 3.7 environment with the following packages installed:
63
86
- six
64
87
- numpy
65
88
- wheel
66
89
- keras_applications==1.0.6
67
90
- keras_preprocessing==1.0.5
68
91
69
-
##Build Script
92
+
# Build Script
70
93
71
94
A helper script, build.py, can be used to build this repository with support for DirectML. This script is a thin wrapper around the bazel commands for configuring and build TensorFlow; you may use bazel directly if you prefer, but make sure to include `--config=dml`. Run `build.py --help` for a full list of options, or inspect this file to get a full list of the bazel commands it executes.
72
95
73
-
For example, to build the repository and produce a Python wheel use `build.py --package` in a Python 3.5-3.8 environment with `bazel` on the PATH.
96
+
# CI/PyPI Builds
97
+
98
+
The wheels published to [tensorflow-directml PyPI](https://pypi.org/project/tensorflow-directml/) are produced using the Azure Pipelines scripts found under [third_party/dml/ci/pipeline](third_party/dml/ci/pipeline).
99
+
100
+
# Linux Wheels (Manylinux) and DirectX Libraries
101
+
102
+
Building *portable* Linux binaries is tricky in comparison to Windows or MacOS. The [Manylinux](https://github.com/pypa/manylinux) project attempts to wrangle some of this complexity by providing standards for build environments to produce binaries that are widely usable across GLIBC-based Linux distros. Manylinux also provides container images that implement the various manylinux standards.
103
+
104
+
Unfortunately, the official manylinux2010 image does not support pre-built versions of bazel (the build tool for TensorFlow); it's necessary to build bazel from source or set up a manylinux2010-compliant toolchain on a different host OS. The official TensorFlow CI builds use Ubuntu 16.04 as the base OS for its container-based builds, and TFDML CI builds take a similar approach by leveraging Google's [custom-op-ubuntu16](https://hub.docker.com/layers/tensorflow/tensorflow/custom-op-ubuntu16/images/sha256-f0676af86cb61665ae20c7935340b4073e325ccbad1f2bc7b904577dd6d511c0?context=explore) image.
105
+
106
+
The DirectX libraries bundled into the TFDML wheel are built using a different cross-compiling technique intended to support both glibc and musl, so they may not be recognized as manylinux2010 compliant. The DX libraries only run in WSL2 distros, so this should not have any real-world consequence, but the auditwheel tool may report issues with these binaries when scanning the TFDML Linux wheels.
107
+
108
+
You will need to build this repository from source if the manylinux wheels on PyPI do not work for your target WSL2 distro (e.g. Alpine Linux uses musl instead of glibc). PyPI does not support uploading Linux binary wheels unless they conform to the manylinux standards.
2. Copy/rename `bazel-0.24.1-windows-x86_64.exe` to a versioned path, such as `C:\bazel\0.24.1\bazel.exe`. TensorFlow will expect bazel.exe on the `%PATH%`, so renaming the executable while retaining the version within the path is useful.
2. Copy/rename `bazel-0.26.1-windows-x86_64.exe` to a versioned path, such as `C:\bazel\0.26.1\bazel.exe`. TensorFlow will expect bazel.exe on the `%PATH%`, so renaming the executable while retaining the version within the path is useful.
101
136
102
137
## Install Miniconda
103
138
@@ -111,7 +146,7 @@ Launch a Miniconda prompt (appears as "Anaconda PowerShell Prompt (Miniconda3)"
111
146
112
147
After launching the prompt, create a new environment. The examples here use the name `tfdml`, but you can choose any name you like. The idea is to sandbox any packages and dependencies in this environment. This will create a separate copy of Python and its packages in a directory `C:\miniconda3\envs\tfdml`.
113
148
114
-
**IMPORTANT**: make sure to use Python 3.7, 3.6, or 3.5. TensorFlow 1.15 is not supported on newer versions of Python.
149
+
**IMPORTANT**: make sure to use Python 3.7or 3.6. TensorFlow 1.15 is not supported on newer versions of Python.
115
150
116
151
```
117
152
(base) PS> conda create --name tfdml python=3.7
@@ -150,7 +185,7 @@ Set the contents of the activation script (`%CONDA_PREFIX%\etc\conda\activate.d\
150
185
151
186
```PowerShell
152
187
$env:TFDML_PATH_RESTORE = $env:PATH
153
-
$env:PATH = "C:\bazel\0.24.1;$env:PATH"
188
+
$env:PATH = "C:\bazel\0.26.1;$env:PATH"
154
189
$env:PATH = "C:\msys64\usr\bin;$env:PATH"
155
190
```
156
191
@@ -167,7 +202,7 @@ Restart your conda environment (launch the Miniconda prompt again and activate `
@@ -201,7 +236,7 @@ To produce the Python package run the following:
201
236
(tfdev) PS> python build.py -p -c release
202
237
```
203
238
204
-
After the package is built you will find a wheel package under `<PATH_TO_CLONE>\..\dml_build\python_package` (e.g. `C:\src\dml_build\python_package` in these examples). You can run `pip install` on the output .whl file to install your locally built copy of TensorFlow-DirectML.
239
+
After the package is built you will find a wheel package under `<PATH_TO_CLONE>\..\tfdml_build\python_package` (e.g. `C:\src\tfdml_build\python_package` in these examples). You can run `pip install` on the output .whl file to install your locally built copy of TensorFlow-DirectML.
205
240
206
241
The build script has additional options you can experiment with. To see more details:
207
242
@@ -210,3 +245,110 @@ The build script has additional options you can experiment with. To see more det
210
245
```
211
246
212
247
Note that the `config` parameter accepts debug or release as an argument, but these are largely the same: debug builds are effectively just "release with debug symbols" since the output PDBs for TensorFlow without optimizations are prohibitively large.
248
+
249
+
# Detailed Instructions: Linux
250
+
251
+
These instructions use Miniconda to sandbox your build environment. This isn't strictly necessary, and there are other ways to do this (e.g. virtual machines, containers), but for the purpose of this walk-through you will use a Python virtual environment to build TFDML.
Miniconda is a bundle that includes [Python](https://www.python.org/), a package and environment manager named [Conda](https://docs.conda.io/projects/conda/en/latest/), and a very small set of Python packages. It is a lighter-weight alternative to Anaconda, which contains hundreds of Python packages that aren't necessary for building. The rest of this document applies equally to Anaconda if you prefer.
After launching the prompt, create a new environment. The examples here use the name `tfdml`, but you can choose any name you like. The idea is to sandbox any packages and dependencies in this environment. This will create a separate copy of Python and its packages.
284
+
285
+
**IMPORTANT**: make sure to use Python 3.7 or 3.6. TensorFlow 1.15 is not supported on newer versions of Python.
286
+
287
+
```
288
+
(base) ~$ conda create --name tfdml python=3.7
289
+
```
290
+
291
+
Next, activate the environment. Activating an environment will set up the `PATH` to use the correct version of Python and its packages.
292
+
293
+
```
294
+
(base) ~$ conda activate tfdml
295
+
```
296
+
297
+
Within the activated environment, install the following Python packages required for TensorFlow:
Set the contents of the activation script (`$CONDA_PREFIX/etc/conda/activate.d/path.sh`). Make sure to use the correct paths where you installed Bazel:
315
+
316
+
```bash
317
+
export TFDML_PATH_RESTORE=$PATH
318
+
export PATH=$HOME/bin/bazel/0.26.1:$PATH
319
+
```
320
+
321
+
Set the contents of the deactivation script (`$CONDA_PREFIX/etc/conda/deactivate.d/path.sh`):
322
+
323
+
```bash
324
+
export PATH=$TFDML_PATH_RESTORE
325
+
unset TFDML_PATH_RESTORE
326
+
```
327
+
328
+
## Clone
329
+
330
+
Clone the repository to a location of your choosing. The examples here will assume you clone to `~/src/tensorflow-directml` for the sake of brevity, but you may clone wherever you like.
Remember to activate your build environment whenever you need to build. To produce the Python package run the following:
340
+
341
+
```
342
+
(tfdml) :~$ cd ~/src/tensorflow-directml
343
+
(tfdml) :~$ python build.py -p -c release
344
+
```
345
+
346
+
After the package is built you will find a wheel package under `~/tfdml_build/python_package`. You can run `pip install` on the output .whl file to install your locally built copy of TensorFlow-DirectML.
347
+
348
+
The build script has additional options you can experiment with. To see more details:
349
+
350
+
```
351
+
(tfdml) :~$ python build.py --help
352
+
```
353
+
354
+
Note that the `config` parameter accepts debug or release as an argument, but these are largely the same: debug builds are effectively just "release with debug symbols" since the symbols for TensorFlow without optimizations are prohibitively large.
0 commit comments