Add Windows and Linux TensorFlow Package (#1)

* update .gitignore

* add windows/linux tensorflow conda package files from conda_packages repo

* update environment name

* add workflow to build tensorflow conda package

* update environment file which is not building, possible because of comments

* fix workflow: python version needs to be in quotes and each step should have a name, remove cache comments

* try different minor version

* add channel arg to upload

* try to enable long path support in the windows runner

* try 2.8.0

* try different github action checkout version

* go back to 2.9.0

* Try shorter env name

* enabling windows long path does not work locally or on the runner

* try to shorten path using substitution

* need drive letter

* create shortened link on windows

* try substitution for input and output

* tf 2.9.2

* try to shorten environment path with `CONDA_PREFIX`

* correct conda prefix

* debug build

* add prefix to build command

* try to config environment path

* setup-miniconda supports paths as environment names

* fix ubuntu environment path

* fix version and try to test import

* setup Miniconda depends on os

* try to upload using token

* try to upload without a password

* `ANACONDA_API_TOKEN` may automatically authenticate

* remove windows long path support that doesn't work

* Remove unnecessary code

* Use correct version number & reset build number

* Use anaconda password instead of token

* Use ' instaead of " in if statements

* Document manual win/linux pacakge testing

* Consolidate workflow and separate build and upload jobs

* Ensure build job runs before upload job

* Use powershell to build Windows packages

* Do not use jinja templating in name

* Make conda prefix directory shorter again

* Use ' not "

* Use jinja formatting for shell

* Split build into windows and not windows

* Add back check for runner in build conda step

* Change output folder to build
not build.tensorflow

* Retry download (w/o build)

* Always run set-up miniconda and upload to anaconda

* Add a repo checkout before setup miniconda

* Always run checkout

* List items in current directory

* Move checkout to the top and list to bottom

* List current working directory

* Move set-up python and miniconda before download

* Separate upload for windows and not windows

---------

Co-authored-by: roomrys <38435167+roomrys@users.noreply.github.com>

.conda.tensorflow/bld.bat

@@ -0,0 +1,7 @@
+@echo off
+
+set PIP_NO_INDEX=False
+set PIP_NO_DEPENDENCIES=False
+set PIP_IGNORE_INSTALLED=False
+
+pip install --no-cache-dir -r .\requirements.tensorflow.txt

.conda.tensorflow/build.sh

@@ -0,0 +1,15 @@
+#!/usr/bin/env bash
+
+export PIP_NO_INDEX=False
+export PIP_NO_DEPENDENCIES=False
+export PIP_IGNORE_INSTALLED=False
+
+pip install --no-cache-dir -r ./requirements.tensorflow.txt
+
+# Copy the activate scripts to $PREFIX/etc/conda/activate.d.
+# This will allow them to be run on environment activation.
+for CHANGE in "activate" "deactivate"
+do
+ mkdir -p "${PREFIX}/etc/conda/${CHANGE}.d"
+ cp "${RECIPE_DIR}/tensorflow_${CHANGE}.sh" "${PREFIX}/etc/conda/${CHANGE}.d/tensorflow_${CHANGE}.sh"
+done

.conda.tensorflow/meta.yaml

@@ -0,0 +1,51 @@
+# Ref: https://docs.conda.io/projects/conda-build/en/latest/resources/define-metadata.html
+
+package:
+ name: tensorflow
+ version: 2.9.2
+
+about:
+ home: https://tensorflow.org
+ license: Apache License
+ summary: 'TensorFlow conda package based on the PyPI wheels.
+
+ For GPU support, install cudatoolkit 11.3.1 and cudnn 8.2.1 which are available as conda packages on the default channel.'
+
+build:
+ number: 0
+
+source:
+ path: ../
+
+requirements:
+ build:
+ - '{{ compiler("c") }}'
+ - '{{ compiler("cxx") }}'
+
+ host:
+ - python>=3.10.0,<3.11.0
+ - numpy
+ - h5py
+ - protobuf
+ - certifi
+ - importlib-metadata
+ - six
+ - typing-extensions
+ - zipp
+ - pip
+
+ run:
+ - python>=3.10.0,<3.11.0
+ - numpy
+ - h5py
+ - protobuf
+ - certifi
+ - importlib-metadata
+ - six
+ - typing-extensions
+ - zipp
+ - pip
+
+test:
+ imports:
+ - tensorflow

.conda.tensorflow/tensorflow_activate.sh

@@ -0,0 +1,6 @@
+#!/bin/sh
+
+# Remember the old library path for when we deactivate
+export SLEAP_OLD_LD_LIBRARY_PATH=$LD_LIBRARY_PATH
+# Help CUDA find GPUs!
+export LD_LIBRARY_PATH=$CONDA_PREFIX/lib:$LD_LIBRARY_PATH

.conda.tensorflow/tensorflow_deactivate.sh

@@ -0,0 +1,4 @@
+#!/bin/sh
+
+# Reset to the old library path for when deactivating the environment
+export LD_LIBRARY_PATH=$SLEAP_OLD_LD_LIBRARY_PATH

.github/workflows/build_tensorflow.yml

@@ -4,54 +4,156 @@ on:
  push:
  branches:
  - main
- - elizabeth/add-tensorflow-macos-conda-package
+ - elizabeth/add-windows-linux-tensorflow-conda-package
 
  paths:
  - ".github/workflows/build_tensorflow.yml"
+ - ".conda.tensorflow/**"
  - ".conda.tensorflow_macos/**"
  - "environment.tensorflow.yml"
+ - "requirements.tensorflow.txt"
  - "requirements.tensorflow_macos.txt"
 
+# If RUN_BUILD_JOB is set to true, then RUN_ID will be overwritten to the current run id
+env:
+ RUN_BUILD_JOB: false # TODO(LM): Set to true
+ RUN_ID: 10824742361 # Only used if RUN_BUILD_JOB is false (to dowload build artifact)
+
 jobs:
  build:
  name: Build package (${{ matrix.os }})
  runs-on: ${{ matrix.os }}
  strategy:
- fail-fast: false
+ fail-fast: true
  matrix:
- os: ["macos-14"]
+ os: ["ubuntu-22.04", "windows-2022", "macos-14"]
+ include:
+ - os-folder: linux-64
+ - build-folder: .conda.tensorflow
+ - os: "windows-2022"
+ os-folder: win-64
+ env-prefix: C:\c\
+ - os: "macos-14"
+ os-folder: osx-arm64
+ build-folder: .conda.tensorflow_macos
  steps:
  # Checkout the repository
  - name: Checkout
+ if: env.RUN_BUILD_JOB == 'true'
  uses: actions/checkout@v4
 
- - name: Setup Miniconda (Ubuntu and MacOS)
- if: matrix.os == 'macos-14'
+ - name: Make conda prefix directory (Windows)
+ if: matrix.os == 'windows-2022'
+ shell: powershell
+ run: |
+ New-Item -ItemType Directory -Force -Path ${{ matrix.env-prefix }}
+
+ - name: Setup Miniconda
+ if: env.RUN_BUILD_JOB == 'true'
  uses: conda-incubator/setup-miniconda@v3.0.3
  with:
  python-version: "3.10"
  environment-file: environment.tensorflow.yml
- activate-environment: tf
+ activate-environment: ${{ matrix.env-prefix }}tf
 
  - name: Print environment info
+ if: env.RUN_BUILD_JOB == 'true'
  shell: bash -l {0}
  run: |
  which python
  conda info
  conda list
 
- - name: Build conda package (MacOS)
- if: matrix.os == 'macos-14'
+ - name: Build conda package (Windows)
+ if: (runner.os == 'Windows') && (env.RUN_BUILD_JOB == 'true')
+ shell: powershell
+ run: |
+ conda build --debug ${{ matrix.build-folder }} --output-folder ${{ matrix.env-prefix }}build -c conda-forge
+
+ - name: Build conda package (not Windows)
+ if: (runner.os != 'Windows') && (env.RUN_BUILD_JOB == 'true')
  shell: bash -l {0}
  run: |
- conda build --debug .conda.tensorflow_macos --output-folder build.tensorflow_macos -c conda-forge
+ conda build --debug ${{ matrix.build-folder }} --output-folder build -c conda-forge
+
+ # Upload the build artifact incase anaconda upload fails
+ - name: Upload conda package artifact
+ if: env.RUN_BUILD_JOB == 'true'
+ uses: actions/upload-artifact@v4
+ with:
+ name: tensorflow-build-${{ matrix.os-folder }}
+ path: ${{ matrix.env-prefix }}build # Upload entire build directory
+ retention-days: 1
+
+ upload:
+ name: Upload package (${{ matrix.os }})
+ runs-on: ${{ matrix.os }}
+ needs: build
+ strategy:
+ fail-fast: false # TODO(LM): Set to true
+ matrix:
+ os: ["ubuntu-22.04", "windows-2022", "macos-14"]
+ include:
+ - os-folder: win-64
+ - os: "ubuntu-22.04"
+ os-folder: linux-64
+ - os: "macos-14"
+ os-folder: osx-arm64
+
+ steps:
+ # Checkout the repository
+ - name: Checkout
+ uses: actions/checkout@v4
+
+ - name: Setup Python
+ uses: actions/setup-python@v5
+ with:
+ python-version: ${{ matrix.pyver }}
+
+ - name: Setup Miniconda
+ uses: conda-incubator/setup-miniconda@v3.0.3
+ with:
+ python-version: "3.10"
+ environment-file: environment.tensorflow.yml
+ activate-environment: tf
+
+ - name: Use current run id for conda package download
+ shell: bash -l {0}
+ if: env.RUN_BUILD_JOB == 'true'
+ run: echo "RUN_ID=${{ github.run_id }}" >> $GITHUB_ENV
+
+ # https://github.com/actions/download-artifact?tab=readme-ov-file#usage
+ - name: Download conda package artifact
+ uses: actions/download-artifact@v4
+ id: download
+ with:
+ name: tensorflow-build-${{ matrix.os-folder }}
+ path: build
+ run-id: ${{ env.RUN_ID }}
+ github-token: ${{ secrets.GITHUB_TOKEN }}
+
+ - name: List items in current directory
+ shell: bash -l {0}
+ run: |
+ ls .
+ ls -R build
+
+ - name: Upload to Anaconda (Windows)
+ if: runner.os == 'Windows'
+ env:
+ ANACONDA_PASSWORD: ${{ secrets.ANACONDA_PASSWORD }}
+ shell: bash -l {0}
+ run: |
+ anaconda login --username sleap-deps --password "$ANACONDA_PASSWORD"
+ anaconda -v upload "build/${{ matrix.os-folder }}/*.tar.bz2" --label dev --user sleap-deps
+ anaconda logout
 
- - name: Upload to Anaconda (MacOS)
- if: matrix.os == 'macos-14'
+ - name: Upload to Anaconda (not Windows)
+ if: runner.os != 'Windows'
  env:
  ANACONDA_PASSWORD: ${{ secrets.ANACONDA_PASSWORD }}
  shell: bash -l {0}
  run: |
  anaconda login --username sleap-deps --password "$ANACONDA_PASSWORD"
- anaconda -v upload build.tensorflow_macos/osx-arm64/*.tar.bz2 --label dev --user sleap-deps
+ anaconda -v upload build/${{ matrix.os-folder }}/*.tar.bz2 --label dev --user sleap-deps
  anaconda logout

README.md

@@ -7,28 +7,32 @@ There is a Github Actions workflow `build_tensorflow.yml` that uses Github runne
 
 1. Create the tensorflow build environment:
 
-```bash
-mamba env create -f environment.tensorflow.yml -n tf
-```
+ ```bash
+ mamba env create -f environment.tensorflow.yml -n tf
+ ```
 
 2. Activate the tensorflow build environment:
 
-```bash
-mamba activate tf
-```
+ ```bash
+ mamba activate tf
+ ```
 
 3. Build the tensorflow conda package:
 
-#### Mac
+ #### Windows or Linux
 
-```bash
-conda build --debug.conda.tensorflow_macos --output-folder build.tensorflow_macos -c conda-forge
-```
+ ```bash
+ conda build --debug .conda.tensorflow --output-folder build.tensorflow -c conda-forge
+ ```
 
-4. Test the conda package:
+ #### Mac
+
+ ```bash
+ conda build --debug .conda.tensorflow_macos --output-folder build.tensorflow -c conda-forge
+ ```
 
-#### Mac
+4. Test the conda package:
 
-```bash
-mamba create -n tf_macos -c ./build.tensorflow_macos -c conda-forge tensorflow
-```
+ ```bash
+ mamba create -n tf_build -c ./build.tensorflow -c conda-forge tensorflow
+ ```

requirements.tensorflow.txt

@@ -0,0 +1,4 @@
+# tensorflow==2.10.1 # https://github.com/talmolab/sleap/issues/1721
+# tensorflow==2.7.0 # 2.7.0 worked in the past
+# tensorflow==2.9.0 # Trying to match the version of tensorflow-macos filename too long?
+tensorflow==2.9.2 # 2.9.2 is the latest version in the 2.9 series