Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add more OS and Python versions to the matrix #16

Merged
merged 25 commits into from
Jul 25, 2022
Merged

add more OS and Python versions to the matrix #16

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
b729455
add more OS and Python versions to the matrix
zacharyburnett Jul 19, 2022
d706e36
add OS version to cache name
zacharyburnett Jul 19, 2022
60ad189
only build YAML file for Linux
zacharyburnett Jul 19, 2022
e16d9c9
supply empty extension for BSD sed
zacharyburnett Jul 19, 2022
436a81a
install Python on macos
zacharyburnett Jul 19, 2022
4bec731
use `runner.os`
zacharyburnett Jul 19, 2022
1381b77
use platform-agnostic paths
zacharyburnett Jul 19, 2022
909dac5
minor fixes
zacharyburnett Jul 19, 2022
8ba53a8
install `pytest`
zacharyburnett Jul 19, 2022
6df466f
explicit home directory
zacharyburnett Jul 19, 2022
0365d66
use home directory
zacharyburnett Jul 19, 2022
511dbe1
use env variables
zacharyburnett Jul 19, 2022
47bb5a5
use tilde
zacharyburnett Jul 19, 2022
f437906
use platform-specific paths (because `$HOME` and `$CONDA` do not work)
zacharyburnett Jul 19, 2022
29a3025
remove `cache` ID
zacharyburnett Jul 19, 2022
2f46347
platform-specific paths
zacharyburnett Jul 19, 2022
65de6c3
test `stsynphot` from source
zacharyburnett Jul 21, 2022
d007ae7
write platform-specific paths to environment variables
zacharyburnett Jul 22, 2022
ea3fe9b
remove tilde
zacharyburnett Jul 22, 2022
146f1e8
parameterize tests
zacharyburnett Jul 22, 2022
cbffd20
fix for variable lookup
zacharyburnett Jul 22, 2022
3040ae7
add `stsynphot` to `from_source` matrix
zacharyburnett Jul 22, 2022
db7fce4
reorder matrices
zacharyburnett Jul 22, 2022
f944beb
test environment variables
zacharyburnett Jul 25, 2022
bf8ebb0
consolidate environment variables
zacharyburnett Jul 25, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
224 changes: 167 additions & 57 deletions .github/workflows/build.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,53 +14,68 @@ defaults:

jobs:
build:
name: build environment (Python ${{ matrix.python }})
runs-on: ubuntu-latest
name: build environment (Python ${{ matrix.python }}, ${{ matrix.os }})
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
python: [ '3.9' ]
os: [ ubuntu-latest, macos-latest ]
python: [ '3.8', '3.9', '3.10' ]
steps:
- uses: actions/checkout@v3.0.2
- run: sed -i "s/python==3.9/python==${{ matrix.python }}.*/g" spacetelescope-env-latest.yml
- if: runner.os == 'Linux'
run: |
echo "HOME=$(echo /home/runner)" >> $GITHUB_ENV
echo "CONDA=$(echo /usr/share/miniconda)" >> $GITHUB_ENV
sed -i "s/python==3.9/python==${{ matrix.python }}.*/g" spacetelescope-env-latest.yml
- if: runner.os == 'MacOS'
run: |
echo "HOME=$(echo /Users/runner)" >> $GITHUB_ENV
echo "CONDA=$(echo /usr/local/miniconda)" >> $GITHUB_ENV
sed -i "" "s/python==3.9/python==${{ matrix.python }}.*/g" spacetelescope-env-latest.yml
- uses: actions/setup-python@v4.1.0
with:
python-version: ${{ matrix.python }}
- uses: mtkennerly/dunamai-action@v1.1.0
id: version
- run: echo "::set-output name=date::$(date +'%Y%m%d')"
id: date
- uses: actions/cache@v3.0.5
id: cache
with:
path: |
/home/runner/conda_pkgs_dir
/usr/share/miniconda/envs/spacetelescope-env-py${{ matrix.python }}-latest
${{ env.HOME }}/conda_pkgs_dir
${{ env.CONDA }}/envs/spacetelescope-env-${{ runner.os }}-py${{ matrix.python }}-latest
key: conda-${{ runner.os }}-py${{ matrix.python }}-${{ steps.date.outputs.date }}
- uses: conda-incubator/setup-miniconda@v2.1.1
with:
activate-environment: spacetelescope-env-py${{ matrix.python }}-latest
activate-environment: spacetelescope-env-${{ runner.os }}-py${{ matrix.python }}-latest
environment-file: spacetelescope-env-latest.yml
python-version: ${{ matrix.python }}.*
auto-update-conda: true
use-only-tar-bz2: true
- run: conda update --all
if: steps.cache.outputs.cache-hit != 'true'
- run: conda install -y pytest pytest-xdist
- run: pytest -n auto tests/test_import.py
- run: conda env export > spacetelescope-env-py${{ matrix.python }}-v${{ steps.version.outputs.version }}.yml
- run: sed -i "s/spacetelescope-env-py${{ matrix.python }}-latest/spacetelescope-env-py${{ matrix.python }}-v${{ steps.version.outputs.version }}/g" spacetelescope-env-py${{ matrix.python }}-v${{ steps.version.outputs.version }}.yml
if: runner.os == 'Linux'
- run: sed -i "s/spacetelescope-env-${{ runner.os }}-py${{ matrix.python }}-latest/spacetelescope-env-py${{ matrix.python }}-v${{ steps.version.outputs.version }}/g" spacetelescope-env-py${{ matrix.python }}-v${{ steps.version.outputs.version }}.yml
if: runner.os == 'Linux'
- run: cat spacetelescope-env-py${{ matrix.python }}-v${{ steps.version.outputs.version }}.yml
if: runner.os == 'Linux'
- uses: actions/upload-artifact@v3.1.0
if: runner.os == 'Linux'
with:
name: spacetelescope-env-py${{ matrix.python }}-v${{ steps.version.outputs.version }}.yml
path: spacetelescope-env-py${{ matrix.python }}-v${{ steps.version.outputs.version }}.yml
unit_tests:
name: run `${{ matrix.package }}` unit tests (Python ${{ matrix.python }})
name: run `${{ matrix.package }}` unit tests (Python ${{ matrix.python }}, ${{ matrix.os }})
needs: [ build ]
runs-on: ubuntu-latest
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
package: [ acstools, ccdproc, costools, jwst, reftools, stsynphot, synphot, wfpc2tools ]
python: [ '3.9' ]
os: [ ubuntu-latest, macos-latest ]
python: [ '3.8', '3.9', '3.10' ]
package: [ acstools, ccdproc, costools, jwst, reftools, synphot, wfpc2tools ]
include:
- package: acstools
dependencies: [ ci-watson, pytest-astropy-header, pytest-remotedata ]
Expand All @@ -77,58 +92,79 @@ jobs:
# dependencies: [ pytest-remotedata, pytest-astropy-header ]
- package: reftools
dependencies: [ pytest-remotedata, pytest-astropy-header ]
- package: stsynphot
dependencies: [ ci-watson, pytest-astropy ]
pytest_args: --ignore=stsynphot/data --ignore=stsynphot/commissioning
- package: synphot
dependencies: [ pytest-astropy ]
#- package: wfc3tools
# dependencies: [ ]
- package: wfpc2tools
steps:
- uses: actions/checkout@v3.0.2
- run: sed -i "s/python==3.9/python==${{ matrix.python }}.*/g" spacetelescope-env-latest.yml
- if: runner.os == 'Linux'
run: |
echo "HOME=$(echo /home/runner)" >> $GITHUB_ENV
echo "CONDA=$(echo /usr/share/miniconda)" >> $GITHUB_ENV
sed -i "s/python==3.9/python==${{ matrix.python }}.*/g" spacetelescope-env-latest.yml
- if: runner.os == 'MacOS'
run: |
echo "HOME=$(echo /Users/runner)" >> $GITHUB_ENV
echo "CONDA=$(echo /usr/local/miniconda)" >> $GITHUB_ENV
sed -i "" "s/python==3.9/python==${{ matrix.python }}.*/g" spacetelescope-env-latest.yml
- run: |
echo "PYSYN_CDBS=${{ env.HOME }}/trds" >> $GITHUB_ENV
echo "CRDS_PATH=${{ env.HOME }}/crds_cache" >> $GITHUB_ENV
- run: echo "::set-output name=date::$(date +'%Y%m%d')"
id: date
- uses: actions/cache@v3.0.5
with:
path: |
/home/runner/conda_pkgs_dir
/usr/share/miniconda/envs/spacetelescope-env-py${{ matrix.python }}-latest
${{ env.HOME }}/conda_pkgs_dir
${{ env.CONDA }}/envs/spacetelescope-env-${{ runner.os }}-py${{ matrix.python }}-latest
key: conda-${{ runner.os }}-py${{ matrix.python }}-${{ steps.date.outputs.date }}
- uses: conda-incubator/setup-miniconda@v2.1.1
with:
activate-environment: spacetelescope-env-py${{ matrix.python }}-latest
activate-environment: spacetelescope-env-${{ runner.os }}-py${{ matrix.python }}-latest
environment-file: spacetelescope-env-latest.yml
python-version: ${{ matrix.python }}.*
auto-update-conda: true
use-only-tar-bz2: true
- run: conda install -y pytest pytest-xdist
- run: pip install ${{ join(matrix.dependencies, ' ') }}
if: matrix.dependencies != ''
- run: echo "CRDS_CONTEXT=$(crds list --operational-context)" >> $GITHUB_ENV
if: ${{ matrix.CRDS_SERVER_URL }} != ''
env:
PYSYN_CDBS: ${{ env.PYSYN_CDBS }}
CRDS_PATH: ${{ env.CRDS_PATH }}
CRDS_SERVER_URL: ${{ matrix.CRDS_SERVER_URL }}
- uses: actions/cache@v3.0.5
if: ${{ matrix.CRDS_SERVER_URL }} != ''
with:
path: ${{ env.CRDS_PATH }}
key: crds-${{ env.CRDS_CONTEXT }}
- run: crds sync ${{ matrix.CRDS_CONTEXT }}
if: ${{ matrix.CRDS_SERVER_URL }} != ''
- run: pytest -n auto ${{ matrix.pytest_args }} --pyargs ${{ matrix.package }}
env:
PYSYN_CDBS: /home/runner/trds
CRDS_PATH: /home/runner/crds_cache
PYSYN_CDBS: ${{ env.PYSYN_CDBS }}
CRDS_PATH: ${{ env.CRDS_PATH }}
CRDS_SERVER_URL: ${{ matrix.CRDS_SERVER_URL }}
unit_tests_from_source:
name: run `${{ matrix.package }}` unit tests (Python ${{ matrix.python }}) from source
name: run `${{ matrix.package }}` unit tests (Python ${{ matrix.python }}, ${{ matrix.os }}) from source
needs: [ build ]
runs-on: ubuntu-latest
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
package: [ calcos, hstcal ]
python: [ '3.9' ]
os: [ ubuntu-latest, macos-latest ]
python: [ '3.8', '3.9', '3.10' ]
package: [ calcos, hstcal, stsynphot ]
include:
- package: calcos
repository: spacetelescope/calcos
dependencies: [ ci_watson ]
test_directory: tests
#- package: drizzlepac
# repository: spacetelescope/drizzlepac
# dependencies: [ ci-watson, crds, pytest-remotedata ]
# test_directory: tests
- package: hstcal
repository: spacetelescope/hstcal
dependencies: [ ci-watson ]
Expand All @@ -137,115 +173,189 @@ jobs:
#- package: stistools
# repository: spacetelescope/stistools
# dependencies: [ six ]
# test_directory: tests
- package: stsynphot
repository: spacetelescope/stsynphot_refactor
dependencies: [ ci-watson, pytest-astropy ]
steps:
- uses: actions/checkout@v3.0.2
with:
path: spacetelescope-env-distribution
- run: sed -i "s/python==3.9/python==${{ matrix.python }}.*/g" spacetelescope-env-distribution/spacetelescope-env-latest.yml
- if: runner.os == 'Linux'
run: |
echo "HOME=$(echo /home/runner)" >> $GITHUB_ENV
echo "CONDA=$(echo /usr/share/miniconda)" >> $GITHUB_ENV
sed -i "s/python==3.9/python==${{ matrix.python }}.*/g" spacetelescope-env-distribution/spacetelescope-env-latest.yml
- if: runner.os == 'MacOS'
run: |
echo "HOME=$(echo /Users/runner)" >> $GITHUB_ENV
echo "CONDA=$(echo /usr/local/miniconda)" >> $GITHUB_ENV
sed -i "" "s/python==3.9/python==${{ matrix.python }}.*/g" spacetelescope-env-distribution/spacetelescope-env-latest.yml
- run: |
echo "PYSYN_CDBS=${{ env.HOME }}/trds" >> $GITHUB_ENV
echo "CRDS_PATH=${{ env.HOME }}/crds_cache" >> $GITHUB_ENV
- run: echo "::set-output name=date::$(date +'%Y%m%d')"
id: date
- uses: actions/cache@v3.0.5
with:
path: |
/home/runner/conda_pkgs_dir
/usr/share/miniconda/envs/spacetelescope-env-py${{ matrix.python }}-latest
${{ env.HOME }}/conda_pkgs_dir
${{ env.CONDA }}/envs/spacetelescope-env-${{ runner.os }}-py${{ matrix.python }}-latest
key: conda-${{ runner.os }}-py${{ matrix.python }}-${{ steps.date.outputs.date }}
- uses: conda-incubator/setup-miniconda@v2.1.1
with:
activate-environment: spacetelescope-env-py${{ matrix.python }}-latest
activate-environment: spacetelescope-env-${{ runner.os }}-py${{ matrix.python }}-latest
environment-file: spacetelescope-env-distribution/spacetelescope-env-latest.yml
python-version: ${{ matrix.python }}.*
auto-update-conda: true
use-only-tar-bz2: true
- run: echo "::set-output name=version::$(pip show ${{ matrix.package }} | awk '$1 == "Version:" {print $2}')"
id: version
- uses: actions/checkout@v3.0.2
with:
path: ${{ matrix.package }}
repository: ${{ matrix.repository }}
ref: ${{ steps.version.outputs.version }}
- run: cd ${{ matrix.package }} && conda install -y pytest pytest-xdist
- run: cd ${{ matrix.package }} && pip install ${{ join(matrix.dependencies, ' ') }}
if: matrix.dependencies != ''
- run: echo "CRDS_CONTEXT=$(crds list --operational-context)" >> $GITHUB_ENV
if: ${{ matrix.CRDS_SERVER_URL }} != ''
env:
PYSYN_CDBS: ${{ env.PYSYN_CDBS }}
CRDS_PATH: ${{ env.CRDS_PATH }}
CRDS_SERVER_URL: ${{ matrix.CRDS_SERVER_URL }}
- uses: actions/cache@v3.0.5
if: ${{ matrix.CRDS_SERVER_URL }} != ''
with:
path: ${{ env.CRDS_PATH }}
key: crds-${{ env.CRDS_CONTEXT }}
- run: crds sync ${{ matrix.CRDS_CONTEXT }}
if: ${{ matrix.CRDS_SERVER_URL }} != ''
- run: cd ${{ matrix.package }} && pytest -n auto ${{ matrix.pytest_args }} ${{ matrix.test_directory }}
env:
PYSYN_CDBS: /home/runner/trds
CRDS_PATH: /home/runner/crds_cache
PYSYN_CDBS: ${{ env.PYSYN_CDBS }}
CRDS_PATH: ${{ env.CRDS_PATH }}
CRDS_SERVER_URL: ${{ matrix.CRDS_SERVER_URL }}
smoke_tests:
name: run `${{ matrix.package }}` smoke tests (Python ${{ matrix.python }})
name: run `${{ matrix.package }}` smoke tests (Python ${{ matrix.python }}, ${{ matrix.os }})
needs: [ build ]
runs-on: ubuntu-latest
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
os: [ ubuntu-latest, macos-latest ]
python: [ '3.8', '3.9', '3.10' ]
package: [ calcos, drizzlepac ]
python: [ '3.9' ]
include:
- package: calcos
run: calcos la8n01qkq_rawtag_a.fits
jref: /home/runner/crds_cache/hst/references/hst/
CRDS_SERVER_URL: https://hst-crds.stsci.edu
jref: hst/references/hst/
- package: drizzlepac
run: pytest -n auto tests/test_drizzlepac.py
jref: /home/runner/crds_cache/hst/references/hst/
CRDS_SERVER_URL: https://hst-crds.stsci.edu
jref: hst/references/hst/
steps:
- uses: actions/checkout@v3.0.2
with:
lfs: true
- run: sed -i "s/python==3.9/python==${{ matrix.python }}.*/g" spacetelescope-env-latest.yml
- if: runner.os == 'Linux'
run: |
echo "HOME=$(echo /home/runner)" >> $GITHUB_ENV
echo "CONDA=$(echo /usr/share/miniconda)" >> $GITHUB_ENV
sed -i "s/python==3.9/python==${{ matrix.python }}.*/g" spacetelescope-env-latest.yml
- if: runner.os == 'MacOS'
run: |
echo "HOME=$(echo /Users/runner)" >> $GITHUB_ENV
echo "CONDA=$(echo /usr/local/miniconda)" >> $GITHUB_ENV
sed -i "" "s/python==3.9/python==${{ matrix.python }}.*/g" spacetelescope-env-latest.yml
- run: |
echo "PYSYN_CDBS=${{ env.HOME }}/trds" >> $GITHUB_ENV
echo "CRDS_PATH=${{ env.HOME }}/crds_cache" >> $GITHUB_ENV
- run: echo "::set-output name=date::$(date +'%Y%m%d')"
id: date
- uses: actions/cache@v3.0.5
with:
path: |
/home/runner/conda_pkgs_dir
/usr/share/miniconda/envs/spacetelescope-env-py${{ matrix.python }}-latest
${{ env.HOME }}/conda_pkgs_dir
${{ env.CONDA }}/envs/spacetelescope-env-${{ runner.os }}-py${{ matrix.python }}-latest
key: conda-${{ runner.os }}-py${{ matrix.python }}-${{ steps.date.outputs.date }}
- uses: conda-incubator/setup-miniconda@v2.1.1
with:
activate-environment: spacetelescope-env-py${{ matrix.python }}-latest
activate-environment: spacetelescope-env-${{ runner.os }}-py${{ matrix.python }}-latest
environment-file: spacetelescope-env-latest.yml
python-version: ${{ matrix.python }}.*
auto-update-conda: true
use-only-tar-bz2: true
- run: pip install pytest pytest-xdist
- uses: actions/cache@v3.0.5
with:
path: 'tests/data/'
key: data-${{ hashFiles('tests/data/*') }}
- run: conda install -y pytest pytest-xdist
- run: echo "CRDS_CONTEXT=$(crds list --operational-context)" >> $GITHUB_ENV
if: ${{ matrix.CRDS_SERVER_URL }} != ''
env:
PYSYN_CDBS: ${{ env.PYSYN_CDBS }}
CRDS_PATH: ${{ env.CRDS_PATH }}
CRDS_SERVER_URL: ${{ matrix.CRDS_SERVER_URL }}
jref: ${{ env.CRDS_PATH }}/${{ matrix.jref }}
- uses: actions/cache@v3.0.5
if: ${{ matrix.CRDS_SERVER_URL }} != ''
with:
path: ${{ env.CRDS_PATH }}
key: crds-${{ env.CRDS_CONTEXT }}
- run: crds sync ${{ matrix.CRDS_CONTEXT }}
if: ${{ matrix.CRDS_SERVER_URL }} != ''
- run: ${{ matrix.run }}
env:
PYSYN_CDBS: /home/runner/trds
CRDS_PATH: /home/runner/crds_cache
PYSYN_CDBS: ${{ env.PYSYN_CDBS }}
CRDS_PATH: ${{ env.CRDS_PATH }}
CRDS_SERVER_URL: ${{ matrix.CRDS_SERVER_URL }}
jref: ${{ matrix.jref }}
jref: ${{ env.CRDS_PATH }}/${{ matrix.jref }}
export:
name: export environment
name: export environment (Python ${{ matrix.python }}, ${{ matrix.os }})
needs: [ unit_tests, unit_tests_from_source, smoke_tests ]
if: github.event_name == 'release' && github.event.action == 'published'
runs-on: ubuntu-latest
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
python: [ '3.9' ]
os: [ ubuntu-latest, macos-latest ]
python: [ '3.8', '3.9', '3.10' ]
steps:
- uses: actions/checkout@v3.0.2
- if: runner.os == 'Linux'
run: |
echo "HOME=$(echo /home/runner)" >> $GITHUB_ENV
echo "CONDA=$(echo /usr/share/miniconda)" >> $GITHUB_ENV
- if: runner.os == 'MacOS'
run: |
echo "HOME=$(echo /Users/runner)" >> $GITHUB_ENV
echo "CONDA=$(echo /usr/local/miniconda)" >> $GITHUB_ENV
- uses: actions/setup-python@v4.1.0
with:
python-version: ${{ matrix.python }}
- uses: mtkennerly/dunamai-action@v1.1.0
id: version
- uses: actions/cache@v3.0.5
id: cache
with:
path: |
/home/runner/conda_pkgs_dir
/usr/share/miniconda/envs/spacetelescope-env-py${{ matrix.python }}-latest
${{ env.HOME }}/conda_pkgs_dir
${{ env.CONDA }}/envs/spacetelescope-env-${{ runner.os }}-py${{ matrix.python }}-latest
key: conda-${{ runner.os }}-py${{ matrix.python }}-${{ steps.date.outputs.date }}
- uses: conda-incubator/setup-miniconda@v2.1.1
with:
activate-environment: spacetelescope-env-py${{ matrix.python }}-latest
activate-environment: spacetelescope-env-${{ runner.os }}-py${{ matrix.python }}-latest
environment-file: spacetelescope-env-latest.yml
python-version: ${{ matrix.python }}.*
auto-update-conda: true
use-only-tar-bz2: true
- run: conda env export > spacetelescope-env-py${{ matrix.python }}-v${{ steps.version.outputs.version }}.yml
- run: sed -i "s/spacetelescope-env-py${{ matrix.python }}-latest/spacetelescope-env-py${{ matrix.python }}-v${{ steps.version.outputs.version }}/g" spacetelescope-env-py${{ matrix.python }}-v${{ steps.version.outputs.version }}.yml
if: runner.os == 'Linux'
- run: sed -i "s/spacetelescope-env-${{ runner.os }}-py${{ matrix.python }}-latest/spacetelescope-env-py${{ matrix.python }}-v${{ steps.version.outputs.version }}/g" spacetelescope-env-py${{ matrix.python }}-v${{ steps.version.outputs.version }}.yml
if: runner.os == 'Linux'
- uses: svenstaro/upload-release-action@2.3.0
if: runner.os == 'Linux'
with:
file: spacetelescope-env-py${{ matrix.python }}-v${{ steps.version.outputs.version }}.yml
Loading