Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CI: Use mamba-org/provision-with-micromamba to setup micromamba #2435

Merged
merged 61 commits into from
Apr 23, 2023
Merged

CI: Use mamba-org/provision-with-micromamba to setup micromamba #2435

Show file tree
Hide file tree
Changes from 20 commits
Commits
Show all changes
61 commits
Select commit Hold shift + click to select a range
f7e9764
CI: Use mamba-org/provision-with-micromamba to setup micromamba
seisman Mar 17, 2023
b05467a
Add comma between packages
seisman Mar 17, 2023
67af438
Remove comma between optional packages
seisman Mar 17, 2023
c103e11
Specify Python version
seisman Mar 17, 2023
5382970
Use separate environment files
seisman Mar 17, 2023
bda67da
sphinx-gallery is required in tests
seisman Mar 17, 2023
aa0dfc8
Merge branch 'main' into try-micromamba
seisman Mar 19, 2023
03a4ddb
Cache downloads and env
seisman Mar 19, 2023
504977b
Trigger a new CI run to see how cache works
seisman Mar 19, 2023
2abf4b3
Merge branch 'main' into try-micromamba
seisman Mar 24, 2023
8cf78c7
Refactor the 'GMT Tests Legacy' workflow
seisman Mar 24, 2023
31d85c7
Rename files using underscores
seisman Mar 24, 2023
9b0050a
Refactor the Docs workflow
seisman Mar 24, 2023
eb89e80
Refactor the Cache Data workflow
seisman Mar 24, 2023
fa7c7d6
Fixes
seisman Mar 24, 2023
464a3ea
Refactor ci_tests_dev.yaml
seisman Mar 24, 2023
8080e77
Temporarily enable GMT Tests Legacy workflow in PR
seisman Mar 24, 2023
b20f9b6
Force GMT 6.3.0 in the GMT Tests Legacy workflow
seisman Mar 24, 2023
44cda10
Always run the Tests Dev workflow in PR
seisman Mar 25, 2023
2c3e0e1
Remove comments from extra-specs
seisman Mar 25, 2023
14837f7
Merge branch 'main' into try-micromamba
seisman Mar 25, 2023
8f886ec
Merge branch 'main' into try-micromamba
seisman Mar 25, 2023
07fa2ac
Fix mamba to micromamba
seisman Mar 26, 2023
754fe52
Merge branch 'main' into try-micromamba
seisman Mar 27, 2023
ff3af0b
Merge branch 'main' into try-micromamba
seisman Mar 27, 2023
3fe4d1a
Merge branch 'main' into try-micromamba
seisman Mar 29, 2023
c640a76
Disable cache in cache-data workflow
seisman Mar 29, 2023
d84a4e1
Merge branch 'main' into try-micromamba
seisman Apr 6, 2023
2659a05
Specify Python version in cache_data workflow
seisman Apr 7, 2023
e6fbc7d
Remove a comment from ci_docs.yml and ci_tests.yaml
seisman Apr 7, 2023
d66fe43
Can not specify the gmt version in the environment file
seisman Apr 7, 2023
a979e70
Change channel defaults to nodefaults in docs.yml
seisman Apr 7, 2023
62bcc8f
Update matplotlib link in gallery example "Scatter plots with a legen…
yvonnefroehlich Apr 8, 2023
f47dedd
Merge branch 'main' into try-micromamba
seisman Apr 9, 2023
417cab4
Add channel list to cache data
seisman Apr 9, 2023
47f58d5
Remove matplotlib as a dependency
seisman Apr 9, 2023
b18d8d8
Update dependency order
seisman Apr 9, 2023
c915e39
Merge branch 'main' into try-micromamba
seisman Apr 9, 2023
b69d741
Update order of dependencies
seisman Apr 9, 2023
cfcdfbe
Temporarily enable cache_data workflow
seisman Apr 9, 2023
a8123b2
Update the release checklist for bumping GMT
seisman Apr 9, 2023
923570b
pip is a dev dependency
seisman Apr 11, 2023
6d7047c
Merge branch 'main' into try-micromamba
seisman Apr 12, 2023
0df26af
Merge branch 'main' into try-micromamba
seisman Apr 12, 2023
f591d7f
Use Micromamba in the doctests workflow
seisman Apr 12, 2023
1562377
Temporarily enable the workflow in PR
seisman Apr 12, 2023
b8490ec
No need to define python-version in matrix
seisman Apr 12, 2023
fc1e1bc
Pin GMT version in the workflow file
seisman Apr 12, 2023
36c4a01
No need to define python-version in matrix
seisman Apr 12, 2023
8bf51be
Specify gmt version in workflow files
seisman Apr 12, 2023
9f1e310
No need to define python-version in matrix
seisman Apr 12, 2023
cf4636c
Update the GMT bump checklist
seisman Apr 12, 2023
08df3b0
Merge branch 'main' into try-micromamba
seisman Apr 13, 2023
ece3db9
Remove tests_legacy.yml and use tests_with_optional_dependencies.yml …
seisman Apr 21, 2023
bd4037f
Merge branch 'main' into try-micromamba
seisman Apr 21, 2023
733ef1b
Merge branch 'main' into try-micromamba
seisman Apr 21, 2023
00c1d0a
List all dependencies in the workflow files
seisman Apr 21, 2023
69bb40d
Remove the 'micromamba list' step because provision-with-micromamba a…
seisman Apr 21, 2023
069e070
Do not specify the minimum package version
seisman Apr 21, 2023
69e0336
Sort pytest packages alphabetically
seisman Apr 23, 2023
df1c606
Disable some workflows in PR
seisman Apr 23, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 13 additions & 16 deletions .github/workflows/cache_data.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,23 +24,20 @@ jobs:
# fetch all history so that setuptools-scm works
fetch-depth: 0

# Install Mambaforge with conda-forge dependencies
- name: Setup Mambaforge
uses: conda-incubator/setup-miniconda@v2.2.0
- name: Setup Micromamba
uses: mamba-org/provision-with-micromamba@v15
with:
channels: conda-forge,nodefaults
channel-priority: strict
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The default channel-priority is strict (https://github.com/mamba-org/provision-with-micromamba#channel-priority), so no need to specify channel-priority here.

miniforge-version: latest
miniforge-variant: Mambaforge
mamba-version: "*"
run-post: false
use-mamba: true

# Install GMT and other required dependencies from conda-forge
- name: Install dependencies
run: |
mamba install gmt=6.4.0 numpy pandas xarray netCDF4 packaging \
build
environment-name: pygmt
cache-downloads: true
cache-env: true
extra-specs: |
gmt=6.4.0
numpy
pandas
xarray
netCDF4
packaging
build

# Install the package that we want to test
- name: Install the package
Expand Down
29 changes: 9 additions & 20 deletions .github/workflows/ci_docs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,30 +53,19 @@ jobs:
# fetch all history so that setuptools-scm works
fetch-depth: 0

# Install Mambaforge with conda-forge dependencies
- name: Setup Mambaforge
uses: conda-incubator/setup-miniconda@v2.2.0
- name: Setup Micromamba
uses: mamba-org/provision-with-micromamba@v15
with:
activate-environment: pygmt
python-version: ${{ matrix.python-version }}
channels: conda-forge,nodefaults
channel-priority: strict
miniforge-version: latest
miniforge-variant: Mambaforge
mamba-version: "*"
run-post: false
use-mamba: true

# Install GMT and other required dependencies from conda-forge
- name: Install dependencies
run: |
mamba install gmt=6.4.0 numpy pandas xarray netCDF4 packaging \
build ipython make myst-parser contextily geopandas \
sphinx sphinx-copybutton sphinx-design sphinx-gallery sphinx_rtd_theme
environment-file: "ci/requirements/docs.yml"
seisman marked this conversation as resolved.
Show resolved Hide resolved
cache-downloads: true
cache-env: true
# override any specifications in the environment file
extra-specs: |
python=${{ matrix.python-version }}

# Show installed pkg information for postmortem diagnostic
- name: List installed packages
run: mamba list
run: micromamba list

# Download cached remote files (artifacts) from GitHub
- name: Download remote data from GitHub
Expand Down
36 changes: 12 additions & 24 deletions .github/workflows/ci_tests.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,10 +44,10 @@ jobs:
include:
- python-version: '3.8'
numpy-version: '1.21'
optional-packages: ''
environment-file: 'ci/requirements/tests.yml'
- python-version: '3.11'
numpy-version: '1.24'
optional-packages: 'contextily geopandas ipython'
environment-file: 'ci/requirements/tests_with_optional_dependencies.yml'
timeout-minutes: 30
defaults:
run:
Expand All @@ -73,32 +73,20 @@ jobs:
# fetch all history so that setuptools-scm works
fetch-depth: 0

# Install Mambaforge with conda-forge dependencies
- name: Setup Mambaforge
uses: conda-incubator/setup-miniconda@v2.2.0
- name: Setup Micromamba
uses: mamba-org/provision-with-micromamba@v15
with:
activate-environment: pygmt
python-version: ${{ matrix.python-version }}
channels: conda-forge,nodefaults
channel-priority: strict
miniforge-version: latest
miniforge-variant: Mambaforge
mamba-version: "*"
run-post: false
use-mamba: true

# Install GMT and other required dependencies from conda-forge
- name: Install dependencies
run: |
mamba install gmt=6.4.0 numpy=${{ matrix.numpy-version }} \
pandas xarray netCDF4 packaging \
${{ matrix.optional-packages }} \
build dvc make 'pytest>=6.0' \
pytest-cov pytest-doctestplus pytest-mpl sphinx-gallery
environment-file: ${{ matrix.environment-file }}
cache-downloads: true
cache-env: true
Comment on lines +107 to +108
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

According to https://github.com/mamba-org/provision-with-micromamba/tree/v15#cache-env-key:

With the default environment cache key, separate caches will be created for each operating system (eg., Linux) and platform (eg., x64) and day (eg., 2022-01-31), and the cache will be invalidated whenever the contents of environment-file or extra-specs change.

And at https://github.com/mamba-org/provision-with-micromamba/tree/v15#notes-on-caching / mamba-org/provision-with-micromamba#38 (comment)

  • Env caching should always work OK given how Conda works, and assuming that tar + untar doesn't lose file system permissions etc. One caveat is that you might have older versions of dependencies but that is limited to the cache TTL (currently hardcoded to 1 day). If you make any changes to your Conda env folder for whatever horrible reason, the changes are included in the cached version and your build may break.

  • Download caching is generally slower because it only eliminates the download time. Actually it doesn't eliminate it, only (presumably) reduce it because it's loaded from a "faster" network (Azure Blob storage?!) Potentially it's even slower than an uncached install because download + extraction are possibly done with less parallelism. In practice it always seems to be faster though. Advantage of download caching is that you are always up to date.

So we want to use both env and download caching? We could try both for now, and can modify things later if it becomes problematic.

# override any specifications in the environment file
extra-specs: |
python=${{ matrix.python-version }}
numpy=${{ matrix.numpy-version }}

# Show installed pkg information for postmortem diagnostic
- name: List installed packages
run: mamba list
run: micromamba list

# Download cached remote files (artifacts) from GitHub
- name: Download remote data from GitHub
Expand Down
46 changes: 28 additions & 18 deletions .github/workflows/ci_tests_dev.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ on:
# push:
# branches: [ main ]
pull_request:
types: [ready_for_review]
# types: [ready_for_review]
seisman marked this conversation as resolved.
Show resolved Hide resolved
paths-ignore:
- 'doc/**'
- 'examples/**'
Expand Down Expand Up @@ -79,31 +79,41 @@ jobs:
comment-id: ${{ github.event.client_payload.github.payload.comment.id }}
body: https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }}

# Install Mambaforge with conda-forge dependencies
- name: Setup Mambaforge
uses: conda-incubator/setup-miniconda@v2.2.0
- name: Setup Micromamba
uses: mamba-org/provision-with-micromamba@v15
with:
activate-environment: pygmt
# python-version: ${{ matrix.python-version }}
channels: conda-forge,nodefaults
channel-priority: strict
miniforge-version: latest
miniforge-variant: Mambaforge
mamba-version: "*"
run-post: false
use-mamba: true

# Install dependencies from conda-forge
cache-downloads: true
cache-env: true
extra-specs: |
python=${{ matrix.python-version }}
cmake
make
ninja
curl
fftw
ghostscript
hdf5
libblas
libcblas
libgdal
liblapack
libnetcdf
pcre
zlib
geopandas

# Install dependencies from PyPI
- name: Install dependencies
run: |
mamba install python=${{ matrix.python-version }} \
ninja cmake libblas libcblas liblapack fftw libgdal \
geopandas ghostscript libnetcdf hdf5 zlib curl pcre make
pip install --pre --prefer-binary \
numpy pandas xarray netCDF4 packaging \
build contextily dvc ipython 'pytest>=6.0' pytest-cov \
pytest-doctestplus pytest-mpl sphinx-gallery

# Show installed pkg information for postmortem diagnostic
- name: List installed packages
run: micromamba list

# Pull baseline image data from dvc remote (DAGsHub)
- name: Pull baseline image data from dvc remote
if: github.event_name == 'schedule'
Expand Down
34 changes: 11 additions & 23 deletions .github/workflows/ci_tests_legacy.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ name: GMT Legacy Tests
on:
# push:
# branches: [ main ]
# pull_request:
pull_request:
seisman marked this conversation as resolved.
Show resolved Hide resolved
# types: [ready_for_review]
# paths-ignore:
# - 'doc/**'
Expand Down Expand Up @@ -47,32 +47,20 @@ jobs:
# fetch all history so that setuptools-scm works
fetch-depth: 0

# Install Mambaforge with conda-forge dependencies
- name: Setup Mambaforge
uses: conda-incubator/setup-miniconda@v2.2.0
- name: Setup Micromamba
uses: mamba-org/provision-with-micromamba@v15
with:
activate-environment: pygmt
python-version: ${{ matrix.python-version }}
channels: conda-forge,nodefaults
channel-priority: strict
miniforge-version: latest
miniforge-variant: Mambaforge
mamba-version: "*"
run-post: false
use-mamba: true

# Install GMT and other required dependencies from conda-forge
- name: Install dependencies
run: |
mamba install gmt=${{ matrix.gmt_version }} numpy \
pandas xarray netCDF4 packaging \
contextily geopandas ipython \
build dvc make 'pytest>=6.0' \
pytest-cov pytest-doctestplus pytest-mpl sphinx-gallery
environment-file: "ci/requirements/tests_legacy.yml"
seisman marked this conversation as resolved.
Show resolved Hide resolved
cache-downloads: true
cache-env: true
# override any specifications in the environment file
extra-specs: |
python=${{ matrix.python-version }}
gmt=${{ matrix.gmt_version }}

# Show installed pkg information for postmortem diagnostic
- name: List installed packages
run: mamba list
run: micromamba list

# Download cached remote files (artifacts) from GitHub
- name: Download remote data from GitHub
Expand Down
24 changes: 24 additions & 0 deletions ci/requirements/tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
name: pygmt
channels:
- conda-forge
- nodefaults
dependencies:
# Required dependencies
- pip
- gmt=6.4.0
- numpy>=1.21
- pandas
- xarray
- netCDF4
- packaging
# Development dependencies (general)
- build
- dvc
- make
# Dev dependencies (unit testing)
- matplotlib
- pytest-cov
- pytest-doctestplus
- pytest-mpl
- pytest>=6.0
- sphinx-gallery
27 changes: 27 additions & 0 deletions ci/requirements/tests_legacy.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
name: pygmt
channels:
- conda-forge
- nodefaults
dependencies:
# Required dependencies
- pip
- gmt=6.3.0
- numpy>=1.21
- pandas
- xarray
- netCDF4
- packaging
# Optional dependencies
- contextily
- geopandas
- ipython
# Development dependencies (general)
- build
- dvc
- make
# Dev dependencies (unit testing)
- matplotlib
- pytest-doctestplus
- pytest-mpl
- pytest>=6.0
- sphinx-gallery
28 changes: 28 additions & 0 deletions ci/requirements/tests_with_optional_dependencies.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
name: pygmt
channels:
- conda-forge
- nodefaults
dependencies:
# Required dependencies
- pip
- gmt=6.4.0
- numpy>=1.21
- pandas
- xarray
- netCDF4
- packaging
# Development dependencies (general)
- build
- dvc
- make
# Dev dependencies (unit testing)
- matplotlib
- pytest-cov
- pytest-doctestplus
- pytest-mpl
- pytest>=6.0
# optional dependencies
- contextily
- geopandas
- ipython
- sphinx-gallery