Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Autogenerate environment.yml file from pyproject.toml #12914

Merged
merged 26 commits into from
Oct 25, 2024
Merged
Show file tree
Hide file tree
Changes from 17 commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
8b14553
additions to pyproject.toml
drammock Oct 18, 2024
ee45879
add VTK pin from YAML
drammock Oct 18, 2024
8a3e78a
spaces around version spec operators
drammock Oct 18, 2024
4cf4f3c
TOML-sort changes induced by spaces around operators
drammock Oct 18, 2024
a5da48d
remove unhelpful comment (lots of deps duplicated between full and te…
drammock Oct 18, 2024
1a0d7b9
sorting and spacing of environment.yml
drammock Oct 18, 2024
a3f2a0e
add env auto-gen hook and associated files
drammock Oct 22, 2024
ef61f28
proof of concept: add fake dependency to trigger hook
drammock Oct 22, 2024
4e99502
remove fake dep used for proof-of-concept
drammock Oct 22, 2024
cdd25e8
Merge branch 'main' into autogen-env-yaml
drammock Oct 22, 2024
5826fa0
eric's review; don't assume space-separator in source file
drammock Oct 22, 2024
0bff690
pytest pin
drammock Oct 22, 2024
44a08e4
workaround micromamba issue
drammock Oct 23, 2024
536558b
Revert "workaround micromamba issue"
drammock Oct 24, 2024
1fb1e27
avoid excluding pyside6==6.7.0 (not found on conda-forge)
drammock Oct 24, 2024
0fc4250
avoid backslash in f-string
drammock Oct 24, 2024
32d5768
workaround micromamba ci bug
drammock Oct 24, 2024
e499ed0
FIX: Comment
larsoner Oct 24, 2024
fa07a66
FIX: Openblas
larsoner Oct 24, 2024
ab1b403
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Oct 24, 2024
f255010
missed whitespace change
drammock Oct 24, 2024
69a790d
make sure imageio installed in pip-pre job
drammock Oct 24, 2024
0e3b018
add nomkl to micromamba init
drammock Oct 24, 2024
1ed4490
make azure_deps match actions_deps for pip-pre (use test_extra)
drammock Oct 25, 2024
9385229
Revert "make sure imageio installed in pip-pre job"
drammock Oct 25, 2024
2f21782
FIX: More
larsoner Oct 25, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -64,10 +64,17 @@ repos:
files: pyproject.toml

# dependencies
- repo: local
hooks:
- id: update-env-file
name: Copy dependency changes from pyproject.toml to environment.yml
language: python
entry: ./tools/hooks/update_environment_file.py
files: pyproject.toml
- repo: local
hooks:
- id: dependency-sync
name: Sync dependency list between pyproject.toml and README.rst
name: Copy core dependencies from pyproject.toml to README.rst
language: python
entry: ./tools/hooks/sync_dependencies.py
files: pyproject.toml
Expand Down
95 changes: 44 additions & 51 deletions environment.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,65 +3,58 @@ channels:
- conda-forge
dependencies:
- python >=3.10
- pip
- numpy
- scipy
- openblas
- matplotlib
- tqdm
- pooch>=1.5
- antio >=0.4.0
- darkdetect
- decorator
- h5io
- packaging
- numba
- pandas
- pyarrow
- xlrd
- scikit-learn
- defusedxml
- dipy
- edfio >=0.2.1
- eeglabio
- h5io >=0.2.4
- h5py
- jinja2
- pillow
- statsmodels
- jupyter
- ipython !=8.7.0
- joblib
- psutil
- numexpr
- imageio
- spyder-kernels >=1.10.0
- imageio >=2.6.1
- imageio-ffmpeg >=0.4.1
- vtk >=9.2
- traitlets
- pyvista >=0.32,!=0.35.2,!=0.38.0,!=0.38.1,!=0.38.2,!=0.38.3,!=0.38.4,!=0.38.5,!=0.38.6,!=0.42.0
- pyvistaqt >=0.4
- qdarkstyle !=3.2.2
- darkdetect
- dipy
- ipyevents
- ipympl
- ipython !=8.7.0
- ipywidgets
- jinja2
- joblib
- jupyter
- lazy_loader >=0.3
- matplotlib >=3.6
- mffpy >=0.5.7
- mne-qt-browser
- nibabel
- openmeeg >=2.5.5
- nilearn
- numba
- numpy >=1.23,<3
- openmeeg >=2.5.5
- packaging
- pandas
- pillow
- pip
- pooch >=1.5
- pyarrow
- pybv
- pymatreader
- PySide6 !=6.8.0,!=6.8.0.1
- python-neo
- python-picard
- pyvista >=0.32,!=0.35.2,!=0.38.0,!=0.38.1,!=0.38.2,!=0.38.3,!=0.38.4,!=0.38.5,!=0.38.6,!=0.42.0
- pyvistaqt >=0.4
- qdarkstyle !=3.2.2
- qtpy
- pyside6 !=6.8.0,!=6.8.0.1
- mne-base
- seaborn-base
- mffpy >=0.5.7
- ipyevents
- ipywidgets
- ipympl
- scikit-learn
- scipy >=1.9
- sip
- snirf
- statsmodels
- threadpoolctl
- tqdm
- traitlets
- trame
- trame-vtk
- trame-vuetify
- jupyter_client
- nbformat
- nbclient
- mne-qt-browser
- pymatreader
- eeglabio
- edfio >=0.2.1
- pybv
- mamba
- lazy_loader
- defusedxml
- python-neo
- vtk >=9.2
- xlrd
72 changes: 38 additions & 34 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -22,12 +22,12 @@ classifiers = [
dependencies = [
"decorator",
"jinja2",
"lazy_loader>=0.3",
"matplotlib>=3.6",
"numpy>=1.23,<3",
"lazy_loader >= 0.3",
"matplotlib >= 3.6",
"numpy >= 1.23,<3",
"packaging",
"pooch>=1.5",
"scipy>=1.9",
"pooch >= 1.5",
"scipy >= 1.9",
"tqdm",
]
description = "MNE-Python project for MEG and EEG data analysis."
Expand All @@ -45,7 +45,7 @@ license = {text = "BSD-3-Clause"}
maintainers = [{email = "dan@mccloy.info", name = "Dan McCloy"}]
name = "mne"
readme = {content-type = "text/x-rst", file = "README.rst"}
requires-python = ">=3.10"
requires-python = ">= 3.10"
scripts = {mne = "mne.commands.utils:main"}

[project.optional-dependencies]
Expand All @@ -55,97 +55,99 @@ dev = ["mne[doc,test]", "rcssmin"]
# Dependencies for building the documentation
doc = [
"graphviz",
"intersphinx_registry>=0.2405.27",
"ipython!=8.7.0",
"intersphinx_registry >= 0.2405.27",
"ipython != 8.7.0", # also in "full-no-qt" and "test"
"memory_profiler",
"mne-bids",
"mne-connectivity",
"mne-gui-addons",
"neo",
"numpydoc",
"psutil",
"pydata_sphinx_theme>=0.15.2",
"pygments>=2.13",
"pydata_sphinx_theme >= 0.15.2",
"pygments >= 2.13",
"pytest",
"pyxdf",
"pyzmq!=24.0.0",
"seaborn!=0.11.2",
"pyzmq != 24.0.0",
"seaborn != 0.11.2",
"selenium",
"sphinx >= 6",
"sphinx-design",
"sphinx-gallery>=0.16",
"sphinx>=6",
"sphinx-gallery >= 0.16",
"sphinx_copybutton",
"sphinxcontrib-bibtex>=2.5",
"sphinxcontrib-bibtex >= 2.5",
"sphinxcontrib-towncrier",
"sphinxcontrib-youtube",
# https://github.com/sphinx-contrib/sphinxcontrib-towncrier/issues/92
"towncrier<24.7",
]
full = ["mne[full-no-qt]", "PyQt6!=6.6.0", "PyQt6-Qt6!=6.6.0,!=6.7.0"]
full = ["mne[full-no-qt]", "PyQt6 != 6.6.0", "PyQt6-Qt6 != 6.6.0, != 6.7.0"]
# Dependencies for full MNE-Python functionality (other than raw/epochs export)
# We first define a variant without any Qt bindings. The "complete" variant, mne[full],
# makes an opinionated choice and installs PyQt6.
# We also offter two more variants: mne[full-qt6] (which is equivalent to mne[full]),
# and mne[full-pyside6], which will install PySide6 instead of PyQt6.
full-no-qt = [
"antio>=0.4.0",
"antio >= 0.4.0",
"darkdetect",
"defusedxml",
"dipy",
"edfio>=0.2.1",
"edfio >= 0.2.1",
"eeglabio",
"h5py",
"imageio-ffmpeg>=0.4.1",
"imageio>=2.6.1",
"imageio >= 2.6.1",
"imageio-ffmpeg >= 0.4.1",
"ipyevents",
"ipympl",
"ipython != 8.7.0", # for notebook backend; also in "doc" and "test"
"ipywidgets",
"joblib",
"jupyter",
"mffpy>=0.5.7",
"mffpy >= 0.5.7",
"mne-qt-browser",
"mne[hdf5]",
"neo",
"nibabel",
"nilearn",
"numba",
"openmeeg>=2.5.5",
"openmeeg >= 2.5.5",
"pandas",
"pillow", # for `Brain.save_image` and `mne.Report`
"pyarrow", # only needed to avoid a deprecation warning in pandas
"pybv",
"pyobjc-framework-Cocoa>=5.2.0; platform_system=='Darwin'",
"pyobjc-framework-Cocoa >= 5.2.0; platform_system == 'Darwin'",
"python-picard",
"pyvista>=0.32,!=0.35.2,!=0.38.0,!=0.38.1,!=0.38.2,!=0.38.3,!=0.38.4,!=0.38.5,!=0.38.6,!=0.42.0",
"pyvistaqt>=0.4",
"qdarkstyle!=3.2.2",
"pyvista >= 0.32, != 0.35.2, != 0.38.0, != 0.38.1, != 0.38.2, != 0.38.3, != 0.38.4, != 0.38.5, != 0.38.6, != 0.42.0",
"pyvistaqt >= 0.4",
"qdarkstyle != 3.2.2",
"qtpy",
"scikit-learn",
"sip",
"snirf",
# duplicated in test_extra:
"statsmodels",
"threadpoolctl",
"traitlets",
"trame",
"trame-vtk",
"trame-vuetify",
"vtk",
"vtk >= 9.2",
"xlrd",
]
full-pyqt6 = ["mne[full]"]
full-pyside6 = ["mne[full-no-qt]", "PySide6!=6.7.0,!=6.8.0,!=6.8.0.1"]
full-pyside6 = ["mne[full-no-qt]", "PySide6 != 6.7.0, != 6.8.0, != 6.8.0.1"]
# Dependencies for MNE-Python functions that use HDF5 I/O
hdf5 = ["h5io>=0.2.4", "pymatreader"]
hdf5 = ["h5io >= 0.2.4", "pymatreader"]
# Dependencies for running the test infrastructure
test = [
"codespell",
"ipython != 8.7.0", # for testing notebook backend; also in "full-no-qt" and "doc"
"mypy",
"numpydoc",
"pre-commit",
"pytest >= 8.0",
"pytest-cov",
"pytest-qt",
"pytest-timeout",
"pytest>=8.0.0rc2",
"ruff",
"toml-sort",
"tomli; python_version<'3.11'",
Expand All @@ -155,13 +157,15 @@ test = [
]
# Dependencies for being able to run additional tests (rare/CIs/advanced devs)
test_extra = [
"edfio>=0.2.1",
"edfio >= 0.2.1",
"eeglabio",
"imageio-ffmpeg>=0.4.1",
"imageio>=2.6.1",
"imageio >= 2.6.1",
"imageio-ffmpeg >= 0.4.1",
"jupyter_client",
"mne-bids",
"mne[test]",
"nbclient",
"nbformat",
"neo",
"nitime",
"pybv",
Expand Down
88 changes: 88 additions & 0 deletions tools/hooks/update_environment_file.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
#!/usr/bin/env python

# Authors: The MNE-Python contributors.
# License: BSD-3-Clause
# Copyright the MNE-Python contributors.

import re
from pathlib import Path

import tomllib

repo_root = Path(__file__).resolve().parents[2]
with open(repo_root / "pyproject.toml", "rb") as fid:
pyproj = tomllib.load(fid)

# Get our "full" dependences from `pyproject.toml`, but actually ignore the
# "full" section as it's just "full-noqt" plus PyQt6, and for conda we need PySide
ignore = ("dev", "doc", "test", "test_extra", "full", "full-pyqt6")
deps = set(pyproj["project"]["dependencies"])
for section, section_deps in pyproj["project"]["optional-dependencies"].items():
if section not in ignore:
deps |= set(section_deps)
recursive_deps = set(d for d in deps if d.startswith("mne["))
deps -= recursive_deps
deps |= {"pip"}


def remove_spaces(version_spec):
"""Remove spaces in version specs (conda is stricter than pip about this).

https://docs.conda.io/projects/conda/en/latest/user-guide/concepts/pkg-specs.html#package-match-specifications
"""
return "".join(version_spec.split())


def split_dep(dep):
"""Separate package name from version spec."""
pattern = re.compile(r"([^!=<>]+)?([!=<>].*)?")
groups = list(pattern.match(dep).groups())
groups[1] = "" if groups[1] is None else remove_spaces(groups[1])
return tuple(map(str.strip, groups))


# python version
req_python = remove_spaces(pyproj["project"]["requires-python"])

# split package name from version spec
translations = dict(neo="python-neo")
pip_deps = set()
conda_deps = set()
for dep in deps:
package_name, version_spec = split_dep(dep)
# handle package name differences
package_name = translations.get(package_name, package_name)
# PySide6==6.7.0 only exists on PyPI, not conda-forge, so excluding it in
# `environment.yaml` breaks the solver
if package_name == "PySide6":
version_spec = version_spec.replace("!=6.7.0,", "")
# rstrip output line in case `version_spec` == ""
line = f" - {package_name} {version_spec}".rstrip()
# use pip for packages needing e.g. `platform_system` or `python_version` triaging
if ";" in version_spec:
pip_deps.add(f" {line}")
else:
conda_deps.add(line)

# TODO: temporary workaround while we wait for a release containing the fix for
# https://github.com/mamba-org/mamba/issues/3467
pip_deps.remove(" - pyobjc-framework-Cocoa >=5.2.0;platform_system=='Darwin'")

# prepare the pip dependencies section
newline = "\n" # python < 3.12 forbids backslash in {} part of f-string
pip_section = f"""\
- pip:
{newline.join(sorted(pip_deps, key=str.casefold))}
"""
pip_section = pip_section if len(pip_deps) else ""
# prepare the env file
env = f"""\
name: mne
channels:
- conda-forge
dependencies:
- python {req_python}
{newline.join(sorted(conda_deps, key=str.casefold))}
{pip_section}"""

(repo_root / "environment.yml").write_text(env)
Loading