DEP: move dev-only dependencies to requirements files

.github/dependabot.yml

@@ -1,5 +1,14 @@
 version: 2
 updates:
+- package-ecosystem: pip
+ directory: /requirements
+ schedule:
+ interval: monthly
+ groups:
+ actions:
+ patterns:
+ - '*'
+
 - package-ecosystem: github-actions
  directory: /.github/workflows
  schedule:

.github/workflows/type-checking.yaml

@@ -37,7 +37,7 @@ jobs:
  - name: Build
  run: |
  python3 -m pip install --upgrade pip
- python3 -m pip install -e .[typecheck]
+ python3 -m pip install -e . -r requirements/typecheck.txt
 
  - run: python -m pip list
 

MANIFEST.in

@@ -70,5 +70,6 @@ include yt/default.mplstyle
 
 prune yt/frontends/_skeleton
 recursive-include yt/frontends/amrvac *.par
+recursive-exclude requirements *.txt
 exclude .codecov.yml .coveragerc .git-blame-ignore-revs .gitmodules .hgchurn .mailmap
 exclude .pre-commit-config.yaml clean.sh nose_answer.cfg nose_unit.cfg nose_ignores.txt

doc/source/developing/building_the_docs.rst

@@ -90,7 +90,7 @@ the top level of a local copy, run
 
 .. code-block:: bash
 
- $ python -m pip install -e ".[doc]"
+ $ python -m pip install -e . -r requirements/docs.txt
 
 Quick versus Full Documentation Builds
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

doc/source/developing/testing.rst

@@ -441,54 +441,47 @@ Handling yt Dependencies
 ------------------------
 
 Our dependencies are specified in ``pyproject.toml``. Hard dependencies are found in
-``options.install_requires``, while optional dependencies are specified in
-``options.extras_require``. The ``full`` target contains the specs to run our
+``project.dependencies``, while optional dependencies are specified in
+``project.optional-dependencies``. The ``full`` target contains the specs to run our
 test suite, which are intended to be as modern as possible (we don't set upper
-limits to versions unless we need to). The ``minimal`` target is used to check
-that we don't break backward compatibility with old versions of upstream
-projects by accident. It is intended to pin strictly our minimal supported
-versions. The ``test`` target specifies the tools needed to run the tests, but
+limits to versions unless we need to).
+
+The ``test`` target specifies the tools needed to run the tests, but
 not needed by yt itself.
 
+Documentation and typechecking requirements are found in ``requirements/``,
+and used in ``tests/ci_install.sh``.
+
 **Python version support.**
-When a new Python version is released, it takes about
-a month or two for yt to support it, since we're dependent on bigger projects
-like numpy and matplotlib. We vow to follow numpy's deprecation plan regarding
-our supported versions for Python and numpy, defined formally in `NEP 29
-<https://numpy.org/neps/nep-0029-deprecation_policy.html>`_. However, we try to
-avoid bumping our minimal requirements shortly before a yt release.
+We vow to follow numpy's deprecation plan regarding our supported versions for Python
+and numpy, defined formally in
+`NEP 29 <https://numpy.org/neps/nep-0029-deprecation_policy.html>`_, but generally
+support larger version intervals than recommended in this document.
 
 **Third party dependencies.**
+We attempt to make yt compatible with a wide variety of upstream software
+versions.
 However, sometimes a specific version of a project that yt depends on
 causes some breakage and must be blacklisted in the tests or a more
 experimental project that yt depends on optionally might change sufficiently
 that the yt community decides not to support an old version of that project.
 
 **Note.**
 Some of our optional dependencies are not trivial to install and their support
-may vary across platforms. To manage such issue, we currently use requirement
-files in additions to ``pyproject.toml``. They are found in
-``tests/*requirements.txt`` and used in ``tests/ci_install.sh``.
-
-We attempt to make yt compatible with a wide variety of upstream software
-versions. However, sometimes a specific version of a project that yt depends on
-causes some breakage and must be blacklisted in the tests or a more
-experimental project that yt depends on optionally might change sufficiently
-that the yt community decides not to support an old version of that project.
-
-To handle cases like this, the versions of upstream software projects installed
-on the machines running the yt test suite are pinned to specific version
-numbers that must be updated manually. This prevents breaking the yt tests when
-a new version of an upstream dependency is released and allows us to manage
-updates in upstream projects at our pace.
+may vary across platforms.
 
 If you would like to add a new dependency for yt (even an optional dependency)
 or would like to update a version of a yt dependency, you must edit the
-``tests/test_requirements.txt`` file, this path is relative to the root of the
-repository. This file contains an enumerated list of direct dependencies and
-pinned version numbers. For new dependencies, simply append the name of the new
+``pyproject.toml`` file. For new dependencies, simply append the name of the new
 dependency to the end of the file, along with a pin to the latest version
 number of the package. To update a package's version, simply update the version
 number in the entry for that package.
 
-Finally, we also run a set of tests with "minimal" dependencies installed. When adding tests that depend on an optional dependency, you can wrap the test with the ``yt.testing.requires_module decorator`` to ensure it does not run during the minimal dependency tests (see yt/frontends/amrvac/tests/test_read_amrvac_namelist.py for a good example). If for some reason you need to update the listing of packages that are installed for the "minimal" dependency tests, you will need to edit ``tests/test_minimal_requirements.txt``.
+Finally, we also run a set of tests with "minimal" dependencies installed.
+When adding tests that depend on an optional dependency, you can wrap the test
+with the ``yt.testing.requires_module decorator`` to ensure it does not run
+during the minimal dependency tests (see
+``yt/frontends/amrvac/tests/test_read_amrvac_namelist.py`` for a good example).
+If for some reason you need to update the listing of packages that are installed
+for the "minimal" dependency tests, you will need to update
+``requirements/minimal_env.txt``.

pyproject.toml

@@ -42,6 +42,8 @@ keywords = [
  "astronomy astrophysics visualization amr adaptivemeshrefinement",
 ]
 requires-python = ">=3.9.2"
+
+# keep in sync with requirements/minimal_env.txt
 dependencies = [
  "cmyt>=1.1.2",
  "ewah-bool-utils>=1.2.0",
@@ -190,37 +192,9 @@ full = [
 ]
 
 # dev-only extra targets
-doc = [
- "alabaster>=0.7.13",
- "bottle>=0.12.25",
- "ipykernel>=6.29.4",
- "jinja2<3.1.0", # see https://github.com/readthedocs/readthedocs.org/issues/9037
- "jupyter-client>=8.3.1",
- "nbsphinx>=0.9.3",
- "nose~=1.3.7; python_version < '3.10'",
- "pytest>=6.1",
- "pyx>=0.15",
- "sphinx>=7.2.5",
- "sphinx-bootstrap-theme>=0.8.1",
- "sphinx-rtd-theme>=1.3.0",
-]
 mapserver = [
  "bottle",
 ]
-minimal = [
- "cmyt==1.1.2",
- "ewah-bool-utils==1.2.0",
- "matplotlib==3.5",
- "more-itertools==8.4",
- "numpy==1.19.3",
- "packaging==20.9",
- "pillow==8.0.0",
- "tomli-w==0.4.0",
- "tqdm==3.4.0",
- "unyt==2.9.2",
- "tomli==1.2.3;python_version < '3.11'",
- "typing-extensions==4.4.0;python_version < '3.12'",
-]
 test = [
  "pyaml>=17.10.0",
  "pytest>=6.1",
@@ -231,13 +205,6 @@ test = [
  "nose-timer~=1.0.0; python_version < '3.10'",
  "imageio!=2.35.0", # see https://github.com/yt-project/yt/issues/4966
 ]
-typecheck = [
- "mypy==1.8.0",
- "types-PyYAML==6.0.12.12",
- "types-chardet==5.0.4.6",
- "types-requests==2.31.0.20240125",
- "typing-extensions==4.4.0; python_version < '3.12'",
-]
 
 [project.scripts]
 yt = "yt.utilities.command_line:run_main"

requirements/docs.txt

@@ -0,0 +1,12 @@
+alabaster>=0.7.13
+bottle>=0.12.25
+ipykernel>=6.29.4
+jinja2<3.1.0 # see https://github.com/readthedocs/readthedocs.org/issues/9037
+jupyter-client>=8.3.1
+nbsphinx>=0.9.3
+nose~=1.3.7; python_version < '3.10'
+pytest>=6.1
+pyx>=0.15
+sphinx>=7.2.5
+sphinx-bootstrap-theme>=0.8.1
+sphinx-rtd-theme>=1.3.0

requirements/minimal_env.txt

@@ -0,0 +1,69 @@
+# This file was autogenerated by uv via the following command:
+# uv pip compile pyproject.toml --python=3.9 --python-platform=x86_64-unknown-linux-gnu --resolution=lowest-direct --no-build
+cmyt==1.1.2
+ # via yt (pyproject.toml)
+colorspacious==1.1.2
+ # via cmyt
+cycler==0.12.1
+ # via matplotlib
+ewah-bool-utils==1.2.0
+ # via yt (pyproject.toml)
+fonttools==4.53.1
+ # via matplotlib
+kiwisolver==1.4.7
+ # via matplotlib
+matplotlib==3.5.0
+ # via
+ # yt (pyproject.toml)
+ # cmyt
+more-itertools==8.4.0
+ # via
+ # yt (pyproject.toml)
+ # cmyt
+mpmath==1.3.0
+ # via sympy
+numpy==1.19.3
+ # via
+ # yt (pyproject.toml)
+ # cmyt
+ # colorspacious
+ # ewah-bool-utils
+ # matplotlib
+ # unyt
+packaging==20.9
+ # via
+ # yt (pyproject.toml)
+ # matplotlib
+ # setuptools-scm
+pillow==8.0.0
+ # via
+ # yt (pyproject.toml)
+ # matplotlib
+pyparsing==3.1.4
+ # via
+ # matplotlib
+ # packaging
+python-dateutil==2.9.0.post0
+ # via matplotlib
+setuptools==74.1.2
+ # via setuptools-scm
+setuptools-scm==8.1.0
+ # via matplotlib
+six==1.16.0
+ # via python-dateutil
+sympy==1.13.2
+ # via unyt
+tomli==1.2.3
+ # via
+ # yt (pyproject.toml)
+ # setuptools-scm
+tomli-w==0.4.0
+ # via yt (pyproject.toml)
+tqdm==3.4.0
+ # via yt (pyproject.toml)
+typing-extensions==4.4.0
+ # via
+ # yt (pyproject.toml)
+ # setuptools-scm
+unyt==2.9.2
+ # via yt (pyproject.toml)

requirements/typecheck.txt

@@ -0,0 +1,5 @@
+mypy==1.8.0
+types-PyYAML==6.0.12.12
+types-chardet==5.0.4.6
+types-requests==2.31.0.20240125
+typing-extensions==4.4.0; python_version < '3.12'

tests/ci_install.sh

@@ -32,7 +32,7 @@ fi
 # but the primary intention is to embed this script in CI jobs
 if [[ ${dependencies} == "minimal" ]]; then
  # test with minimal versions of runtime dependencies
- python -m pip install -e ".[test,minimal]"
+ python -m pip install -e ".[test]" -r requirements/minimal_env.txt
 elif [[ ${dependencies} == "cartopy" ]]; then
  python -m pip install 'cartopy>=0.22'
  # scipy is an optional dependency to cartopy