Weird dependency resolving (for magika → numpy → distutils)

[kytta]

My issue is similar to the one described in #6509, but I wanted to dig deeper, and I really fail to understand how the resolver works.

Description

I am working on a CLI tool, which depends on magika, which is a file detection software with deep learning. I want my tool to be available for every Python starting with 3.8. Magika does not support Python 3.13, and I want to handle it gracefully, so I only install it when the version of Python is under 3.13.

# pyproject.toml
[project]
name = "mytool"
version = "0.1.0"
dependencies = [
    "magika; python_version < '3.13'",
]
requires-python = ">=3.8"

[build-system]
requires = ["flit_core >=3.2,<4"]
build-backend = "flit_core.buildapi"

I then want to lock and sync the dependencies, but uv lock fails because it fails to find distutils.

$ uv lock
Using Python 3.12.5 interpreter at: /Library/Frameworks/Python.framework/Versions/3.12/bin/python3
Resolved 18 packages in 14ms

$ uv sync
Using Python 3.12.5 interpreter at: /Library/Frameworks/Python.framework/Versions/3.12/bin/python3
Creating virtualenv at: .venv
Resolved 18 packages in 1ms
error: Failed to prepare distributions
  Caused by: Failed to fetch wheel: numpy==1.24.4
  Caused by: Build backend failed to determine extra requires with `build_wheel()` with exit status: 1
--- stdout:

--- stderr:
Traceback (most recent call last):
  File "<string>", line 8, in <module>
  File "/Users/nikita/Library/Caches/uv/builds-v0/.tmp0idacA/lib/python3.12/site-packages/setuptools/__init__.py", line 10, in <module>
    import distutils.core
ModuleNotFoundError: No module named 'distutils'
---

Problem

The comments in #6509 correctly suggest that the numpy version is too old for the environment, as distutils is gone from 3.12. Alas, I do not understand why it resolves to this version in the first place.

The pyproject.toml from above defined the range for my CLI to be >=3.8, and it depends on magika if the version is <3.13. So, just from this I expect at least two resolution markers: "below 3.13" and "3.13 and above".

magika itself does have some variable dependencies. magika==0.5.0 has the following in its wheel's metadata:

...
Requires-Python: >=3.8,<3.12
...
Requires-Dist: numpy (>=1.24.4,<2.0.0)
...

So, For every Python between 3.8 and 3.11, use any numpy older than 1.24.24.

magika==0.5.1 added support for Python 3.12, with the following metadata:

...
Requires-Python: >=3.8,<3.13
...
Requires-Dist: numpy (>=1.24,<2.0) ; python_version >= "3.8" and python_version < "3.9"
Requires-Dist: numpy (>=1.26,<2.0) ; python_version >= "3.9" and python_version < "3.13"
...

So, now it says: use any numpy older than 1.24.24, unless you are above 3.9, in which case it's stricter.

Expected behaviour

Based on this, I expect three different lockfile resolutions for my package:

Python >=3.8,<3.9 aka 3.8.*

mytool
  - magika==0.5.1
    - numpy==1.24.*

Python >=3.9,<3.13

mytool
  - magika==0.5.1
    - numpy==1.26.*

Python >=3.13

mytool
  - (no magika because version to high)

This looks easy (at least for me). Just take the latest magika==0.5.1 (it supports 3.8), and take numpy 1.24 for Python 3.8 and 1.26 for anything else.

Actual behaviour

This is not what happens, though. uv.lock does define resolution markers that I expect:

# uv.lock
version = 1
requires-python = ">=3.8"
resolution-markers = [
    "python_full_version < '3.9'",
    "python_full_version >= '3.9' and python_full_version < '3.13'",
    "python_full_version >= '3.13'",
]
...

But then, it weirdly decides to use an older magika version for the newer Pythons:

[[package]]
name = "mytool"
version = "0.1.0"
source = { editable = "." }
dependencies = [
    { name = "magika", version = "0.5.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.9' and python_full_version < '3.13'" },
    { name = "magika", version = "0.5.1", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.9'" },
]

Which then proceeds to only pull numpy==1.24.4, as it theoretically satisfies both ranges:

[[package]]
name = "numpy"
version = "1.24.4"
...

I kinda understand why it might do this; after all, we want as little packages as possible to simplify the lock file, but isn't it a better idea to use the newer package whenever possible? I assume one can say that numpy is at fault for not releasing a patch for 1.24 that would specify that it does not support Python 3.12, but isn't there a different way to solve this?

Things I've tried

Set Python <3.13 for mytool

If I tell that my whole too does not support Python 3.13, uv just resolves to the latest magika==0.5.1 and installs without problem. It also correctly locks two numpy versions: 1.24 for Python 3.8 and 1.26 for everything else

Set magika>=0.5.1

If I explicitly request the latest version of magika, uv can't lock, giving me this error:

  × No solution found when resolving dependencies for split (python_full_version >= '3.9' and python_full_version < '3.13'):
  ╰─▶ Because only the following versions of numpy{python_full_version >= '3.9' and python_full_version < '3.13'} are available:
          numpy{python_full_version >= '3.9' and python_full_version < '3.13'}<=1.26.0
          numpy{python_full_version >= '3.9' and python_full_version < '3.13'}==1.26.1
          numpy{python_full_version >= '3.9' and python_full_version < '3.13'}==1.26.2
          numpy{python_full_version >= '3.9' and python_full_version < '3.13'}==1.26.3
          numpy{python_full_version >= '3.9' and python_full_version < '3.13'}==1.26.4
          numpy{python_full_version >= '3.9' and python_full_version < '3.13'}>2.0
      and the requested Python version (>=3.8) does not satisfy Python>=3.9, we can conclude that all of:
          numpy{python_full_version >= '3.9' and python_full_version < '3.13'}>=1.26.0,<1.26.2
          numpy{python_full_version >= '3.9' and python_full_version < '3.13'}>1.26.2,<1.26.3
          numpy{python_full_version >= '3.9' and python_full_version < '3.13'}>1.26.3,<1.26.4
          numpy{python_full_version >= '3.9' and python_full_version < '3.13'}>1.26.4,<2.0
       are incompatible.
      And because the requested Python version (>=3.8) does not satisfy Python>=3.9 and magika{python_full_version < '3.13'}==0.5.1 depends on numpy{python_full_version >= '3.9' and python_full_version <
      '3.13'}>=1.26,<2.0, we can conclude that magika{python_full_version < '3.13'}==0.5.1 cannot be used.
      And because only magika{python_full_version < '3.13'}<=0.5.1 is available and your project depends on magika{python_full_version < '3.13'}>=0.5.1, we can conclude that your project's requirements are
      unsatisfiable.

I'm surprised to read that "magika{python_full_version < '3.13'}==0.5.1 depends on numpy{python_full_version >= '3.9' and python_full_version < '3.13'}>=1.26,<2.0". Sure, it does, but it also does depend on numpy{python_full_version >= '3.8' and python_full_version < '3.9'}>=1.24,<2.0. Can't it use it for "the requested Python version (>=3.8)"?

Environment

uv platform: macOS 13.6.9 Ventura, Python 3.12.5 from python.org
uv version: uv 0.4.1 (Homebrew 2024-08-30)

---

I had troubles searching for similar issues, as I didn't know what keywords to use; sorry, if this turns out to be a duplicate!

[kytta]

TL;DR: I want to have this pyproject:

# pyproject.toml
[project]
name = "mytool"
requires-python = ">=3.8"
dependencies = [
    "magika>=0.5.1; python_version < '3.13'",
]

resolve to this:

	Python 3.8	Python 3.9–3.12	Python 3.13–
magika	0.5.1	0.5.1	none
numpy	1.24.4	1.26.4	none

---

You can do this manually with PDM:

pdm lock --python="<3.9"
pdm lock --python=">=3.9,<3.13" --append
pdm lock --python=">=3.13" --append

But since uv already correctly defines the resolution markers, this should be possible to be done automatically by it

[charliermarsh]

In general we try to use the fewest number of versions to satisfy dependencies. But I think we should consider adding an alternate strategy where we try to use the newest versions for each environment.

[charliermarsh]

Something like "fewest-versions" vs. "most-compatible".

[dimbleby]

There is perhaps more then one thing going on in this report.

The failure to solve the case magika>=0.5.1 is a straight bug regardless of strategy, no?

[kytta]

In general we try to use the fewest number of versions to satisfy dependencies. But I think we should consider adding an alternate strategy where we try to use the newest versions for each environment.

Fact is, for my problem it's not really required, as both the current and the desired approach use the same amount of dependencies. The only difference is "two magikas, one numpy" versus "one magika, two numpys".

Maybe the strategy should consider how deep the dependencies are? Although, picking the "shallowest" is not necessarily the best dedupe strategy... 🤔

[charliermarsh]

I'm also slightly confused by the error but we have some open PRs that will fix a lot of the Python-version-related stuff so hesitant to dig in until those merge (#6882, #6268).

[charliermarsh]

By the way, there actually is a way to instruct uv to get the exact solve you want here, but it also needs #6268 to work properly:

[project]
name = "mytool"
version = "0.0.1"
requires-python = ">=3.8"
dependencies = [
    "magika>=0.5.1; python_version < '3.13'",
]

[tool.uv]
environments = [
    "python_version >= '3.13'",
    "python_version >= '3.9' and python_version < '3.13'",
    "python_version < '3.9'"
]

(Beware, I'm sharing this even though it will not generate the correct resolution for Python 3.8 until #6268 merges.)

[dimbleby]

hesitant to dig in until those merge (#6882, #6268)

I believe both are merged in uv 0.4.4, but still uv does not find a solution for "magika==0.5.1"

$ uv add "magika==0.5.1; python_version < '3.13'"
  × No solution found when resolving dependencies for split (python_full_version >= '3.9' and python_full_version < '3.13'):
  ╰─▶ Because only the following versions of numpy{python_full_version >= '3.9' and python_full_version < '3.13'} are available:
          numpy{python_full_version >= '3.9' and python_full_version < '3.13'}<=1.26.0
          numpy{python_full_version >= '3.9' and python_full_version < '3.13'}==1.26.1
          numpy{python_full_version >= '3.9' and python_full_version < '3.13'}==1.26.2
          numpy{python_full_version >= '3.9' and python_full_version < '3.13'}==1.26.3
          numpy{python_full_version >= '3.9' and python_full_version < '3.13'}==1.26.4
          numpy{python_full_version >= '3.9' and python_full_version < '3.13'}>2.0
      and the requested Python version (>=3.8) does not satisfy Python>=3.9, we can conclude that all of:
          numpy{python_full_version >= '3.9' and python_full_version < '3.13'}>=1.26.0,<1.26.2
          numpy{python_full_version >= '3.9' and python_full_version < '3.13'}>1.26.2,<1.26.3
          numpy{python_full_version >= '3.9' and python_full_version < '3.13'}>1.26.3,<1.26.4
          numpy{python_full_version >= '3.9' and python_full_version < '3.13'}>1.26.4,<2.0
       are incompatible.
      And because the requested Python version (>=3.8) does not satisfy Python>=3.9, we can conclude that numpy{python_full_version >= '3.9' and python_full_version < '3.13'}>=1.26.0,<2.0 is incompatible.
      And because magika{python_full_version < '3.13'}==0.5.1 depends on numpy{python_full_version >= '3.9' and python_full_version < '3.13'}>=1.26,<2.0 and your project depends on magika{python_full_version < '3.13'}==0.5.1, we can
      conclude that your project's requirements are unsatisfiable.
  help: If this is intentional, run `uv add --frozen` to skip the lock and sync steps.

[charliermarsh]

Thank you for following up, I appreciate it.

[charliermarsh]

Thanks, I see the bug here.