latest pip (23.3) causes dependency oddity

[jap]

When running pip-compile + pip 23.3 with a requirements.in containing pytest-cov, it shows a weird dependency list with coverage[toml] being installed because of a dependency on coverage. Note that actual package versions are completely fine, this seems to be purely cosmetic.

Things seem better with the previous pip 23.2.1.

Not sure what causes this -- it could pip, pip-tools, pytest-cov or coverage, but pip-tools has the best community vibes so I'm trying here first ;)

Environment Versions

MacOS 13.6, python 3.11.6 managed through pyenv, fresh virtualenv with latest pip and pip-tools:

$ pip freeze --all
build==1.0.3
click==8.1.7
packaging==23.2
pip==23.3
pip-tools==7.3.0
pyproject_hooks==1.0.0
setuptools==65.5.0
wheel==0.41.2

Steps to replicate

$ cat requirements.in
pytest-cov
$ pip-compile requirements.in

Expected result

$ pip-compile requirements.in
WARNING: --strip-extras is becoming the default in version 8.0.0. To silence this warning, either use --strip-extras to opt into the new default or use --no-strip-extras to retain the existing behavior.
#
# This file is autogenerated by pip-compile with Python 3.11
# by the following command:
#
#    pip-compile requirements.in
#
coverage[toml]==7.3.2
    # via pytest-cov
iniconfig==2.0.0
    # via pytest
packaging==23.2
    # via pytest
pluggy==1.3.0
    # via pytest
pytest==7.4.2
    # via pytest-cov
pytest-cov==4.1.0
    # via -r requirements.in

Actual result

$ pip-compile requirements.in
WARNING: --strip-extras is becoming the default in version 8.0.0. To silence this warning, either use --strip-extras to opt into the new default or use --no-strip-extras to retain the existing behavior.
#
# This file is autogenerated by pip-compile with Python 3.11
# by the following command:
#
#    pip-compile requirements.in
#
coverage[toml]==7.3.2
    # via
    #   coverage
    #   pytest-cov
iniconfig==2.0.0
    # via pytest
packaging==23.2
    # via pytest
pluggy==1.3.0
    # via pytest
pytest==7.4.2
    # via pytest-cov
pytest-cov==4.1.0
    # via -r requirements.in

[webknjaz]

Hi @jap!

[toml] is called an extra and is a way to provide optional dependencies of distribution packages.
pytest-cov == 4.1.0 specifies a dependency on coverage[toml] >= 5.2.1 @ https://github.com/pytest-dev/pytest-cov/blob/v4.1.0/setup.py#L133. The toml extra itself is declared in coverage[toml] == 7.3.2 @ https://github.com/nedbat/coveragepy/blob/7.3.2/setup.py#L106 and brings a conditional dependency tomli under older CPython versions under v3.11.0a6 and older. Since you're running pip-tools under Python v3.11.6 and 3.11.6 > 3.11.0a6, the tomli dependency is not pulled into the tree by coverage.
To summarize, coverage[toml] is indeed met in your dependency tree, but it doesn't contribute any actual dependency to the resulting lockfile.

What may feel weird to you is probably a representation issue in the resulting constraint files. You can explicitly use --strip-extras on CLI or strip-extras = true in .pip-tools.toml to have the extras stripped off, which is going to become a default at some point per #1613.

[AndydeCleyre]

@webknjaz I think the report is not about the inclusion of [toml], but the fact that the annotation now adds that coverage[toml] is installed via coverage, where it didn't before.

This can be seen in current main when testing with the latest pip release, where, at least locally, test_combine_different_extras_of_the_same_package is failing because now fake-ray[default,tune] is annotated as via fake-ray (other via items are still there, I'm paraphrasing to highlight the new change).

[AndydeCleyre]

Note: this only happens with the backtracking resolver.

[jap]

@webknjaz I think the report is not about the inclusion of [toml], but the fact that the annotation now adds that coverage[toml] is installed via coverage, where it didn't before.

Exactly, that is the only difference between the actual and expected result.

[webknjaz]

Oh, should we close this in favor of #2004, then?

[AndydeCleyre]

That seems like a different issue to me. That one is about the non-comment requirement format, and this one is about dependency annotations.

[AndydeCleyre]

Another phrasing that may or may not help:

For #2004, extras need to be canonicalized/combined (and possibly restored, though I hope not), for proper ireq line format.

Here in #2003, we should probably ensure an ireq with extras doesn't depend on itself in extra-less form, at least according to annotations.

[chrysle]

Looking into this.

[AndydeCleyre]

I found this a little surprising:

$ <<<'pytest-cov' pip-compile --annotation-style line - -o - 2>/dev/null | grep coverage
coverage[toml]==7.3.2     # via coverage, pytest-cov

$ <<<'coverage[toml]' pip-compile --annotation-style line - -o - 2>/dev/null | grep coverage
coverage[toml]==7.3.2     # via -r -

$ <<<'pytest-cov'$'\n''coverage[toml]' pip-compile --annotation-style line - -o - 2>/dev/null | grep coverage
coverage[toml]==7.3.2     # via -r -, pytest-cov

[AndydeCleyre]

I did a git bisect with pip and see the change happened in 5f8f40eb1d0610e530d5e035ba8c7f99d9af9df1, message: "refinements" -- so maybe it's unintentional and should be addressed in pip, but then again it's part of pypa/pip#12095, and from the description there it may indeed be intentional.

[AndydeCleyre]

@chrysle @webknjaz

Don't miss @sanderr's helpful comment here

[chrysle]

Thanks, I can confirm the patch is working! I'm still crafting a test.