Possible regression in 20.2b1: AssertionError: a direct req shouldn't have a parent...

[JulienPalard]

Environment

• pip version: 20.2b1
• Python version: 3.8.4
• OS: Debian testing

Description

Given those two files:

$ cat setup.cfg
[metadata]
name = test
version = 0.0.1
description = test

[options]
install_requires =
    oeis

$ cat setup.py
from setuptools import setup

setup()

when I run pip-compile setup.py (from pip-tools), I get:

Traceback (most recent call last):
  File "/tmp/pto/.venv/bin/pip-compile", line 8, in <module>
    sys.exit(cli())
  File "/tmp/pto/.venv/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/tmp/pto/.venv/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/tmp/pto/.venv/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/tmp/pto/.venv/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/tmp/pto/.venv/lib/python3.8/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/tmp/pto/.venv/lib/python3.8/site-packages/piptools/scripts/compile.py", line 458, in cli
    results = resolver.resolve(max_rounds=max_rounds)
  File "/tmp/pto/.venv/lib/python3.8/site-packages/piptools/resolver.py", line 169, in resolve
    has_changed, best_matches = self._resolve_one_round()
  File "/tmp/pto/.venv/lib/python3.8/site-packages/piptools/resolver.py", line 274, in _resolve_one_round
    their_constraints.extend(self._iter_dependencies(best_match))
  File "/tmp/pto/.venv/lib/python3.8/site-packages/piptools/resolver.py", line 384, in _iter_dependencies
    dependencies = self.repository.get_dependencies(ireq)
  File "/tmp/pto/.venv/lib/python3.8/site-packages/piptools/repositories/pypi.py", line 231, in get_dependencies
    self._dependencies_cache[ireq] = self.resolve_reqs(
  File "/tmp/pto/.venv/lib/python3.8/site-packages/piptools/repositories/pypi.py", line 171, in resolve_reqs
    reqset.add_requirement(ireq)
  File "/tmp/pto/.venv/lib/python3.8/site-packages/pip/_internal/req/req_set.py", line 113, in add_requirement
    assert install_req.is_direct == (parent_req_name is None), (
AssertionError: a direct req shouldn't have a parent and also, a non direct req should have a parent

But I don't get this with pip 20.1.1.

I did not tried to reproduce with pip only yet, so my reproducer is still bound to pip-tools, which is not ideal, I know.

[McSinyx]

Hello there, is this still reproducible with pip-tools 5.3.0, which was released yesterday and included jazzband/pip-tools#1168?

[AndydeCleyre]

FWIW with the above example files, and the current release (5.3.0, pip 20.1.1, python 3.8.4), tested on linux, it works fine:

#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile setup.py
#
cycler==0.10.0            # via matplotlib
kiwisolver==1.2.0         # via matplotlib
matplotlib==3.3.0         # via oeis
mpmath==1.1.0             # via sympy
numpy==1.19.1             # via matplotlib
oeis==2020.1.20           # via test (setup.py)
pillow==7.2.0             # via matplotlib
pyparsing==2.4.7          # via matplotlib
python-dateutil==2.8.1    # via matplotlib
six==1.15.0               # via cycler, python-dateutil
sympy==1.6.1              # via oeis

[JulienPalard]

I tested again, in the exact same environment and can't reproduce it. It's funny because I still have the terminal where I made it fail:

The initial failure, leading me to open this issue, that I still had in a terminal history.

mdk@seraph:~
$ cd /tmp
mdk@seraph:/tmp
$ mkdir pto
mdk@seraph:/tmp
$ cd pto
mdk@seraph:/tmp/pto
$ cp ~/clones/JulienPalard/oeis/setup.py .
mdk@seraph:/tmp/pto
$ cp ~/clones/JulienPalard/oeis/setup.cfg .
mdk@seraph:/tmp/pto
$ emacs setup.*  # Simplifying the files to the minimum
mdk@seraph:/tmp/pto
$ venv  # a bash function of mine to create a venv with pip (in beta  because you called for testers :p) and some other usefull packages
Collecting black
  Using cached black-19.10b0-py36-none-any.whl (97 kB)
Collecting jedi
  Using cached jedi-0.17.2-py2.py3-none-any.whl (1.4 MB)
Collecting wheel
  Using cached wheel-0.34.2-py2.py3-none-any.whl (26 kB)
Collecting pip
  Using cached pip-20.2b1-py2.py3-none-any.whl (1.5 MB)
Collecting appdirs
  Using cached appdirs-1.4.4-py2.py3-none-any.whl (9.6 kB)
Collecting pathspec<1,>=0.6
  Using cached pathspec-0.8.0-py2.py3-none-any.whl (28 kB)
Collecting click>=6.5
  Using cached click-7.1.2-py2.py3-none-any.whl (82 kB)
Collecting regex
  Using cached regex-2020.7.14-cp38-cp38-manylinux2010_x86_64.whl (672 kB)
Collecting toml>=0.9.4
  Using cached toml-0.10.1-py2.py3-none-any.whl (19 kB)
Collecting attrs>=18.1.0
  Using cached attrs-19.3.0-py2.py3-none-any.whl (39 kB)
Collecting typed-ast>=1.4.0
  Using cached typed_ast-1.4.1-cp38-cp38-manylinux1_x86_64.whl (768 kB)
Collecting parso<0.8.0,>=0.7.0
  Using cached parso-0.7.1-py2.py3-none-any.whl (109 kB)
Installing collected packages: appdirs, pathspec, click, regex, toml, attrs, typed-ast, black, parso, jedi, wheel, pip
  Attempting uninstall: pip
    Found existing installation: pip 20.1.1
    Uninstalling pip-20.1.1:
      Successfully uninstalled pip-20.1.1
Successfully installed appdirs-1.4.4 attrs-19.3.0 black-19.10b0 click-7.1.2 jedi-0.17.2 parso-0.7.1 pathspec-0.8.0 pip-20.2b1 regex-2020.7.14 toml-0.10.1 typed-ast-1.4.1 wheel-0.34.2
(pto) mdk@seraph:/tmp/pto
$ pip install pip-tools
Collecting pip-tools
  Using cached pip_tools-5.3.0-py2.py3-none-any.whl (45 kB)
Collecting six
  Using cached six-1.15.0-py2.py3-none-any.whl (10 kB)
Requirement already satisfied: pip>=20.0 in ./.venv/lib/python3.8/site-packages (from pip-tools) (20.2b1)
Requirement already satisfied: click>=7 in ./.venv/lib/python3.8/site-packages (from pip-tools) (7.1.2)
Installing collected packages: six, pip-tools
Successfully installed pip-tools-5.3.0 six-1.15.0
(pto) mdk@seraph:/tmp/pto
$ pip-compile setup.py
Traceback (most recent call last):
  File "/tmp/pto/.venv/bin/pip-compile", line 8, in <module>
    sys.exit(cli())
  File "/tmp/pto/.venv/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/tmp/pto/.venv/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/tmp/pto/.venv/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/tmp/pto/.venv/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/tmp/pto/.venv/lib/python3.8/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/tmp/pto/.venv/lib/python3.8/site-packages/piptools/scripts/compile.py", line 458, in cli
    results = resolver.resolve(max_rounds=max_rounds)
  File "/tmp/pto/.venv/lib/python3.8/site-packages/piptools/resolver.py", line 169, in resolve
    has_changed, best_matches = self._resolve_one_round()
  File "/tmp/pto/.venv/lib/python3.8/site-packages/piptools/resolver.py", line 274, in _resolve_one_round
    their_constraints.extend(self._iter_dependencies(best_match))
  File "/tmp/pto/.venv/lib/python3.8/site-packages/piptools/resolver.py", line 384, in _iter_dependencies
    dependencies = self.repository.get_dependencies(ireq)
  File "/tmp/pto/.venv/lib/python3.8/site-packages/piptools/repositories/pypi.py", line 231, in get_dependencies
    self._dependencies_cache[ireq] = self.resolve_reqs(
  File "/tmp/pto/.venv/lib/python3.8/site-packages/piptools/repositories/pypi.py", line 171, in resolve_reqs
    reqset.add_requirement(ireq)
  File "/tmp/pto/.venv/lib/python3.8/site-packages/pip/_internal/req/req_set.py", line 113, in add_requirement
    assert install_req.is_direct == (parent_req_name is None), (
AssertionError: a direct req shouldn't have a parent and also, a non direct req should have a parent
/home/mdk/.local/lib/python3.8/tempfile.py:819: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpauzbybpfsource'>
  _warnings.warn(warn_message, ResourceWarning)
/home/mdk/.local/lib/python3.8/tempfile.py:819: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpa6a84oijbuild'>
  _warnings.warn(warn_message, ResourceWarning)
sys:1: ResourceWarning: unclosed <ssl.SSLSocket fd=4, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('10.0.0.13', 58334), raddr=('151.101.192.223', 443)>

Having this in my terminal history is great because I have the exact versions of each packets so I can try an exact reproducer:

Me trying to reproduce it again, failing.

$ python3.8 -m venv .venv
mdk@seraph:/tmp/pt1
$ source .venv/bin/activate
(.venv) mdk@seraph:/tmp/pt1
$ mdk@seraph:~
$ pip install appdirs==1.4.4 attrs==19.3.0 black==19.10b0 click==7.1.2 jedi==0.17.2 parso==0.7.1 pathspec==0.8.0 pip==20.2b1 regex==2020.7.14 toml==0.10.1 typed-ast==1.4.1 wheel==0.34.2 pip-tools==5.3.0 six==1.15.0
Collecting appdirs==1.4.4
  Using cached appdirs-1.4.4-py2.py3-none-any.whl (9.6 kB)
Collecting attrs==19.3.0
  Using cached attrs-19.3.0-py2.py3-none-any.whl (39 kB)
Collecting black==19.10b0
  Using cached black-19.10b0-py36-none-any.whl (97 kB)
Collecting click==7.1.2
  Using cached click-7.1.2-py2.py3-none-any.whl (82 kB)
Collecting jedi==0.17.2
  Using cached jedi-0.17.2-py2.py3-none-any.whl (1.4 MB)
Collecting parso==0.7.1
  Using cached parso-0.7.1-py2.py3-none-any.whl (109 kB)
Collecting pathspec==0.8.0
  Using cached pathspec-0.8.0-py2.py3-none-any.whl (28 kB)
Collecting pip==20.2b1
  Using cached pip-20.2b1-py2.py3-none-any.whl (1.5 MB)
Collecting regex==2020.7.14
  Using cached regex-2020.7.14-cp38-cp38-manylinux2010_x86_64.whl (672 kB)
Collecting toml==0.10.1
  Using cached toml-0.10.1-py2.py3-none-any.whl (19 kB)
Collecting typed-ast==1.4.1
  Using cached typed_ast-1.4.1-cp38-cp38-manylinux1_x86_64.whl (768 kB)
Collecting wheel==0.34.2
  Using cached wheel-0.34.2-py2.py3-none-any.whl (26 kB)
Collecting pip-tools==5.3.0
  Using cached pip_tools-5.3.0-py2.py3-none-any.whl (45 kB)
Collecting six==1.15.0
  Using cached six-1.15.0-py2.py3-none-any.whl (10 kB)
Installing collected packages: appdirs, attrs, toml, click, pathspec, regex, typed-ast, black, parso, jedi, pip, wheel, six, pip-tools
  Attempting uninstall: pip
    Found existing installation: pip 20.1.1
    Uninstalling pip-20.1.1:
      Successfully uninstalled pip-20.1.1
Successfully installed appdirs-1.4.4 attrs-19.3.0 black-19.10b0 click-7.1.2 jedi-0.17.2 parso-0.7.1 pathspec-0.8.0 pip-20.2b1 pip-tools-5.3.0 regex-2020.7.14 six-1.15.0 toml-0.10.1 typed-ast-1.4.1 wheel-0.34.2
(.venv) mdk@seraph:/tmp/pt1
$ diff /tmp/pto/setup.py setup.py 
(.venv) mdk@seraph:/tmp/pt1
$ diff /tmp/pto/setup.cfg setup.cfg
(.venv) mdk@seraph:/tmp/pt1
$ pip-compile setup.py 
/home/mdk/.local/lib/python3.8/tempfile.py:819: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmp5zdzfdvnbuild'>
  _warnings.warn(warn_message, ResourceWarning)
/home/mdk/.local/lib/python3.8/tempfile.py:819: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpwvwuk76ysource'>
  _warnings.warn(warn_message, ResourceWarning)
/home/mdk/.local/lib/python3.8/tempfile.py:819: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpqutjlnv4build'>
  _warnings.warn(warn_message, ResourceWarning)
/home/mdk/.local/lib/python3.8/tempfile.py:819: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpb_0sfl4_source'>
  _warnings.warn(warn_message, ResourceWarning)
/home/mdk/.local/lib/python3.8/tempfile.py:819: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpl9lf7s5ubuild'>
  _warnings.warn(warn_message, ResourceWarning)
/home/mdk/.local/lib/python3.8/tempfile.py:819: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpnmtxnsj_source'>
  _warnings.warn(warn_message, ResourceWarning)
#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile setup.py
#
cycler==0.10.0            # via matplotlib
kiwisolver==1.2.0         # via matplotlib
matplotlib==3.3.0         # via oeis
mpmath==1.1.0             # via sympy
numpy==1.19.1             # via matplotlib
oeis==2020.1.20           # via test (setup.py)
pillow==7.2.0             # via matplotlib
pyparsing==2.4.7          # via matplotlib
python-dateutil==2.8.1    # via matplotlib
six==1.15.0               # via cycler, python-dateutil
sympy==1.6.1              # via oeis
/home/mdk/.local/lib/python3.8/tempfile.py:819: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmp8uxcysx7build'>
  _warnings.warn(warn_message, ResourceWarning)
/home/mdk/.local/lib/python3.8/tempfile.py:819: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmp046fe90xsource'>
  _warnings.warn(warn_message, ResourceWarning)
/home/mdk/.local/lib/python3.8/tempfile.py:819: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpz8b921rysource'>
  _warnings.warn(warn_message, ResourceWarning)
/home/mdk/.local/lib/python3.8/tempfile.py:819: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmphsa83je_build'>
  _warnings.warn(warn_message, ResourceWarning)

OK so this is harder to reproduce than initially though, I don't understand what's happening here, either a cache issue, or a Schrödinger bug?

[pfmoore]

pip-tools calls pip internals directly, which we don't support (and pip-tools are fine with that, as far as I know). So this should be tracked via the pip-tools issue, not here. In particular, pip-tools is modifying is_direct/user_supplied on an ireq, which seems very brave of them if you ask me 🙂

If it's possible to reproduce with just pip, then we can take a look.

[JulienPalard]

OK to track it pip-tools side :)

[atugushev]

FWIW it's fixed in #8026.