Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error when package requires itself #95

Closed
jdchn opened this issue Aug 21, 2022 · 6 comments · Fixed by #101
Closed

Error when package requires itself #95

jdchn opened this issue Aug 21, 2022 · 6 comments · Fixed by #101
Labels
bug Something isn't working

Comments

@jdchn
Copy link

jdchn commented Aug 21, 2022

What you were trying to do (and why)

Analyze the dependencies for a requirements.txt file containing jax.

The error can be isolated to the jax dependency etils[epath].

What happened (including command output)

Command output

$ pipgrip -vv --tree jax
INFO: discovering jax
INFO: fact: _root_ is root
INFO: derived: root
INFO: fact: root depends on jax (*)
INFO: selecting _root_ (0.0.0)
INFO: derived: jax (*)
INFO: fact: jax (0.3.16) depends on absl-py (*)
INFO: fact: jax (0.3.16) depends on etils[epath] (*)
INFO: fact: jax (0.3.16) depends on numpy (>=1.20)
INFO: fact: jax (0.3.16) depends on opt-einsum (*)
INFO: fact: jax (0.3.16) depends on scipy (>=1.5)
INFO: fact: jax (0.3.16) depends on typing-extensions (*)
INFO: selecting jax (0.3.16)
INFO: derived: typing-extensions (*)
INFO: derived: scipy (>=1.5)
INFO: derived: opt-einsum (*)
INFO: derived: numpy (>=1.20)
INFO: derived: etils[epath] (*)
INFO: derived: absl-py (*)
INFO: discovering typing-extensions
INFO: discovering scipy>=1.5
INFO: discovering opt-einsum
INFO: discovering numpy>=1.20
INFO: discovering etils[epath]
INFO: discovering absl-py
Traceback (most recent call last):
  File "/home/vagrant/.local/bin/pipgrip", line 8, in <module>
    sys.exit(main())
  File "/usr/lib/python3/dist-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3/dist-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3/dist-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/vagrant/.local/lib/python3.8/site-packages/pipgrip/cli.py", line 434, in main
    solution = solver.solve()
  File "/home/vagrant/.local/lib/python3.8/site-packages/pipgrip/libs/mixology/version_solver.py", line 69, in solve
    if not self._run():
  File "/home/vagrant/.local/lib/python3.8/site-packages/pipgrip/libs/mixology/version_solver.py", line 85, in _run
    next_package = self._choose_package_version()
  File "/home/vagrant/.local/lib/python3.8/site-packages/pipgrip/libs/mixology/version_solver.py", line 363, in _choose_package_version
    for incompatibility in self._source.incompatibilities_for(
  File "/home/vagrant/.local/lib/python3.8/site-packages/pipgrip/libs/mixology/package_source.py", line 112, in incompatibilities_for
    incompatibility = Incompatibility(
  File "/home/vagrant/.local/lib/python3.8/site-packages/pipgrip/libs/mixology/incompatibility.py", line 61, in __init__
    assert by_ref[ref] is not None
AssertionError
$ pipgrip -vvv --tree etils[epath]
DEBUG: environment: {'implementation_name': 'cpython', 'implementation_version': '3.8.10', 'os_name': 'posix', 'platform_machine': 'x86_64', 'platform_release': '5.15.0-41-generic', 'platform_system': 'Linux', 'platform_version': '#44~20.04.1-Ubuntu SMP Fri Jun 24 13:27:29 UTC 2022', 'python_full_version': '3.8.10', 'platform_python_implementation': 'CPython', 'python_version': '3.8', 'sys_platform': 'linux'}
DEBUG: pip version: [20, 0, 2]
DEBUG: pipgrip version: 0.8.5
INFO: discovering etils[epath]
DEBUG: Downloading/building wheel for etils[epath] into cache_dir /home/vagrant/.cache/pip/wheels/pipgrip
DEBUG: ['/usr/bin/python3', '-m', 'pip', 'wheel', '--no-deps', '--disable-pip-version-check', '--wheel-dir', '/home/vagrant/.cache/pip/wheels/pipgrip', '--progress-bar=off', 'etils[epath]']
DEBUG: {'etils[epath]': '/home/vagrant/.cache/pip/wheels/pipgrip/etils-0.7.1-py3-none-any.whl'}
DEBUG: Searching metadata in /home/vagrant/.cache/pip/wheels/pipgrip/etils-0.7.1-py3-none-any.whl
DEBUG: dropped conditional dep etils[array-types] ; extra == "all"
DEBUG: dropped conditional dep etils[ecolab] ; extra == "all"
DEBUG: dropped conditional dep etils[edc] ; extra == "all"
DEBUG: dropped conditional dep etils[enp] ; extra == "all"
DEBUG: dropped conditional dep etils[epath] ; extra == "all"
DEBUG: dropped conditional dep etils[epy] ; extra == "all"
DEBUG: dropped conditional dep etils[etqdm] ; extra == "all"
DEBUG: dropped conditional dep etils[etree] ; extra == "all"
DEBUG: dropped conditional dep etils[etree-dm] ; extra == "all"
DEBUG: dropped conditional dep etils[etree-jax] ; extra == "all"
DEBUG: dropped conditional dep etils[etree-tf] ; extra == "all"
DEBUG: dropped conditional dep etils[enp] ; extra == "array-types"
DEBUG: dropped conditional dep pytest ; extra == "dev"
DEBUG: dropped conditional dep pytest-subtests ; extra == "dev"
DEBUG: dropped conditional dep pytest-xdist ; extra == "dev"
DEBUG: dropped conditional dep pylint>=2.6.0 ; extra == "dev"
DEBUG: dropped conditional dep yapf ; extra == "dev"
DEBUG: dropped conditional dep chex ; extra == "dev"
DEBUG: dropped conditional dep jupyter ; extra == "ecolab"
DEBUG: dropped conditional dep numpy ; extra == "ecolab"
DEBUG: dropped conditional dep mediapy ; extra == "ecolab"
DEBUG: dropped conditional dep etils[enp] ; extra == "ecolab"
DEBUG: dropped conditional dep etils[epy] ; extra == "ecolab"
DEBUG: dropped conditional dep etils[epy] ; extra == "edc"
DEBUG: dropped conditional dep numpy ; extra == "enp"
DEBUG: dropped conditional dep etils[epy] ; extra == "enp"
DEBUG: included conditional dep importlib_resources ; extra == "epath"
DEBUG: included conditional dep typing_extensions ; extra == "epath"
DEBUG: included conditional dep zipp ; extra == "epath"
DEBUG: included conditional dep etils[epy] ; extra == "epath"
DEBUG: dropped conditional dep typing_extensions ; extra == "epy"
DEBUG: dropped conditional dep absl-py ; extra == "etqdm"
DEBUG: dropped conditional dep tqdm ; extra == "etqdm"
DEBUG: dropped conditional dep etils[epy] ; extra == "etqdm"
DEBUG: dropped conditional dep etils[array_types] ; extra == "etree"
DEBUG: dropped conditional dep etils[epy] ; extra == "etree"
DEBUG: dropped conditional dep etils[enp] ; extra == "etree"
DEBUG: dropped conditional dep etils[etqdm] ; extra == "etree"
DEBUG: dropped conditional dep dm-tree ; extra == "etree-dm"
DEBUG: dropped conditional dep etils[etree] ; extra == "etree-dm"
DEBUG: dropped conditional dep jax[cpu] ; extra == "etree-jax"
DEBUG: dropped conditional dep etils[etree] ; extra == "etree-jax"
DEBUG: dropped conditional dep tf-nightly ; extra == "etree-tf"
DEBUG: dropped conditional dep etils[etree] ; extra == "etree-tf"
DEBUG: Finding possible versions for etils[epath]
DEBUG: ['/usr/bin/python3', '-m', 'pip', 'wheel', '--no-deps', '--disable-pip-version-check', '--progress-bar=off', 'etils[epath]==42.42.post424242']
DEBUG: {'etils[epath]': ['0.2.0', '0.3.0', '0.3.1', '0.3.2', '0.3.3', '0.4.0', '0.5.0', '0.5.1', '0.6.0', '0.7.0', '0.7.1']}
INFO: fact: _root_ is root
INFO: derived: root
INFO: fact: root depends on etils[epath] (*)
INFO: selecting _root_ (0.0.0)
INFO: derived: etils[epath] (*)
Traceback (most recent call last):
  File "/home/vagrant/.local/bin/pipgrip", line 8, in <module>
    sys.exit(main())
  File "/usr/lib/python3/dist-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3/dist-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3/dist-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/vagrant/.local/lib/python3.8/site-packages/pipgrip/cli.py", line 434, in main
    solution = solver.solve()
  File "/home/vagrant/.local/lib/python3.8/site-packages/pipgrip/libs/mixology/version_solver.py", line 69, in solve
    if not self._run():
  File "/home/vagrant/.local/lib/python3.8/site-packages/pipgrip/libs/mixology/version_solver.py", line 85, in _run
    next_package = self._choose_package_version()
  File "/home/vagrant/.local/lib/python3.8/site-packages/pipgrip/libs/mixology/version_solver.py", line 363, in _choose_package_version
    for incompatibility in self._source.incompatibilities_for(
  File "/home/vagrant/.local/lib/python3.8/site-packages/pipgrip/libs/mixology/package_source.py", line 112, in incompatibilities_for
    incompatibility = Incompatibility(
  File "/home/vagrant/.local/lib/python3.8/site-packages/pipgrip/libs/mixology/incompatibility.py", line 61, in __init__
    assert by_ref[ref] is not None
AssertionError

What you expected to happen

Expected to see dependency tree for jax.

Step-by-step reproduction instructions

In an Ubuntu 20.04 or RHEL UBI 8 environment (VM or container) with Python 3.8...

  1. Install pipgrip.
  2. Run pipgrip -vv --tree etils[epath].
@jdchn jdchn added the bug Something isn't working label Aug 21, 2022
@ddelange
Copy link
Owner

Hi @jdchn 👋

First of all, thanks for the reproducible bug report!

This is a hard crash in the vendored and adapted mixology library:

# If we have two terms that refer to the same package but have a null
# intersection, they're mutually exclusive, making this incompatibility
# irrelevant, since we already know that mutually exclusive version
# ranges are incompatible. We should never derive an irrelevant
# incompatibility.
assert by_ref[ref] is not None

From a first glance, this one might be triggered because the package requires itself, something I haven't seen in the wild before (or even knew was allowed).

Let me think how to properly handle this cyclic dependency, it might not be a trivial fix.

@ddelange
Copy link
Owner

ddelange commented Aug 22, 2022

It looks like poetry does not support the pattern, whereas pip happily installs it. I strive for pip compliance, so this needs a fix for sure.

pipgrip currently combines multiple extras alphabetically, e.g. here you could expect to encounter somewhere in pipgrip's output: smart_open[azure,http,gcs,s3]. In the example below that should also be the case, although you'd have to select some other etils extras in order to observe this behaviour.

What etils is doing is next level. It will make for a beautiful test case... I would expect to find etils listed as dependency of itself in the tree (even though it's only a "trick" used by their devs to express the needed dependencies depending on scenario), it would be incorrect to expect pipgrip to "flatten" and resolve this into a list of (only) external dependencies.

Not strictly true:

etils[epath] (0.7.1)
|_ numpy (42)
|_ typing_extensions (42)
... 

Expected:

etils[epath] (0.7.1)
|_ etils[enp] (0.7.1)
   |_ numpy (42)
   |_ etils[epy] (0.7.1)
      |_ typing_extensions (42)
... 

without warnings of cyclic dependencies. It only gets cyclic if the same extra is encountered down the tree, e.g. if epy extra would ask for epath extra:

etils[epath] (0.7.1)
|_ etils[enp] (0.7.1)
   |_ numpy (42)
   |_ etils[epy] (0.7.1)
      |_ typing_extensions (42)
      |_ etils[epath] (0.7.1, cyclic)
... 

This gets iffy, because there might additionally be version specifiers for etils added to this scenario, in which case there is potential for a "etils>=0.7 depends on etils==0.6.0, thats a no-no" exit. That leads me to the conclusion that this needs a fix in the vendored package, as full version solving capabilities need to be retained.

@jdchn
Copy link
Author

jdchn commented Aug 22, 2022

Let me think how to properly handle this cyclic dependency, it might not be a trivial fix.

I don't feel confident enough in myself to provide any useful comments :|

It does seem like etils is a playing some "interesting" games with packaging to avoid the hassle of multiple projects — See etils#185.

@jdchn
Copy link
Author

jdchn commented Aug 22, 2022

It may be a little trickier than it seems. It will be tempting to simply treat self-references as normal dependencies, but there should be some caveats:

  • The self-reference without extras and without version should be ignored.
  • The self-reference with extras and without version should use the metadata version.

The above is probably what would be expected, but I don't know what Pip will actually do.


I think there's also another related issue:

  • pipgrip -vv --tree ray[tune] → Successful
  • pipgrip -vv --tree ray[rllib] → Successful
  • pipgrip -vv --tree ray[tune,rllib] → Successful
  • pipgrip -vv --tree ray[tune] ray[rllib] → Fails
Command output

Traceback (most recent call last):
  File "/home/vagrant/.local/bin/pipgrip", line 8, in <module>
    sys.exit(main())
  File "/usr/lib/python3/dist-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3/dist-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3/dist-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/vagrant/.local/lib/python3.8/site-packages/pipgrip/cli.py", line 460, in main
    raise RuntimeError(
RuntimeError: Unexpected partial solution encountered, not all packages have decisions

@ddelange
Copy link
Owner

Thanks for reporting that one too! Separate bug, yet related 🤔

@github-actions
Copy link

Released 0.9.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants