Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Failed to resolve compatible distributions when building Pex from .whl with local dependencies #2038

Closed
rajaie-sg opened this issue Jan 17, 2023 · 5 comments · Fixed by #2041
Assignees
Labels

Comments

@rajaie-sg
Copy link

rajaie-sg commented Jan 17, 2023

I have a pyproject.toml file as below at ~/code/mylibrary/:

[tool.poetry]
name = "mylibrary"
version = "0.1.0"

[tool.poetry.dependencies]
python = ">=3.8,<4"
anotherlibrary = {path = "../anotherlibrary", develop=true}
[tool.poetry.group.dev.dependencies]

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

I build a .whl for mylibrary and here's what its mylibrary-0.1.0.dist-info/METADATA file contains:

Metadata-Version: 2.1
Name: mylibrary
Version: 0.1.0
Requires-Python: >=3.8,<4
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Requires-Dist: anotherlibrary @ file:///Users/rajaiealkorani/code/anotherlibrary
Description-Content-Type: text/markdown

I am trying to build a Pex file by passing that .whl as a dependency, but I am getting the error below:

pex --resolver-version pip-2020-resolver dist/mylibrary-0.1.0-py3-none-any.whl -o /tmp/testing.pex 
Failed to resolve compatible distributions:
1: mylibrary==0.1.0 requires anotherlibrary@ file:///Users/rajaiealkorani/code/anotherlibrary but no version was resolved

Is this a bug in Pex, or is this kind of dependency specification not supported?

I'm able to run pip install mylibrary-0.1.0-py3-none-any.whl in a clean virtual environment as long as file:///Users/rajaiealkorani/code/anotherlibrary exists

I am using Pex 2.1.120

@jsirois
Copy link
Member

jsirois commented Jan 18, 2023

@rajaie-sg can you provide a link to your repository or is it closed source? That would be the easiest path to a repro here.

@jsirois jsirois added the bug label Jan 18, 2023
@jsirois jsirois self-assigned this Jan 18, 2023
@jsirois
Copy link
Member

jsirois commented Jan 18, 2023

Ok, I built a repro. This is a bug.

@jsirois
Copy link
Member

jsirois commented Jan 18, 2023

@rajaie-sg I have a fix over in #2041 and the test added there is instructive. If you have anything more complex than the toy case you outlined (I.E.: If you have many Poetry projects with complex or at least >1 level deep interdependencies), you'll need to add --pip-version 22.3 to your Pex invocations to get it using modern Pip, which is needed to properly resolve relative path deps.

jsirois added a commit that referenced this issue Jan 18, 2023
This has been a very long standing bug. The root issue is that
`pip download`, although it processes these deps and fails if not
present on the local filesystem, only handles resolving recursive
dependencies and it does not "download" the local distribution from its
referenced path on the file system to the `pip download` download dir.
As such, the normal Pex post-processing is foiled for these deps.

The fix provided here is in common infra used by both normal Pex builds
and `--lock` builds and uses a recursive post-processing step that
gathers all such dependencies from the downloaded distributions and
injects them into the normal Pex post-processing chain.

Fixes #2038
@rajaie-sg
Copy link
Author

@jsirois - I confirmed that upgrading to 2.1.121 fixed the toy case I shared in my original post.

We do have Poetry projects with multi-level local dependencies, where ABC.whl depends on the library at local path XYZ, which depends on the library at local path FOO

When I run pex --resolver-version pip-2020-resolver -o testing.pex dist/mywheel-0.1.0-py3-none-any.whl, I get the same error in my initial post. Re-running with --pip-version 22.3 causes the command to hang permanently.

Here's the log when I add -vvvvvvvv to the pex command:

pex: Building pex :: Resolving distributions (dist/mywheel-0.1.0-py3-none-any.whl) :: Resolving requirements. :: Resolving for:
pex: Spawning a maximum of 8 parallel jobs to process:
  /usr/bin/python3.8                                                                          
pex: Hashing pex: 41.0ms
pex: Isolating pex: 0.1ms
pex: Executing: TERM=xterm PATH=/home/hadoop/.local/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin LIVY_HOME=/usr/lib/livy PWD=/app/mywheel JAVA_HOME=/etc/alternatives/jre SHLVL=1 HOME=/root _=/usr/local/bin/pex OLDPWD=/app/mywheel/dist LC_CTYPE=C.UTF-8 TMPDIR=/root/.pex/pip_cache/.tmp PEX_ROOT=/root/.pex PEX_VERBOSE=9 __PEX_UNVENDORED__=1 /root/.pex/venvs/fba4e7b60e4f5d185993509a81648df993b86a67/ddab8011daaee380698ac2fb9701af18c90c03f6/bin/python -sE /root/.pex/venvs/fba4e7b60e4f5d185993509a81648df993b86a67/ddab8011daaee380698ac2fb9701af18c90c03f6/pex --disable-pip-version-check --no-python-version-warning --exists-action a --no-input --isolated -vvv --cache-dir /root/.pex/pip_cache --log /tmp/pex-pip-log.jz16iscl/pip.log download --dest /root/.pex/downloads/resolver_download.eoy2myyz/usr.bin.python3.8 dist/mywheel-0.1.0-py3-none-any.whl --index-url https://pypi.org/simple --retries 5 --timeout 15

When I Ctrl+C out of the process, I get this trace:

^CTraceback (most recent call last):
  File "/usr/local/bin/pex", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.8/site-packages/pex/bin/pex.py", line 819, in main
    catch(
  File "/usr/local/lib/python3.8/site-packages/pex/result.py", line 105, in catch
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/pex/bin/pex.py", line 844, in do_main
    pex_builder = build_pex(
  File "/usr/local/lib/python3.8/site-packages/pex/bin/pex.py", line 718, in build_pex
    result = resolve(
  File "/usr/local/lib/python3.8/site-packages/pex/resolver.py", line 1024, in resolve
    build_requests, download_results = _download_internal(
  File "/usr/local/lib/python3.8/site-packages/pex/resolver.py", line 1117, in _download_internal
    download_results = download_request.download_distributions(
  File "/usr/local/lib/python3.8/site-packages/pex/resolver.py", line 111, in download_distributions
    return list(
  File "/usr/local/lib/python3.8/site-packages/pex/jobs.py", line 541, in execute_parallel
    yield spawn_result.spawned_job.await_result()
  File "/usr/local/lib/python3.8/site-packages/pex/jobs.py", line 220, in await_result
    job.wait()
  File "/usr/local/lib/python3.8/site-packages/pex/jobs.py", line 78, in wait
    _, stderr = self._process.communicate()
  File "/usr/lib64/python3.8/subprocess.py", line 1018, in communicate
    stderr = self.stderr.read()
KeyboardInterrupt

I am on pex 2.1.121 and built mywheel using Poetry (version 1.3.2)

This is how the METADATA for mywheel looks like (cleaned up a little)

bash-4.2# cat mywheel-0.1.0.dist-info/METADATA 
Metadata-Version: 2.1
Name: mywheel
Version: 0.1.0
Summary: mywheel python whl
Author: mywheel
Requires-Python: >=3.8,<4
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Requires-Dist: bidict (>=0.22.0)
Requires-Dist: boto3 (>=1.17.14)
Requires-Dist: xyz @ file:///app/xyz
Requires-Dist: foo @ file:///app/foo
Description-Content-Type: text/markdown

The pyproject.toml for foo only has dependencies from PyPI.

The dependencies section for pyproject.toml for xyz looks something like this:

[tool.poetry.dependencies]
pyspark = "==3.1.1"
python = ">=3.8,<4"
pathlib2 = ">=2.3.5"
foo = {path = "../foo", develop=true}

FYI I am running the pex command as root (to rule out any permission issues)

@rajaie-sg
Copy link
Author

You can ignore the comment above, I think I was doing something wrong on my end. I was able to get it working with the --pip-version flag. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants