Failed to resolve compatible distributions when building Pex from .whl with local dependencies

[rajaie-sg]

I have a pyproject.toml file as below at ~/code/mylibrary/:

[tool.poetry]
name = "mylibrary"
version = "0.1.0"

[tool.poetry.dependencies]
python = ">=3.8,<4"
anotherlibrary = {path = "../anotherlibrary", develop=true}
[tool.poetry.group.dev.dependencies]

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

I build a .whl for mylibrary and here's what its mylibrary-0.1.0.dist-info/METADATA file contains:

Metadata-Version: 2.1
Name: mylibrary
Version: 0.1.0
Requires-Python: >=3.8,<4
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Requires-Dist: anotherlibrary @ file:///Users/rajaiealkorani/code/anotherlibrary
Description-Content-Type: text/markdown

I am trying to build a Pex file by passing that .whl as a dependency, but I am getting the error below:

pex --resolver-version pip-2020-resolver dist/mylibrary-0.1.0-py3-none-any.whl -o /tmp/testing.pex 
Failed to resolve compatible distributions:
1: mylibrary==0.1.0 requires anotherlibrary@ file:///Users/rajaiealkorani/code/anotherlibrary but no version was resolved

Is this a bug in Pex, or is this kind of dependency specification not supported?

I'm able to run pip install mylibrary-0.1.0-py3-none-any.whl in a clean virtual environment as long as file:///Users/rajaiealkorani/code/anotherlibrary exists

I am using Pex 2.1.120

[jsirois]

@rajaie-sg can you provide a link to your repository or is it closed source? That would be the easiest path to a repro here.

[jsirois]

Ok, I built a repro. This is a bug.

[jsirois]

@rajaie-sg I have a fix over in #2041 and the test added there is instructive. If you have anything more complex than the toy case you outlined (I.E.: If you have many Poetry projects with complex or at least >1 level deep interdependencies), you'll need to add --pip-version 22.3 to your Pex invocations to get it using modern Pip, which is needed to properly resolve relative path deps.

[rajaie-sg]

@jsirois - I confirmed that upgrading to 2.1.121 fixed the toy case I shared in my original post.

We do have Poetry projects with multi-level local dependencies, where ABC.whl depends on the library at local path XYZ, which depends on the library at local path FOO

When I run pex --resolver-version pip-2020-resolver -o testing.pex dist/mywheel-0.1.0-py3-none-any.whl, I get the same error in my initial post. Re-running with --pip-version 22.3 causes the command to hang permanently.

Here's the log when I add -vvvvvvvv to the pex command:

pex: Building pex :: Resolving distributions (dist/mywheel-0.1.0-py3-none-any.whl) :: Resolving requirements. :: Resolving for:
pex: Spawning a maximum of 8 parallel jobs to process:
  /usr/bin/python3.8                                                                          
pex: Hashing pex: 41.0ms
pex: Isolating pex: 0.1ms
pex: Executing: TERM=xterm PATH=/home/hadoop/.local/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin LIVY_HOME=/usr/lib/livy PWD=/app/mywheel JAVA_HOME=/etc/alternatives/jre SHLVL=1 HOME=/root _=/usr/local/bin/pex OLDPWD=/app/mywheel/dist LC_CTYPE=C.UTF-8 TMPDIR=/root/.pex/pip_cache/.tmp PEX_ROOT=/root/.pex PEX_VERBOSE=9 __PEX_UNVENDORED__=1 /root/.pex/venvs/fba4e7b60e4f5d185993509a81648df993b86a67/ddab8011daaee380698ac2fb9701af18c90c03f6/bin/python -sE /root/.pex/venvs/fba4e7b60e4f5d185993509a81648df993b86a67/ddab8011daaee380698ac2fb9701af18c90c03f6/pex --disable-pip-version-check --no-python-version-warning --exists-action a --no-input --isolated -vvv --cache-dir /root/.pex/pip_cache --log /tmp/pex-pip-log.jz16iscl/pip.log download --dest /root/.pex/downloads/resolver_download.eoy2myyz/usr.bin.python3.8 dist/mywheel-0.1.0-py3-none-any.whl --index-url https://pypi.org/simple --retries 5 --timeout 15

When I Ctrl+C out of the process, I get this trace:

^CTraceback (most recent call last):
  File "/usr/local/bin/pex", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.8/site-packages/pex/bin/pex.py", line 819, in main
    catch(
  File "/usr/local/lib/python3.8/site-packages/pex/result.py", line 105, in catch
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/pex/bin/pex.py", line 844, in do_main
    pex_builder = build_pex(
  File "/usr/local/lib/python3.8/site-packages/pex/bin/pex.py", line 718, in build_pex
    result = resolve(
  File "/usr/local/lib/python3.8/site-packages/pex/resolver.py", line 1024, in resolve
    build_requests, download_results = _download_internal(
  File "/usr/local/lib/python3.8/site-packages/pex/resolver.py", line 1117, in _download_internal
    download_results = download_request.download_distributions(
  File "/usr/local/lib/python3.8/site-packages/pex/resolver.py", line 111, in download_distributions
    return list(
  File "/usr/local/lib/python3.8/site-packages/pex/jobs.py", line 541, in execute_parallel
    yield spawn_result.spawned_job.await_result()
  File "/usr/local/lib/python3.8/site-packages/pex/jobs.py", line 220, in await_result
    job.wait()
  File "/usr/local/lib/python3.8/site-packages/pex/jobs.py", line 78, in wait
    _, stderr = self._process.communicate()
  File "/usr/lib64/python3.8/subprocess.py", line 1018, in communicate
    stderr = self.stderr.read()
KeyboardInterrupt

I am on pex 2.1.121 and built mywheel using Poetry (version 1.3.2)

This is how the METADATA for mywheel looks like (cleaned up a little)

bash-4.2# cat mywheel-0.1.0.dist-info/METADATA 
Metadata-Version: 2.1
Name: mywheel
Version: 0.1.0
Summary: mywheel python whl
Author: mywheel
Requires-Python: >=3.8,<4
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Requires-Dist: bidict (>=0.22.0)
Requires-Dist: boto3 (>=1.17.14)
Requires-Dist: xyz @ file:///app/xyz
Requires-Dist: foo @ file:///app/foo
Description-Content-Type: text/markdown

The pyproject.toml for foo only has dependencies from PyPI.

The dependencies section for pyproject.toml for xyz looks something like this:

[tool.poetry.dependencies]
pyspark = "==3.1.1"
python = ">=3.8,<4"
pathlib2 = ">=2.3.5"
foo = {path = "../foo", develop=true}

FYI I am running the pex command as root (to rule out any permission issues)

[rajaie-sg]

You can ignore the comment above, I think I was doing something wrong on my end. I was able to get it working with the --pip-version flag. Thanks!