Building Pex with requirements.txt that includes local directory + Python version specifier fails

[rajaie-sg]

Assume I have a file, reqs.txt with the contents below:

mylibrary @ file:///Users/rajaiealkorani/code/mylibrary ; python_version >= "3.7" and python_version < "4.0"

Let's say I generated this file running poetry export.

If I run pex --resolver-version pip-2020-resolver -r reqs.txt -o /tmp/testing.pex, then run /tmp/testing.pex, and then import mylibrary in the Python interpreter, I get an error No module named 'mylibrary'

If I try building the Pex file using a reqs.txt as below, then I am able to properly import mylibrary

mylibrary @ file:///Users/rajaiealkorani/code/mylibrary"

It looks like adding Python version specifiers for the local package causes it not to get installed. Is this expected?

I am using Pex 2.1.120

[jsirois]

@rajaie-sg Is there any way you can provide a link to your repository? I can't repro using another Poetry-using repo. My steps:

The poetry-using project with a distribution name of dagger-io and a root package named dagger on that distribution

$ git clone https://github.com/dagger/dagger
$ grep -A28 "tool.poetry]" dagger/sdk/python/pyproject.toml
[tool.poetry]
name = "dagger-io"
version = "0.0.0"
description = "A client package for running Dagger pipelines in Python."
license = "Apache-2.0"
authors = ["Dagger <hello@dagger.io>"]
readme = "README.md"
homepage = "https://dagger.io"
documentation = "https://docs.dagger.io/sdk/python"
repository = "https://github.com/dagger/dagger/tree/main/sdk/python"
classifiers = [
    "Development Status :: 3 - Alpha",
    "Intended Audience :: Developers",
    "Framework :: AnyIO",
    "Framework :: Pytest",
    "Programming Language :: Python",
    "Programming Language :: Python :: 3",
    "Programming Language :: Python :: 3 :: Only",
    "Programming Language :: Python :: 3.10",
    "Programming Language :: Python :: 3.11",
    "License :: OSI Approved :: Apache Software License",
    # FIXME: just waiting on windows tests for this
    # "Operating System :: OS Independent",
    "Typing :: Typed",
]
packages = [
    { include = "dagger", from = "src" },
]

$ ls -1 dagger/sdk/python/src/dagger/
api
cli.py
codegen.py
config.py
connection.py
context.py
engine
exceptions.py
__init__.py
log.py
__main__.py
py.typed
server
session.py
transport

Setup the same Pex scenario as yours:

$ pwd
/home/jsirois/support/pex/issue-2037
$ cat reqs.txt
dagger-io @ file:///home/jsirois/support/pex/issue-2037/dagger/sdk/python; python_version >= "3.7" and python_version < "4.0"
$ pex --resolver-version pip-2020-resolver -r reqs.txt -o testing.pex
$ ./testing.pex
Python 3.10.6 (main, Nov 14 2022, 16:10:14) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> import dagger
>>> print(dagger.__file__)
/home/jsirois/.pex/installed_wheels/f484d70da8f2229255c496a2c36f193557e8fa503757d4250defbd1f57252cc7/dagger_io-0.0.0-py3-none-any.whl/dagger/__init__.py
>>>
now exiting InteractiveConsole...

So that worked fine.
The PEX-INFO:

$ pex-tools ./testing.pex info -i2
{
  "bootstrap_hash": "9ed54e64b9faf41fa3d45c68e63cf8ecfded096c",
  "build_properties": {
    "pex_version": "2.1.120"
  },
  "code_hash": "92b2fed89fd0041bfb5028089dddb2836d6b50c1",
  "distributions": {
    "Pygments-2.14.0-py3-none-any.whl": "73066efa74d4eb1e3e08fa2c826cc02e0a6d4747ccb109ba9f680cbb9c4ebe19",
    "anyio-3.6.2-py3-none-any.whl": "53a34c34cbe93a116a834b73d42831762deba88284d839485219d61a630e3fd4",
    "attrs-22.2.0-py3-none-any.whl": "b7d965a018ed5fbe33529f7ffdd208e203bd38ae6313018bfef8b830e329941c",
    "backoff-2.2.1-py3-none-any.whl": "5c174f49512471d2172cc79b1a753a61015dfca7d5ce0c0398d6807cb81960d8",
    "beartype-0.12.0-py3-none-any.whl": "fccc75121b5f869040b6770bb4cfbcb508f466ac2828fe21c08c88581a44a33d",
    "cattrs-22.2.0-py3-none-any.whl": "5fa669f1f898e6eadd0b6269ba35d4123a4f011de0aaeceac0652feb3bbe523f",
    "certifi-2022.12.7-py3-none-any.whl": "fa660808b834d51826f2d58643aad338e42b6d2bae69df8c8e5a4829dbeabdcf",
    "click-8.1.3-py3-none-any.whl": "78086359bc4a576338dbcaacad4a42784cdd0755b6327b984812fe0913265abf",
    "colorama-0.4.6-py2.py3-none-any.whl": "fd6b1d1a0d109220ed687cd5f6fc6d412195ff651bf4c2d489f43ad1692f9381",
    "commonmark-0.9.1-py2.py3-none-any.whl": "8b1e8f03d3f19f4d61005c0c0002b10c7e2e7ef1e384eae1100de1af0f50fd29",
    "dagger_io-0.0.0-py3-none-any.whl": "daf91252552c5629dd0287019edcbbd7a3c2a76d404a8785f3d66c58b09d414c",
    "exceptiongroup-1.1.0-py3-none-any.whl": "e5b0fb9e8292f20a4acde6ecb2561c7bbf6876fcab0f63488920364b089eab7a",
    "gql-3.4.0-py2.py3-none-any.whl": "0865932b918923c7cd5757e3185d4f3a2dee8d1f27d897596ae14e71a7d27f96",
    "graphql_core-3.2.3-py3-none-any.whl": "975acfdad27f28fe6ddd8e33db8bf9bab7e37504f6c945d4c346f2b9a5568ffe",
    "h11-0.14.0-py3-none-any.whl": "37ba404269b3b437819ec176737a4bd872c1b88cd0543d7b8067b86821221442",
    "httpcore-0.16.3-py3-none-any.whl": "1e3f389af00556ce91d5d52a6b0004005121425d6c9b71bd44cefe9f456f39ad",
    "httpx-0.23.3-py3-none-any.whl": "f64ef9925ce2d486a155f42f4dce5697f7a2d0ebe311166aad7cfd794f858f92",
    "idna-3.4-py3-none-any.whl": "3baea0ffbdea9f349783bacbdb82dbd50eadfec75339388aaf1597754786e3ed",
    "multidict-6.0.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl": "ad5568ce92a1052a77bb75596402940b7edb4e80321a77f63dc448a7ddf54401",
    "platformdirs-2.6.2-py3-none-any.whl": "adfe2a4dd4d028a204cdc6a1e387e4bd9023727e0ec9c9f8b1b88affe2ff8f60",
    "rfc3986-1.5.0-py2.py3-none-any.whl": "38619c25a3ffb7cea10505911075fa58f6393e396d7ad81ba997a7ac4fd0e2dd",
    "rich-12.6.0-py3-none-any.whl": "ee715cddff848a868d2ecc40d4385a81b0ae6529e49a4d983b11291f61ecf27c",
    "shellingham-1.5.0.post1-py2.py3-none-any.whl": "93f1adcb68573b949d805d7d4662a2892467180569976f29d7e135edaf842f81",
    "sniffio-1.3.0-py3-none-any.whl": "feffd920899e99d56efbafd1469bc06ccab4337c224960b4ca43f5c937f56a6a",
    "typer-0.7.0-py3-none-any.whl": "2034b0e305ac03042430d375175c0c865b191edec5326ef46fa039ab2ef32d5b",
    "yarl-1.8.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl": "38499a67e9e8dd1b615e309e423f29e898738ebd234ac680011494e92033e22a"
  },
  "emit_warnings": true,
  "ignore_errors": false,
  "includes_tools": false,
  "inherit_path": "false",
  "inject_args": [],
  "inject_env": {},
  "interpreter_constraints": [],
  "pex_hash": "fd87987df00f543d01807227049405bad6f4f57a",
  "pex_path": "",
  "pex_paths": [],
  "requirements": [
    "dagger-io==0.0.0; python_version >= \"3.7\" and python_version < \"4.0\""
  ],
  "strip_pex_env": true,
  "venv": false,
  "venv_bin_path": "false",
  "venv_copies": false,
  "venv_site_packages_copies": false,
  "pex_root": "/home/jsirois/.pex"
}

[jsirois]

I'm going to tentatively label this as a question since Pex appears to work just fine with file:// URL requirements and markers.

[rajaie-sg]

@jsirois thank you for your reply. It took me a quite a bit of trying to figure it out, and turns out it's the space before the semicolon that's causing the issue.

In your repro attempt, your reqs.txt doesn't contain a space before the semicolon.

cat reqs.txt
dagger-io @ file:///home/jsirois/support/pex/issue-2037/dagger/sdk/python; python_version >= "3.7" and python_version < "4.0"

however, in my file, there is a space before the semicolon. I was using the output of poetry export which seems to have added that space in this PR to fix a related issue. So I wonder if this a bug in Pex?

When I run pip install -r reqs.txt, it works whether there is or isn't a space before the semicolon.

[jsirois]

Ok. Yeah, that's a bug.