Pex fails to resolve arbitrary equality requirements when candidate package uses unnormalized version number

[navijation]

I'm trying to deal with a project that uses an internal fork of a package with version number 2.49.0a1. Unfortunately, this is not a canonicalized version number. The canonical form is 2.49a1. As a result, Pex fails to deal with the lockfile generated by Pants because it tries to match the canonical form 2.49a1 against the exact specifier ===2.49.0a1.

Although the best solution is probably to avoid adding package versions not in canonical form, it wouldn't hurt if Pex could handle this use case. A couple options:

1. Convert every specifier to canonical form. I'm not sure if this is viable in general beyond exact equality specifiers.
2. Test only the raw version string against the specifier to check for a match.
3. Test both the normalized and raw version strings against the specifier to check for a match.

The latter solution would look like this

diff --git a/pex/resolve/locked_resolve.py b/pex/resolve/locked_resolve.py
index 8064058..669f76d 100644
--- a/pex/resolve/locked_resolve.py
+++ b/pex/resolve/locked_resolve.py
@@ -501,7 +501,6 @@ class LockedResolve(object):
         include_all_matches=False,  # type: bool
     ):
         # type: (...) -> Union[Resolved, Error]
-
         is_local_interpreter = isinstance(target, LocalInterpreter)
         if not use_wheel:
             if not build:
@@ -605,9 +604,11 @@ class LockedResolve(object):
                     # https://peps.python.org/pep-0440/#handling-of-pre-releases during the lock
                     # resolve; so we trust that resolve's conclusion about prereleases and are
                     # permissive here.
-                    if not resolve_request.requirement.specifier.contains(
-                        str(locked_requirement.pin.version), prereleases=True
-                    ):
+                    def specifier_contains(version_str):
+                        # type: (str) -> bool
+                        return resolve_request.requirement.specifier.contains(version_str, prereleases=True)
+                    version = locked_requirement.pin.version
+                    if not specifier_contains(version.normalized) and not specifier_contains(version.raw):
                         version_mismatches.append(
                             "{specifier} ({via})".format(
                                 specifier=resolve_request.requirement.specifier,

[jsirois]

As a result, Pex fails to deal with the lockfile generated by Pants because it tries to match the canonical form 2.49a1 against the specifier 2.49.0a1.

As demonstrated in Slack, Pex can handle 2.49a1 or 2.49.0a1 - it must be Pants that is erroring here.

[jsirois]

Although I don't have access to boto 2.49.0a1, I do have access to pantsbuild.pants 2.14.0a0 which has the same normalization properties. I find Pex handles that completely for the entire lifecycle.

Lock creation works:

$ pex3 lock create --style universal --resolver-version pip-2020-resolver pantsbuild.pants==2.14.0a0 --indent 2 -o lock.json --interpreter-constraint "CPython>=3.7,<3.10" --target-system linux --target-system mac
$ jq .requirements lock.json
[
  "pantsbuild.pants==2.14.0a0"
]
$ jq -r '.locked_resolves[] | .locked_requirements[] | select(.project_name == "pantsbuild-pants") | .version' lock.json
2.14a0

Lock consumption works wether via implicit (resolve the whole lock) or explicit with version normalized or not:

$ pex --lock lock.json -ofull.pex --interpreter-constraint "CPython>=3.7,<3.10"
$ pex --lock lock.json pantsbuild.pants==2.14.0a0 -oexplicit-root-req.pex --interpreter-constraint "CPython>=3.7,<3.10"
$ pex --lock lock.json pantsbuild.pants==2.14a0 -oexplicit-root-req-normalized.pex --interpreter-constraint "CPython>=3.7,<3.10"
$ ls -lrt *.pex
-rwxr-xr-x 1 jsirois jsirois 158380109 Oct 18 11:53 full.pex
-rwxr-xr-x 1 jsirois jsirois 158380109 Oct 18 11:54 explicit-root-req.pex
-rwxr-xr-x 1 jsirois jsirois 158380107 Oct 18 11:55 explicit-root-req-normalized.pex
$ diff -u <(pex-tools full.pex info -i2) <(pex-tools explicit-root-req.pex info -i2)
$ diff -u <(pex-tools explicit-root-req-normalized.pex info -i2) <(pex-tools explicit-root-req.pex info -i2)
--- /dev/fd/63  2022-10-18 12:06:41.181807816 -0700
+++ /dev/fd/62  2022-10-18 12:06:41.181807816 -0700
@@ -52,11 +52,11 @@
   "interpreter_constraints": [
     "CPython<3.10,>=3.7"
   ],
-  "pex_hash": "d2bcc8934d8b8eb3659333598c2878b1ad2ee50a",
+  "pex_hash": "66738a3a3a993b8c5165248e2017c8876d5dff93",
   "pex_path": "",
   "pex_paths": [],
   "requirements": [
-    "pantsbuild.pants==2.14a0"
+    "pantsbuild.pants==2.14.0a0"
   ],
   "strip_pex_env": true,
   "venv": false,
$ ./full.pex -c 'import pants.version; print(pants.version.__file__)'
/home/jsirois/.pex/installed_wheels/7bd29822f3c7d8ef101cf687dd4bee6afed808fb143fd1b15dabb8c428a50e75/pantsbuild.pants-2.14.0a0-cp37-cp37m-manylinux2014_x86_64.whl/pants/version.py
$ ./explicit-root-req.pex -c 'import pants.version; print(pants.version.__file__)'
/home/jsirois/.pex/installed_wheels/7bd29822f3c7d8ef101cf687dd4bee6afed808fb143fd1b15dabb8c428a50e75/pantsbuild.pants-2.14.0a0-cp37-cp37m-manylinux2014_x86_64.whl/pants/version.py
$ ./explicit-root-req-normalized.pex -c 'import pants.version; print(pants.version.__file__)'
/home/jsirois/.pex/installed_wheels/7bd29822f3c7d8ef101cf687dd4bee6afed808fb143fd1b15dabb8c428a50e75/pantsbuild.pants-2.14.0a0-cp37-cp37m-manylinux2014_x86_64.whl/pants/version.py

[jsirois]

@navneethjayendran I found a non-repro here using pantsbuild.pants since it uses a0 releases too. Can you maybe do the same and get a reproducible error that way for us to share? If not - I expect you can't given my experiment above - I think this issue is over in Pants, but that too could use a reproducible case we can both run and observe the same error.

[jsirois]

Ok, talking with @navneethjayendran offline, the key here is arbitrary equality (===). Using that the issue is like so:

$ pex3 lock create --style universal --resolver-version pip-2020-resolver pantsbuild.pants===2.14.0a0 --indent 2 -o lock.json --interpreter-constra
int "CPython>=3.7,<3.10" --target-system linux --target-system mac
$ jq .requirements lock.json
[
  "pantsbuild.pants===2.14.0a0"
]
$ jq -r '.locked_resolves[] | .locked_requirements[] | select(.project_name == "pantsbuild-pants") | .version' lock.json
2.14a0

And then consuming:

$ pex --lock lock.json -ofull.pex --interpreter-constraint "CPython>=3.7,<3.10"
Failed to resolve compatible artifacts from lock lock.json for 3 targets:
1. /home/jsirois/.pyenv/versions/3.7.14/bin/python3.7:
    Failed to resolve all requirements for cp37-cp37m-manylinux_2_35_x86_64 interpreter at /home/jsirois/.pyenv/versions/3.7.14/bin/python3.7 from lock.json:

Configured with:
    build: True
    use_wheel: True

Dependency on pantsbuild-pants not satisfied, 1 incompatible candidate found:
1.) pantsbuild-pants 2.14a0 does not satisfy the following requirements:
    ===2.14.0a0 (via: pantsbuild.pants===2.14.0a0)
2. /home/jsirois/.pyenv/versions/3.8.14/bin/python3.8:
    Failed to resolve all requirements for cp38-cp38-manylinux_2_35_x86_64 interpreter at /home/jsirois/.pyenv/versions/3.8.14/bin/python3.8 from lock.json:

Configured with:
    build: True
    use_wheel: True

Dependency on pantsbuild-pants not satisfied, 1 incompatible candidate found:
1.) pantsbuild-pants 2.14a0 does not satisfy the following requirements:
    ===2.14.0a0 (via: pantsbuild.pants===2.14.0a0)
3. /home/jsirois/.pyenv/versions/3.9.14/bin/python3.9:
    Failed to resolve all requirements for cp39-cp39-manylinux_2_35_x86_64 interpreter at /home/jsirois/.pyenv/versions/3.9.14/bin/python3.9 from lock.json:

Configured with:
    build: True
    use_wheel: True

Dependency on pantsbuild-pants not satisfied, 1 incompatible candidate found:
1.) pantsbuild-pants 2.14a0 does not satisfy the following requirements:
    ===2.14.0a0 (via: pantsbuild.pants===2.14.0a0)

[jsirois]

So the issue here is specifically dealing with ===: https://peps.python.org/pep-0440/#arbitrary-equality

[navijation]

Okay, I updated the issue with clarifying context.

It seems like the right way of dealing with arbitrary equality is to supply the raw version to the specifier. If the specifier really aims to forbids textual mismatches, Pex probably should as well.