Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pipenv lock fails for packages with extras as of 2022.8.13 #5273

Closed
clewingriffith opened this issue Aug 18, 2022 · 33 comments · Fixed by #5274, #5309 or #5321
Closed

Pipenv lock fails for packages with extras as of 2022.8.13 #5273

clewingriffith opened this issue Aug 18, 2022 · 33 comments · Fixed by #5274, #5309 or #5321
Labels
Type: Bug 🐛 This issue is a bug.

Comments

@clewingriffith
Copy link

Issue description

There was a regression in the ability to lock Pipfiles containing packages with extras on the 2022.8.13 release. This was working fine on versions prior to 2022.8.13. I believe this is as a result of the changes introduced in #5234

Expected result

Given the Pipfile in the replicate section, it should lock, showing

Locking [dev-packages] dependencies...
Building requirements...
Resolving dependencies...
✔ Success! 
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
✔ Success! 
Updated Pipfile.lock (3e8e27)!

Actual result

[ResolutionFailure]:   File "***.venv/lib/python3.10/site-packages/pipenv/resolver.py", line 824, in _main
[ResolutionFailure]:       resolve_packages(
[ResolutionFailure]:   File "***.venv/lib/python3.10/site-packages/pipenv/resolver.py", line 772, in resolve_packages
[ResolutionFailure]:       results, resolver = resolve(
[ResolutionFailure]:   File "***.venv/lib/python3.10/site-packages/pipenv/resolver.py", line 751, in resolve
[ResolutionFailure]:       return resolve_deps(
[ResolutionFailure]:   File "***.venv/lib/python3.10/site-packages/pipenv/utils/resolver.py", line 1126, in resolve_deps
[ResolutionFailure]:       results, hashes, markers_lookup, resolver, skipped = actually_resolve_deps(
[ResolutionFailure]:   File "***.venv/lib/python3.10/site-packages/pipenv/utils/resolver.py", line 919, in actually_resolve_deps
[ResolutionFailure]:       resolver.resolve()
[ResolutionFailure]:   File "***.venv/lib/python3.10/site-packages/pipenv/utils/resolver.py", line 723, in resolve
[ResolutionFailure]:       raise ResolutionFailure(message=str(e))
[pipenv.exceptions.ResolutionFailure]: Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  You can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
  Hint: try $ pipenv lock --pre if it is a pre-release dependency.
ERROR: Constraints cannot have extras

Steps to replicate

With the following Pipfile

[packages]
"uvicorn[standard]" = "~=0.18.2"

[dev-packages]
pytest = "~=7.1.1"

[requires]
python_version = "3.9"

Run pipenv lock

Note that there is a package with 'extras', namely uvicorn[standard]. This issue does not occur with normal packages.

The change to lock the packages first and then treat that lock as constraints for the dev packages seems to raise the exception that the extras are not allowed.

@matteius
Copy link
Member

@dqkqd fyi -- do you have some cycles to help me think more about this at it relates to the constraint files work?

@dqkqd
Copy link
Contributor

dqkqd commented Aug 18, 2022

@matteius Yes, there's a bug here.

def is_constraint(dep):
# https://pip.pypa.io/en/stable/user_guide/#constraints-files
# constraints must have a name, they cannot be editable, and they cannot specify extras.
return dep.name and not dep.editable and not dep.extras

I will open a PR for this.

@dqkqd
Copy link
Contributor

dqkqd commented Aug 18, 2022

Function to check constraints from pip has InstallRequirement as parameter:

def check_invalid_constraint_type(req: InstallRequirement) -> str:
# Check for unsupported forms
problem = ""
if not req.name:
problem = "Unnamed requirements are not allowed as constraints"
elif req.editable:
problem = "Editable requirements are not allowed as constraints"
elif req.extras:
problem = "Constraints cannot have extras"
if problem:
deprecated(
reason=(
"Constraints are only allowed to take the form of a package "
"name and a version specifier. Other forms were originally "
"permitted as an accident of the implementation, but were "
"undocumented. The new implementation of the resolver no "
"longer supports these forms."
),
replacement="replacing the constraint with a requirement",
# No plan yet for when the new resolver becomes default
gone_in=None,
issue=8210,
)
return problem

While is_constraint use Requiement as parameter:

def is_constraint(dep):
# https://pip.pypa.io/en/stable/user_guide/#constraints-files
# constraints must have a name, they cannot be editable, and they cannot specify extras.
return dep.name and not dep.editable and not dep.extras
constraints = []
for dep_name, dep in deps.items():
new_dep = Requirement.from_pipfile(dep_name, dep)
if is_constraint(new_dep):
c = new_dep.as_line().strip()
constraints.append(c)
return constraints

@matteius
Copy link
Member

This is fixed in pipenv==2022.8.19

@clewingriffith
Copy link
Author

Thankyou guys for the extremely quick turnaround on this.

@antoniomika
Copy link

In pipenv, version 2022.8.24, we're still seeing a deprecation notice with extras in the constraints:

Warning: /usr/local/lib/python3.7/site-packages/pipenv/patched/pip/_internal/req/req_install.py:879: PipDeprecationWarning: DEPRECATION: Constraints are only allowed to take the form of a package name and a version specifier. Other forms were originally permitted as an accident of the implementation, but were undocumented. The new implementation of the resolver no longer supports these forms. A possible replacement is replacing the constraint with a requirement. Discussion can be found at https://github.com/pypa/pip/issues/8210
  issue=8210,

And failures occurring if a file was used to install a package (which from what I can tell, is still supported). pip-compile seems to have a --strip-extras flag (something similar might be the proper solution?) but we still have the failing installs when using a file configuration setting.

@matteius matteius reopened this Aug 30, 2022
@matteius
Copy link
Member

@antoniomika Is the file nor marked editable = "true" in the Pipfile? We spotted a bug with #egg fragment getting appended to some file installs ... expecting a new release soon that fixes that. Do you have a stack trace?

@dqkqd Could you look into that PipDeprecationWarning when you get a chance?

@antoniomika
Copy link

Here's the stacktrace we were seeing:

[ResolutionFailure]:   File "/usr/local/lib/python3.7/site-packages/pipenv/resolver.py", line 825, in _main
[ResolutionFailure]:       pre, clear, verbose, system, write, requirements_dir, packages, dev
[ResolutionFailure]:   File "/usr/local/lib/python3.7/site-packages/pipenv/resolver.py", line 780, in resolve_packages
[ResolutionFailure]:       requirements_dir=requirements_dir,
[ResolutionFailure]:   File "/usr/local/lib/python3.7/site-packages/pipenv/resolver.py", line 760, in resolve
[ResolutionFailure]:       req_dir=requirements_dir,
[ResolutionFailure]:   File "/usr/local/lib/python3.7/site-packages/pipenv/utils/resolver.py", line 1098, in resolve_deps
[ResolutionFailure]:       req_dir=req_dir,
[ResolutionFailure]:   File "/usr/local/lib/python3.7/site-packages/pipenv/utils/resolver.py", line 882, in actually_resolve_deps
[ResolutionFailure]:       resolver.resolve()
[ResolutionFailure]:   File "/usr/local/lib/python3.7/site-packages/pipenv/utils/resolver.py", line 686, in resolve
[ResolutionFailure]:       raise ResolutionFailure(message=str(e))
[pipenv.exceptions.ResolutionFailure]: Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  You can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
  Hint: try $ pipenv lock --pre if it is a pre-release dependency.
ERROR: Unnamed requirements are not allowed as constraints

We don't have editable set to true either, but I imagine that is also an issue with the new pip resolver. Here's the three that could cause a deprecation (or failure):

    if not req.name:
        problem = "Unnamed requirements are not allowed as constraints"
    elif req.editable:
        problem = "Editable requirements are not allowed as constraints"
    elif req.extras:
        problem = "Constraints cannot have extras"

from here. Let me know what else I can do to help!

@dqkqd
Copy link
Contributor

dqkqd commented Aug 30, 2022

I see, function check_invalid_constraint_type is used to check if a package is valid as a constraint.

problem = check_invalid_constraint_type(new_dep.as_ireq())
if not problem:
c = new_dep.as_line().strip()

That check_invalid_constraint_type throws out the warning even we are not using invalid package as constraint.

def check_invalid_constraint_type(req: InstallRequirement) -> str:
# Check for unsupported forms
problem = ""
if not req.name:
problem = "Unnamed requirements are not allowed as constraints"
elif req.editable:
problem = "Editable requirements are not allowed as constraints"
elif req.extras:
problem = "Constraints cannot have extras"
if problem:
deprecated(
reason=(
"Constraints are only allowed to take the form of a package "
"name and a version specifier. Other forms were originally "
"permitted as an accident of the implementation, but were "
"undocumented. The new implementation of the resolver no "
"longer supports these forms."
),
replacement="replacing the constraint with a requirement",
# No plan yet for when the new resolver becomes default
gone_in=None,
issue=8210,
)
return problem

Rewrite that function is a solution but we have to keep track if pip changes it as well. Is there anyway we could use it without emitting any warning?

@antoniomika
Copy link

pip-compile seems to just strip the extras from a constraint file, which should be okay.

Unnamed reqs (files) and editable ones might be a different story.

@matteius
Copy link
Member

I see, function check_invalid_constraint_type is used to check if a package is valid as a constraint.

problem = check_invalid_constraint_type(new_dep.as_ireq())
if not problem:
c = new_dep.as_line().strip()

That check_invalid_constraint_type throws out the warning even we are not using invalid package as constraint.

def check_invalid_constraint_type(req: InstallRequirement) -> str:
# Check for unsupported forms
problem = ""
if not req.name:
problem = "Unnamed requirements are not allowed as constraints"
elif req.editable:
problem = "Editable requirements are not allowed as constraints"
elif req.extras:
problem = "Constraints cannot have extras"
if problem:
deprecated(
reason=(
"Constraints are only allowed to take the form of a package "
"name and a version specifier. Other forms were originally "
"permitted as an accident of the implementation, but were "
"undocumented. The new implementation of the resolver no "
"longer supports these forms."
),
replacement="replacing the constraint with a requirement",
# No plan yet for when the new resolver becomes default
gone_in=None,
issue=8210,
)
return problem

Rewrite that function is a solution but we have to keep track if pip changes it as well. Is there anyway we could use it without emitting any warning?

I don't think we need to rewrite the pip function, I think its actually warning us of valid errors. I think the issue is that we need to filter out certain requirements from the Pipfile if they are constraints, because it ends up confusing the resolver if they get included. Probably for the extras we can do something like:

new_dep = Requirement.from_pipfile(dep_name, dep).as_ireq()
if new_dep .extras:
    new_dep .extras = None
if not new_dep.name or new_dep.editble:
    continue
....

@dqkqd
Copy link
Contributor

dqkqd commented Aug 30, 2022

Because all of the packages from [packages] will be put into that function to filter out non-constraint packages. So if there is, for example editable package in [packages], the warning messages will be thrown out. But of course, those non-constraint packages are not used in constraint-file.

@matteius
Copy link
Member

matteius commented Aug 30, 2022

@dqkqd I see what you mean now, we are using that method to filter them out already, but thats causing the emitting of a deprecation warning because that method seems to expect that data was already cleaned -- probably we should just have our own method for this like you were suggesting, be it a similar copy or a simplified refactor.

That begs the question of what is causing resolution failures for @antoniomika -- can you elaborate on the file configuration setting you are using? Do you just mean local editable installs, or something else?

@antoniomika
Copy link

It was a package fork with some additions from a PR. Here's the line:

django-redis = {file = "https://github.com/antoniomika/django-redis/archive/4.12.2.zip"}

@matteius
Copy link
Member

matteius commented Aug 30, 2022

@antoniomika It would appear you must have a conflicting requirement with something that specific zip requires, because I can install it with requests by themselves:

Have you tried running pipenv install --skip-lock and inspect pipenv graph?

matte@LAPTOP-N5VSGIBD MINGW64 ~/Projects/pipenv-triage/pipenv-5273
$ pipenv install requests
Creating a virtualenv for this project...
Pipfile: C:\Users\matte\Projects\pipenv-triage\pipenv-5273\Pipfile
Using default python from C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.1520.0_x64__qbz5n2kfra8p0\python.exe (3.10.5) to create virtualenv...
[   =] Creating virtual environment...created virtual environment CPython3.10.5.final.0-64 in 2595ms
  creator CPython3Windows(dest=C:\c\users\matte\.virtualenvs\pipenv-5273-3M5UDgwN, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=C:\Users\matte\AppData\Local\pypa\virtualenv)
    added seed packages: pip==22.2.2, setuptools==63.3.0, wheel==0.37.1
  activators BashActivator,BatchActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator

Successfully created virtual environment!
Virtualenv location: \c\Users\matte\.virtualenvs\pipenv-5273-3M5UDgwN
Creating a Pipfile for this project...
Installing requests...
Adding requests to Pipfile's [packages]...
Installation Succeeded
Pipfile.lock not found, creating...
Locking [packages] dependencies...
           Building requirements...
Resolving dependencies...
Success!
Locking [dev-packages] dependencies...
Updated Pipfile.lock (a290a1)!
Installing dependencies from Pipfile.lock (a290a1)...
  ================================ 0/0 - 00:00:00
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.

matte@LAPTOP-N5VSGIBD MINGW64 ~/Projects/pipenv-triage/pipenv-5273
$ vim Pipfile

matte@LAPTOP-N5VSGIBD MINGW64 ~/Projects/pipenv-triage/pipenv-5273
$ pipenv install
Pipfile.lock (a290a1) out of date, updating to (10741c)...
Locking [packages] dependencies...
           Building requirements...
Resolving dependencies...
Success!
Locking [dev-packages] dependencies...
Updated Pipfile.lock (10741c)!
Installing dependencies from Pipfile.lock (10741c)...
  ================================ 11/11 - 00:00:08
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.

matte@LAPTOP-N5VSGIBD MINGW64 ~/Projects/pipenv-triage/pipenv-5273
$ cat Pipfile
[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
requests = "*"
django-redis = {file = "https://github.com/antoniomika/django-redis/archive/4.12.2.zip"}

[dev-packages]

[requires]
python_version = "3.10"

matte@LAPTOP-N5VSGIBD MINGW64 ~/Projects/pipenv-triage/pipenv-5273
$ cat Pipfile.lock
{
    "_meta": {
        "hash": {
            "sha256": "22b8f975edd17661ac607aa5a0741f6496d3b9e841c72b7a7411746eef10741c"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.10"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "asgiref": {
            "hashes": [
                "sha256:1d2880b792ae8757289136f1db2b7b99100ce959b2aa57fd69dab783d05afac4",
                "sha256:4a29362a6acebe09bf1d6640db38c1dc3d9217c68e6f9f6204d72667fc19a424"
            ],
            "markers": "python_version >= '3.7'",
            "version": "==3.5.2"
        },
        "async-timeout": {
            "hashes": [
                "sha256:2163e1640ddb52b7a8c80d0a67a08587e5d245cc9c553a74a847056bc2976b15",
                "sha256:8ca1e4fcf50d07413d66d1a5e416e42cfdf5851c981d679a09851a6853383b3c"
            ],
            "markers": "python_version >= '3.6'",
            "version": "==4.0.2"
        },
        "certifi": {
            "hashes": [
                "sha256:84c85a9078b11105f04f3036a9482ae10e4621616db313fe045dd24743a0820d",
                "sha256:fe86415d55e84719d75f8b69414f6438ac3547d2078ab91b67e779ef69378412"
            ],
            "markers": "python_version >= '3.6'",
            "version": "==2022.6.15"
        },
        "charset-normalizer": {
            "hashes": [
                "sha256:5a3d016c7c547f69d6f81fb0db9449ce888b418b5b9952cc5e6e66843e9dd845",
                "sha256:83e9a75d1911279afd89352c68b45348559d1fc0506b054b346651b5e7fee29f"
            ],
            "markers": "python_version >= '3.6'",
            "version": "==2.1.1"
        },
        "deprecated": {
            "hashes": [
                "sha256:43ac5335da90c31c24ba028af536a91d41d53f9e6901ddb021bcc572ce44e38d",
                "sha256:64756e3e14c8c5eea9795d93c524551432a0be75629f8f29e67ab8caf076c76d"
            ],
            "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'",
            "version": "==1.2.13"
        },
        "django": {
            "hashes": [
                "sha256:031ccb717782f6af83a0063a1957686e87cb4581ea61b47b3e9addf60687989a",
                "sha256:032f8a6fc7cf05ccd1214e4a2e21dfcd6a23b9d575c6573cacc8c67828dbe642"
            ],
            "markers": "python_version >= '3.8'",
            "version": "==4.1"
        },
        "django-redis": {
            "file": "https://github.com/antoniomika/django-redis/archive/4.12.2.zip",
            "hashes": [
                "sha256:2b84c91a174f5cb06e26d9772567ae8752f6cfe090f2d3fff2c39fd77548183e"
            ],
            "version": "==4.12.1"
        },
        "idna": {
            "hashes": [
                "sha256:84d9dd047ffa80596e0f246e2eab0b391788b0503584e8945f2368256d2735ff",
                "sha256:9d643ff0a55b762d5cdb124b8eaa99c66322e2157b69160bc32796e824360e6d"
            ],
            "markers": "python_version >= '3.5'",
            "version": "==3.3"
        },
        "packaging": {
            "hashes": [
                "sha256:dd47c42927d89ab911e606518907cc2d3a1f38bbd026385970643f9c5b8ecfeb",
                "sha256:ef103e05f519cdc783ae24ea4e2e0f508a9c99b2d4969652eed6a2e1ea5bd522"
            ],
            "markers": "python_version >= '3.6'",
            "version": "==21.3"
        },
        "pyparsing": {
            "hashes": [
                "sha256:2b020ecf7d21b687f219b71ecad3631f644a47f01403fa1d1036b0c6416d70fb",
                "sha256:5026bae9a10eeaefb61dab2f09052b9f4307d44aee4eda64b309723d8d206bbc"
            ],
            "markers": "python_full_version >= '3.6.8'",
            "version": "==3.0.9"
        },
        "redis": {
            "hashes": [
                "sha256:a52d5694c9eb4292770084fa8c863f79367ca19884b329ab574d5cb2036b3e54",
                "sha256:ddf27071df4adf3821c4f2ca59d67525c3a82e5f268bed97b813cb4fabf87880"
            ],
            "markers": "python_version >= '3.6'",
            "version": "==4.3.4"
        },
        "requests": {
            "hashes": [
                "sha256:7c5599b102feddaa661c826c56ab4fee28bfd17f5abca1ebbe3e7f19d7c97983",
                "sha256:8fefa2a1a1365bf5520aac41836fbee479da67864514bdb821f31ce07ce65349"
            ],
            "index": "pypi",
            "version": "==2.28.1"
        },
        "sqlparse": {
            "hashes": [
                "sha256:0c00730c74263a94e5a9919ade150dfc3b19c574389985446148402998287dae",
                "sha256:48719e356bb8b42991bdbb1e8b83223757b93789c00910a616a071910ca4a64d"
            ],
            "markers": "python_version >= '3.5'",
            "version": "==0.4.2"
        },
        "tzdata": {
            "hashes": [
                "sha256:21f4f0d7241572efa7f7a4fdabb052e61b55dc48274e6842697ccdf5253e5451",
                "sha256:c3119520447d68ef3eb8187a55a4f44fa455f30eb1b4238fa5691ba094f2b05b"
            ],
            "markers": "sys_platform == 'win32'",
            "version": "==2022.2"
        },
        "urllib3": {
            "hashes": [
                "sha256:3fa96cf423e6987997fc326ae8df396db2a8b7c667747d47ddd8ecba91f4a74e",
                "sha256:b930dd878d5a8afb066a637fbb35144fe7901e3b209d1cd4f524bd0e9deee997"
            ],
            "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4, 3.5' and python_version < '4'",
            "version": "==1.26.12"
        },
        "wrapt": {
            "hashes": [
                "sha256:00b6d4ea20a906c0ca56d84f93065b398ab74b927a7a3dbd470f6fc503f95dc3",
                "sha256:01c205616a89d09827986bc4e859bcabd64f5a0662a7fe95e0d359424e0e071b",
                "sha256:02b41b633c6261feff8ddd8d11c711df6842aba629fdd3da10249a53211a72c4",
                "sha256:07f7a7d0f388028b2df1d916e94bbb40624c59b48ecc6cbc232546706fac74c2",
                "sha256:11871514607b15cfeb87c547a49bca19fde402f32e2b1c24a632506c0a756656",
                "sha256:1b376b3f4896e7930f1f772ac4b064ac12598d1c38d04907e696cc4d794b43d3",
                "sha256:21ac0156c4b089b330b7666db40feee30a5d52634cc4560e1905d6529a3897ff",
                "sha256:257fd78c513e0fb5cdbe058c27a0624c9884e735bbd131935fd49e9fe719d310",
                "sha256:2b39d38039a1fdad98c87279b48bc5dce2c0ca0d73483b12cb72aa9609278e8a",
                "sha256:2cf71233a0ed05ccdabe209c606fe0bac7379fdcf687f39b944420d2a09fdb57",
                "sha256:2fe803deacd09a233e4762a1adcea5db5d31e6be577a43352936179d14d90069",
                "sha256:3232822c7d98d23895ccc443bbdf57c7412c5a65996c30442ebe6ed3df335383",
                "sha256:34aa51c45f28ba7f12accd624225e2b1e5a3a45206aa191f6f9aac931d9d56fe",
                "sha256:36f582d0c6bc99d5f39cd3ac2a9062e57f3cf606ade29a0a0d6b323462f4dd87",
                "sha256:380a85cf89e0e69b7cfbe2ea9f765f004ff419f34194018a6827ac0e3edfed4d",
                "sha256:40e7bc81c9e2b2734ea4bc1aceb8a8f0ceaac7c5299bc5d69e37c44d9081d43b",
                "sha256:43ca3bbbe97af00f49efb06e352eae40434ca9d915906f77def219b88e85d907",
                "sha256:4fcc4649dc762cddacd193e6b55bc02edca674067f5f98166d7713b193932b7f",
                "sha256:5a0f54ce2c092aaf439813735584b9537cad479575a09892b8352fea5e988dc0",
                "sha256:5a9a0d155deafd9448baff28c08e150d9b24ff010e899311ddd63c45c2445e28",
                "sha256:5b02d65b9ccf0ef6c34cba6cf5bf2aab1bb2f49c6090bafeecc9cd81ad4ea1c1",
                "sha256:60db23fa423575eeb65ea430cee741acb7c26a1365d103f7b0f6ec412b893853",
                "sha256:642c2e7a804fcf18c222e1060df25fc210b9c58db7c91416fb055897fc27e8cc",
                "sha256:6a9a25751acb379b466ff6be78a315e2b439d4c94c1e99cb7266d40a537995d3",
                "sha256:6b1a564e6cb69922c7fe3a678b9f9a3c54e72b469875aa8018f18b4d1dd1adf3",
                "sha256:6d323e1554b3d22cfc03cd3243b5bb815a51f5249fdcbb86fda4bf62bab9e164",
                "sha256:6e743de5e9c3d1b7185870f480587b75b1cb604832e380d64f9504a0535912d1",
                "sha256:709fe01086a55cf79d20f741f39325018f4df051ef39fe921b1ebe780a66184c",
                "sha256:7b7c050ae976e286906dd3f26009e117eb000fb2cf3533398c5ad9ccc86867b1",
                "sha256:7d2872609603cb35ca513d7404a94d6d608fc13211563571117046c9d2bcc3d7",
                "sha256:7ef58fb89674095bfc57c4069e95d7a31cfdc0939e2a579882ac7d55aadfd2a1",
                "sha256:80bb5c256f1415f747011dc3604b59bc1f91c6e7150bd7db03b19170ee06b320",
                "sha256:81b19725065dcb43df02b37e03278c011a09e49757287dca60c5aecdd5a0b8ed",
                "sha256:833b58d5d0b7e5b9832869f039203389ac7cbf01765639c7309fd50ef619e0b1",
                "sha256:88bd7b6bd70a5b6803c1abf6bca012f7ed963e58c68d76ee20b9d751c74a3248",
                "sha256:8ad85f7f4e20964db4daadcab70b47ab05c7c1cf2a7c1e51087bfaa83831854c",
                "sha256:8c0ce1e99116d5ab21355d8ebe53d9460366704ea38ae4d9f6933188f327b456",
                "sha256:8d649d616e5c6a678b26d15ece345354f7c2286acd6db868e65fcc5ff7c24a77",
                "sha256:903500616422a40a98a5a3c4ff4ed9d0066f3b4c951fa286018ecdf0750194ef",
                "sha256:9736af4641846491aedb3c3f56b9bc5568d92b0692303b5a305301a95dfd38b1",
                "sha256:988635d122aaf2bdcef9e795435662bcd65b02f4f4c1ae37fbee7401c440b3a7",
                "sha256:9cca3c2cdadb362116235fdbd411735de4328c61425b0aa9f872fd76d02c4e86",
                "sha256:9e0fd32e0148dd5dea6af5fee42beb949098564cc23211a88d799e434255a1f4",
                "sha256:9f3e6f9e05148ff90002b884fbc2a86bd303ae847e472f44ecc06c2cd2fcdb2d",
                "sha256:a85d2b46be66a71bedde836d9e41859879cc54a2a04fad1191eb50c2066f6e9d",
                "sha256:a9a52172be0b5aae932bef82a79ec0a0ce87288c7d132946d645eba03f0ad8a8",
                "sha256:aa31fdcc33fef9eb2552cbcbfee7773d5a6792c137b359e82879c101e98584c5",
                "sha256:b014c23646a467558be7da3d6b9fa409b2c567d2110599b7cf9a0c5992b3b471",
                "sha256:b21bb4c09ffabfa0e85e3a6b623e19b80e7acd709b9f91452b8297ace2a8ab00",
                "sha256:b5901a312f4d14c59918c221323068fad0540e34324925c8475263841dbdfe68",
                "sha256:b9b7a708dd92306328117d8c4b62e2194d00c365f18eff11a9b53c6f923b01e3",
                "sha256:d1967f46ea8f2db647c786e78d8cc7e4313dbd1b0aca360592d8027b8508e24d",
                "sha256:d52a25136894c63de15a35bc0bdc5adb4b0e173b9c0d07a2be9d3ca64a332735",
                "sha256:d77c85fedff92cf788face9bfa3ebaa364448ebb1d765302e9af11bf449ca36d",
                "sha256:d79d7d5dc8a32b7093e81e97dad755127ff77bcc899e845f41bf71747af0c569",
                "sha256:dbcda74c67263139358f4d188ae5faae95c30929281bc6866d00573783c422b7",
                "sha256:ddaea91abf8b0d13443f6dac52e89051a5063c7d014710dcb4d4abb2ff811a59",
                "sha256:dee0ce50c6a2dd9056c20db781e9c1cfd33e77d2d569f5d1d9321c641bb903d5",
                "sha256:dee60e1de1898bde3b238f18340eec6148986da0455d8ba7848d50470a7a32fb",
                "sha256:e2f83e18fe2f4c9e7db597e988f72712c0c3676d337d8b101f6758107c42425b",
                "sha256:e3fb1677c720409d5f671e39bac6c9e0e422584e5f518bfd50aa4cbbea02433f",
                "sha256:ee2b1b1769f6707a8a445162ea16dddf74285c3964f605877a20e38545c3c462",
                "sha256:ee6acae74a2b91865910eef5e7de37dc6895ad96fa23603d1d27ea69df545015",
                "sha256:ef3f72c9666bba2bab70d2a8b79f2c6d2c1a42a7f7e2b0ec83bb2f9e383950af"
            ],
            "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4'",
            "version": "==1.14.1"
        }
    },
    "develop": {}
}

@antoniomika
Copy link

@matteius interesting. I went down that route and couldn't find anything specific. I also found pipenv==2022.8.5 works fine (prior to the constraint changes) which brought me here.

I'll keep digging, but at the least the deprecation fix would still be a nice to have.

@matteius
Copy link
Member

@antoniomika are you locking dev dependencies?

@antoniomika
Copy link

We have both. I'm running a pipenv update.

@matteius
Copy link
Member

@antoniomika There may be a package shared between default and dev that are on conflicting versions?

@antoniomika
Copy link

@matteius not that I know of, but I'm doing a deeper dive now. I have confirmed everything works fine with the older version but I'll update you once I know more.

@matteius
Copy link
Member

@antoniomika if you see two package with the same name in the Pipfile.lock one in dev, and one in default, that might be a good indicator.

@antoniomika
Copy link

antoniomika commented Sep 1, 2022

@matteius I was able to find the constraint that was the issue, there were two different versions. Is it possible to get that information outputted as an error?

I'm now seeing this issue with a different package:

{'req': None, 'comes_from': '-c /tmp/pipenv5itk0bj7requirements/pipenv-cbcnfu2m-constraints.txt (line 9)', 'constraint': True, 'editable': False, 'permit_editable_wheels': False, 'legacy_install_reason': None, 'source_dir': None, 'link': <Link https://github.com/antoniomika/channels_rabbitmq/archive/v1.3.2.zip>, 'original_link': <Link https://github.com/antoniomika/channels_rabbitmq/archive/v1.3.2.zip>, 'original_link_is_in_wheel_cache': False, 'download_info': None, 'local_file_path': None, 'extras': set(), 'markers': None, 'satisfied_by': None, 'should_reinstall': False, '_temp_build_dir': None, 'install_succeeded': None, 'install_options': [], 'global_options': [], 'hash_options': {}, 'config_settings': None, 'prepared': False, 'user_supplied': False, 'isolated': True, 'build_env': <pipenv.patched.pip._internal.build_env.NoOpBuildEnvironment object at 0x7f5f9728d7d0>, 'metadata_directory': None, 'pyproject_requires': None, 'requirements_to_check': [], 'pep517_backend': None, 'use_pep517': None, 'needs_more_preparation': False}
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/pipenv/resolver.py", line 858, in <module>
    main()
  File "/usr/local/lib/python3.7/site-packages/pipenv/resolver.py", line 853, in main
    dev=parsed.dev,
  File "/usr/local/lib/python3.7/site-packages/pipenv/resolver.py", line 825, in _main
    pre, clear, verbose, system, write, requirements_dir, packages, dev
  File "/usr/local/lib/python3.7/site-packages/pipenv/resolver.py", line 780, in resolve_packages
    requirements_dir=requirements_dir,
  File "/usr/local/lib/python3.7/site-packages/pipenv/resolver.py", line 760, in resolve
    req_dir=requirements_dir,
  File "/usr/local/lib/python3.7/site-packages/pipenv/utils/resolver.py", line 1102, in resolve_deps
    req_dir=req_dir,
  File "/usr/local/lib/python3.7/site-packages/pipenv/utils/resolver.py", line 886, in actually_resolve_deps
    resolver.resolve()
  File "/usr/local/lib/python3.7/site-packages/pipenv/utils/resolver.py", line 685, in resolve
    self.constraints  # For some reason it is important to evaluate constraints before resolver context
  File "/usr/local/lib/python3.7/site-packages/pipenv/utils/resolver.py", line 650, in constraints
    self._constraints.sort(key=lambda ireq: ireq.name)
TypeError: '<' not supported between instances of 'NoneType' and 'str'

I outputted the actual constraint and it doesn't look like a name is associated with it (the output is above). Not sure if that's something that needs to be fixed or is another problem on our end (which I assume it is?).

@matteius matteius reopened this Sep 1, 2022
@matteius
Copy link
Member

matteius commented Sep 1, 2022

That looks like a possible bug, somehow the Install requirement name is None. I'll think more about it. Cc @dqkqd

@dqkqd
Copy link
Contributor

dqkqd commented Sep 1, 2022

That looks like a possible bug, somehow the Install requirement name is None. I'll think more about it. Cc @dqkqd

Yes, it's from the commit d33f4ec.
I didn't aware that Install Requirement name could be None. To suppress this error, there's a dirty fix:

self._constraints.sort(key=lambda ireq: ireq.name or "")

At first, I thought package name is unique. Turn out I was wrong, when there are multiple no-name packages, they will be duplicated.
Now I'm not sure if just using name as sort criteria is stable in that case. Is there anything unique could be used to compare two Install Requirement?
If there isn't, something like the the path of requirements file: /tmp/random-dir/random-file.txt may enter as criteria, and because it's random, the sort might not be stable and issue like #5239, might happen again.

@antoniomika
Copy link

Does making the name the url not solve the issue as well (and removes the duplicate issue)? That wouldn't necessarily solve all cases though.

@matteius
Copy link
Member

matteius commented Sep 1, 2022

@dqkqd we had a report that the constraints issue was not solved even with your change: #4660 (comment)

I am going to open a PR to revert this since based on the prior report: https://github.com/pypa/pipenv/pull/5299/files

I also did some sorting in this PR yesterday but I think these are only pip constraints that get passed this way: https://github.com/pypa/pipenv/pull/5313/files#diff-3542045ed7f41738cd4519e02753c9e83af5654c56c124598f0fbbd82405ec49R134

@matteius
Copy link
Member

matteius commented Sep 1, 2022

Well reverting the commit @dqkqd made does cause the constraints to jump around a lot more when locking, probably we need to think through this more than a revert. I like the idea of making it safer, but defaulting to empty string seems problematic.

@dqkqd
Copy link
Contributor

dqkqd commented Sep 1, 2022

I knew the root of this issue now, sorting or not isn't the problem. It's the constraints with name is None is passed into resolver. Somehow this package django-redis = {file = "https://github.com/antoniomika/django-redis/archive/4.12.2.zip"} passed the check is_constraint but have the name is None.

@antoniomika
Copy link

(for reference, the fork I had was unneeded and I removed that dependency in favor of the upstream version. The latest issue I had was with channels-rabbitmq = {file = "https://github.com/antoniomika/channels_rabbitmq/archive/v1.3.2.zip"})

@matteius
Copy link
Member

matteius commented Sep 1, 2022

@dqkqd Is it the is_constraints method do you think? If so I suspect we could also check if the InstallRequirement does not have a name in your is_constraints check. Also I see there is also a spot in requirementslib that does a weird is_constraint check: https://github.com/pypa/pipenv/blob/main/pipenv/vendor/requirementslib/models/dependencies.py#L241

@dqkqd
Copy link
Contributor

dqkqd commented Sep 1, 2022

I think it's the bug in requirementlib. Actually the result from new_dep.as_line() at line 284.

def get_constraints_from_deps(deps):
"""Get contraints from Pipfile-formatted dependency"""
def is_constraints(dep: InstallRequirement) -> bool:
return dep.name and not dep.editable and not dep.extras
constraints = []
for dep_name, dep in deps.items():
new_dep = Requirement.from_pipfile(dep_name, dep)
if is_constraints(new_dep.as_ireq()):
c = new_dep.as_line().strip()
constraints.append(c)
return constraints

For example:
if I passed in the packages django-redis = {file = "https://github.com/antoniomika/django-redis/archive/4.12.2.zip"} the output I expected is django-redis @ https://github.com/antoniomika/django-redis/archive/4.12.2.zip as mentioned in
https://pip.pypa.io/en/stable/reference/requirements-file-format/#example

But the actual output is https://github.com/antoniomika/django-redis/archive/4.12.2.zip, and it doesn't contain name anymore, it's name is None now.

The name is disappeared because:
as_line call self.line_instance.get_line

parts = self.line_instance.get_line(
with_prefix=True,
with_hashes=include_hashes,
with_markers=include_markers,
as_list=as_list,

self.line_instance.get_line check if package should have a name or not:

if self.is_named:
line = self.name_and_specifier

And because it has url, so it's name is discarded from now.

def is_named(self):
# type: () -> bool
return not (
self.is_file_url
or self.is_url
or self.is_file
or self.is_vcs
or self.is_direct_url

@dqkqd
Copy link
Contributor

dqkqd commented Sep 1, 2022

Or maybe I'm wrong, PipDeprecationWarning says Constraints are only allowed to take the form of a package name and a version specifier so it should not containts url. In that case we could easily fix that by using new_dep.is_named.

new_dep = Requirement.from_pipfile(dep_name, dep) 
    if new_dep.is_named and is_constraints(new_dep.as_ireq()): 
        c = new_dep.as_line().strip() 
        constraints.append(c) 

But the documentation does not say that. So maybe it could have url as constraints? I tried using this constraints file with pip and it did work.

# constraints file
django-redis @ https://github.com/antoniomika/django-redis/archive/4.12.2.zip 

@matteius
Copy link
Member

matteius commented Sep 2, 2022

@dqkqd I found this interesting and has a lot of context: pypa/pip#8210

That was a long read, but the conclusion is that "unnamed constraints are not allowed."

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment