"Could not find a version that satisfies the requirement" for package in private repository starting from pipenv==2022.8.31

[aawilson]

Issue description

With a Pipfile like this...

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[[source]]
url = "https://pypi.ourinternalrepo.com/simple"
verify_ssl = true
name = "our-pypi"

[packages]
..etc..
ourinternalpackage = {version = "==0.5.9", index = "our-pypi"}

that generates a Pipfile.lock like this...

{
    "_meta": {
        "hash": {
            "sha256": "something"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.8"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            },
            {
                "name": "our-pypi",
                "url": "https://pypi.ourinternalrepo.com/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "ourinternalpackage ": {
            "hashes": [
                "sha256:something"
            ],
            "index": "our-pypi",
            "version": "==0.1.1"
        },
        ...etc

but when we execute pipenv sync --verbose, it generates a pip install command like so

/usr/local/lib/python3.8/site-packages/pipenv/patched/pip/__pip-runner__.py install -i https://pypi.org/simple --no-input --upgrade --no-deps --exists-action=i -r /tmp/pipenv-etc-requirements/pipenv-etc-hashed-reqs.txt

(note "-i https://pypi.org/simple")

and as might be expected, this fails with 6.552 ERROR: Could not find a version that satisfies the requirement ourinternalpackage==0.1.1 (from versions: none). This appears to have started happening with 2022.8.31--when we pip install -U pipenv==2022.8.30 and run pipenv sync from there, the package resolves successfully.

Expected result

I would expect the package install to generate with a -i pointed at the our-pypi source rather than the default.

Actual result

6.552 ERROR: Could not find a version that satisfies the requirement ourinternalpackage==0.1.1 (from versions: none)

There are also bunches of these, that may or may not be irrelevant. I can't provide the full --verbose output because cleansing it of work-internal information would be too much hassle, but I have it around if anyone wants specific snippets.

An error occurred while installing alembic==1.8.0; python_version >= '3.7' --hash=sha256:a2d4d90da70b30e70352cd9455e35873a255a31402a438fe24815758d7a0e5e1 --hash=sha256:b5ae4bbfc7d1302ed413989d39474d102e7cfa158f6d5969d2497955ffe85a30! Will try again.

Steps to replicate

See above

---

Please run $ pipenv --support, and paste the results here. Don't put backticks (`) around it! The output already contains Markdown formatting.

$ pipenv --support

Pipenv version: '2022.9.8'

Pipenv location: '/usr/local/lib/python3.8/site-packages/pipenv'

Python location: '/usr/local/bin/python'

OS Name: 'posix'

User pip version: '22.2.2'

user Python installations found:

• 3.8.14: /usr/local/bin/python
• 3.8.14: /usr/local/bin/python3
• 3.8.14: /usr/local/bin/python3.8
• 3.7.3: /usr/bin/python3
• 3.7.3: /usr/bin/python3.7
• 3.7.3: /usr/bin/python3.7m
• 2.7.16: /usr/bin/python
• 2.7.16: /usr/bin/python2.7
• 2.7.16: /usr/bin/python2

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.8.14',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '5.4.0-1051-gcp',
 'platform_system': 'Linux',
 'platform_version': '#55~18.04.1-Ubuntu SMP Sun Aug 1 20:38:04 UTC 2021',
 'python_full_version': '3.8.14',
 'python_version': '3.8',
 'sys_platform': 'linux'}

System environment variables:

• PIPENV_VENV_IN_PROJECT
• HOSTNAME
• PYTHON_VERSION
• PWD
• PYTHON_SETUPTOOLS_VERSION
• PIPENV_CACHE_DIR
• HOME
• LANG
• LS_COLORS
• GPG_KEY
• PYTHONPATH
• TERM
• SHLVL
• PYTHON_PIP_VERSION
• PYTHON_GET_PIP_SHA256
• PYTHON_GET_PIP_URL
• PATH
• _
• PIP_DISABLE_PIP_VERSION_CHECK
• PIP_PYTHON_PATH
• PYTHONDONTWRITEBYTECODE
• PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenv–specific environment variables:

• PIPENV_VENV_IN_PROJECT: 1
• PIPENV_CACHE_DIR: /tmp

Debug–specific environment variables:

• PATH: /usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
• LANG: C.UTF-8
• PWD: /code

---

Contents of Pipfile ('/code/Pipfile'):

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]

[dev-packages]

[requires]
python_version = "3.8"

Contents of Pipfile.lock ('/code/Pipfile.lock'):

(see above for selected snippets, full output not included for reasons stated above)

[aawilson]

perusing the output above, I'm seeing that the contents of the Pipfile there are completely different than they should be (even though Pipfile.lock is correct; it should contain the same source entry mentioned above), I'll investigate more on that front

[aawilson]

Ok, I see that the root issue seems to be that we were only copying Pipfile.lock into the container, rather than Pipfile. When I add a COPY statement for the Pipfile as well, pipenv sync works as expected. This changes the bug report: Why does pipenv sync suddenly need a Pipfile to work correctly, rather than being able to correctly resolve all dependencies with the provided Pipfile.lock?

[matteius]

@aawilson I am not sure that this is a new behavior, so there may be something else going on, but you found that having the Pipfile allows it to work? It is an interesting observation, apparently get_source_list gets its sources from the Pipfile, even though its a method that appears to only be used in the install phase. This is not a new behavior though 😕 We should improve that though to pull the sources from just the lock file.

[aawilson]

It's pretty interesting, I definitely get the "broken" behavior in 2022.8.31 and not in 2022.8.30 (since it's a container it's easy to remake the intermediate image up to failure and try different versions). Something seems to have changed between those two versions to get the specified behavior.

[matteius]

@aawilson Just to clarify -- does it work to install if you include the Pipfile as well?

[dqkqd]

Inside pip_install_deps, we call get_source_list with index=None and extra_indexes=None

pipenv/pipenv/core.py

Lines 1638 to 1643 in 9572c31

	sources = get_source_list(
	project,
	index=None,
	extra_indexes=None,
	trusted_hosts=trusted_hosts,
	pypi_mirror=pypi_mirror,

Therefore get_source_list will get source from Pipfile instead of Pipfile.lock (line 91)

pipenv/pipenv/utils/indexes.py

Lines 68 to 98 in 9572c31

	def get_source_list(
	project: Project,
	index: Optional[Union[str, TSource]] = None,
	extra_indexes: Optional[Union[str, List[str]]] = None,
	trusted_hosts: Optional[List[str]] = None,
	pypi_mirror: Optional[str] = None,
	) -> List[TSource]:
	sources: List[TSource] = []
	if index:
	sources.append(get_project_index(project, index))
	if extra_indexes:
	if isinstance(extra_indexes, str):
	extra_indexes = [extra_indexes]
	for source in extra_indexes:
	extra_src = get_project_index(project, source)
	if not sources or extra_src["url"] != sources[0]["url"]:
	sources.append(extra_src)
	for source in project.pipfile_sources:
	if not sources or source["url"] != sources[0]["url"]:
	sources.append(source)
	if not sources:
	sources = project.pipfile_sources[:]
	if pypi_mirror:
	sources = [
	create_mirror_source(pypi_mirror) if is_pypi_url(source["url"]) else source
	for source in sources
	]
	return sources

[dqkqd]

To address this issue, there's a fix that is throwing all the indexes written inside Pipfile.lock into index and extra_indexes from the call of get_source_list.
But in that case, all index and extra_indexes will be passed in pip_command, therefore we could not declare index for specific package anymore.

This could lead to an issue like: "I want to install A from index X, but A is installed from index Y. I don't want that because A from Y is broken."

[matteius]

To address this issue, there's a fix that is throwing all the indexes written inside Pipfile.lock into index and extra_indexes from the call of get_source_list.

Yes, this is what I am thinking get_source_list should be getting its sources from the lock file, not the Pipfile.

But in that case, all index and extra_indexes will be passed in pip_command, therefore we could not declare index for specific package anymore.

This is not quite accurate -- the resolver was patched for the specific indexes, but once packages resolve they are hashed.

This could lead to an issue like: "I want to install A from index X, but A is installed from index Y. I don't want that because A from Y is broken."

Because the normal index specified packages are hashed, the install phase is less sensitive -- If you are installing package A with hash Z then it has to be the same package whether you get it from X or Y -- that is how it works today with batch install as long as you have your Pipfile in the project with all of its sources, so changing where we get the sources from shouldn't matter.

In fact the default behavior is to search all sources. For regular pip_install we check if not search_all_sources and requirement.index in source_names: and restrict to just that index, but we cannot do this for batch install because its more than one item in the batch. If there is a real use case for it, we may have to add additional batches per index if the setting if not search_all_sources but I think that could be addressed outside of an effort to make sure that get_source_list is built from Pipfile.lock.

[aawilson]

@aawilson Just to clarify -- does it work to install if you include the Pipfile as well?

Yeah, it's up there but to confirm explicitly, adding Pipfile to the COPY does cause sync to succeed.

[matteius]

@aawilson Would you be able to help verify the fix of the Pipfile.lock sources being preferred for the install phase? #5380

[aawilson]

Yeah I'll give it a whirl today

[aawilson]

Confirmed success, thanks very much everyone who chipped in. (for posterity, confirmation was done by rolling back my fix above (adding Pipfile to the COPY in my Dockerfile), then installing pipenv with RUN pip install --upgrade git+https://github.com/pypa/pipenv.git@432ced65aaaa19f02d5cae4cf4c341983281c3c4#egg=pipenv, the ref at the end of that PR there. The container built successfully and correctly installed our internal packages from our internal PyPI).

[matteius]

Awesome, thanks for checking @aawilson -- I cut a new version last night to pypi which included this fix.