Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

poetry sets incorrect package source type when multiple site-package directories are present #2225

Closed
3 tasks done
abn opened this issue Mar 26, 2020 · 2 comments · Fixed by #2360
Closed
3 tasks done
Labels
kind/bug Something isn't working as expected
Milestone

Comments

@abn
Copy link
Member

abn commented Mar 26, 2020

  • I am on the latest Poetry version.
  • I have searched the issues of this repo and believe that this is not a duplicate.
  • If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

Issue

TLDR; installed package detection breaks when multiple site-specific directories exist.

In certain scenarios, there exists multiple site-specific package directories. The most
common of this is due to the distinction between purelib and platlib paths
(see sysconfig#Installation Paths).

Another example is the python:2.7 image.

docker run --rm -it python:2.7 python -c 'import site; print(site.getsitepackages())'
['/usr/local/lib/python2.7/site-packages', '/usr/local/lib/site-python']

Different distros have their own view of site-specific file directories. And I suspect, different python installers do too.

The issue can be reproduced using the snippet provided at the end. Here, in an environment with virtualenv >= 20.0.0, tox creates the environment such that lib64 directory is not linked to lib.

Note that;

  • we use a Red Hat distro (Fedora) as this is one where the platlib directory differs from the purelib directory.
  • if the virtual environment was created using the venv module or virtualenv < 20.0.0, lib64 will be a symlink to lib.
  • the change in behaviour for virtualenv is by design (lib64 directory is no longer symlinked to lib? pypa/virtualenv#1751)

Once executed you will notice that the first tox run executes as expected.

<snipped>
Package operations: 13 installs, 0 updates, 0 removals, 1 skipped
<snipped>

However, the second run will update all packages. This is not expected.

<snipped>
Package operations: 0 installs, 13 updates, 0 removals, 1 skipped
<snipped>

The cause for this is the differing source types for the installed package and required package.

elif package.source_type != pkg.source_type:
operations.append(Update(pkg, package))

The root cause for this is because the package is being detected as a non-standard package since
only the first seen site-package directory is being checked.

is_standard_package = True
try:
path.relative_to(env.site_packages)
except ValueError:
is_standard_package = False
if is_standard_package:
continue

The checking of multiple directories were briefly talked about in a comment in #1786. However, I do not think that was resolved there.

Possible Solutions

In both these cases, Env.site_packages will need to be an iterable.

  1. Iterate over all paths in sys.path filtering for site-packages. This will be the simplest solution, however will not account for cases like above where the directory is /usr/local/lib/site-python.
  2. Use site.getsitepackages(). This will not work for environments created using virtualenv < 20.0.0 as a bundled version for site.py was being used. Since in these environments lib64 is already being symlinked, we could simply fallback to searching for site-packages assuming that the virtual environments will not be creating dist-packages. We should also bump minimum version for poetry's virtualenv dependency as well.

Not sure if there is a solution without any trade-off.

Reproducer

docker run --rm -i fedora:31 <<SCRIPT_EOF
set -x

dnf -qy install -y pip

pip install --quiet --upgrade poetry tox virtualenv>=20.0.0

install -d poetry-issue-reproducer/poetry_issue_reproducer
pushd poetry-issue-reproducer

touch poetry_issue_reproducer/__init__.py

cat > pyproject.toml <<EOF
[build-system]
requires = ["poetry>=1.0"]
build-backend = "poetry.masonry.api"

[tool.poetry]
name = "poetry-issue-reproducer"
packages = [
    { include = "poetry_issue_reproducer" }
]
version = "1.0.0-alpha.0"
description = "A reproducer for a poetry issue"
authors = [
    "Issue Reproducer <issues@noreply.com>"
]
license = "MIT"

[tool.poetry.dependencies]
python = "^3.7"

[tool.poetry.dev-dependencies]
tox = "^3.13"

[tool.tox]
legacy_tox_ini = """
[tox]
isolated_build = True
envlist = py

[testenv]
whitelist_externals =
    poetry
commands =
    poetry install -vvv

"""
EOF

# first run works as expected
tox

# second run updates packages, this is not expected
tox
SCRIPT_EOF
@abn abn added the kind/bug Something isn't working as expected label Mar 26, 2020
@kasteph
Copy link
Member

kasteph commented Apr 12, 2020

#2281 looks related.

Copy link

github-actions bot commented Mar 3, 2024

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Mar 3, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
kind/bug Something isn't working as expected
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants