Poetry from `install-poetry.py` does not install packages to system Python even with `POETRY_VIRTUALENVS_CREATE=0` #3870

ruohola · 2021-04-01T12:42:40Z

I am on the latest Poetry version.
I have searched the issues of this repo and believe that this is not a duplicate.

Poetry installed from the latest install-poetry.py does not install packages to system Python even with POETRY_VIRTUALENVS_CREATE=0.

Steps to reproduce:

Spin up a fresh container for demoing:

$ docker run -it python:3.9.2-buster bash

In the container:

$ curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/45a9b8f20384591d0a33ae876bcf23656f928ec0/install-poetry.py | python -
$ export PATH="/root/.local/bin:$PATH"
$ export POETRY_VIRTUALENVS_CREATE=0
$ poetry --version
Poetry version 1.1.5
$ poetry init  # go with the bare defaults

Install a package (note the "Skipping virtualenv creation" message):

$ poetry add django
Skipping virtualenv creation, as specified in config file.
Using version ^3.1.7 for Django

Updating dependencies
Resolving dependencies... (0.6s)

Writing lock file

Package operations: 4 installs, 0 updates, 0 removals

  • Installing asgiref (3.3.1)
  • Installing pytz (2021.1)
  • Installing sqlparse (0.4.1)
  • Installing django (3.1.7)

Then try to run

$ python -c 'import django'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'django'

We can see that django got installed to here:

$ find / -type d -name django
/root/.local/share/pypoetry/venv/lib/python3.9/site-packages/django
/root/.local/share/pypoetry/venv/lib/python3.9/site-packages/django/forms/templates/django
/root/.local/share/pypoetry/venv/lib/python3.9/site-packages/django/forms/jinja2/django

When pip would install it here:

$ pip install django
$ python -c 'import django'  # no errors
$ find / -type d -name django
/usr/local/lib/python3.9/site-packages/django
/usr/local/lib/python3.9/site-packages/django/forms/templates/django
/usr/local/lib/python3.9/site-packages/django/forms/jinja2/django
/root/.local/share/pypoetry/venv/lib/python3.9/site-packages/django
/root/.local/share/pypoetry/venv/lib/python3.9/site-packages/django/forms/templates/django
/root/.local/share/pypoetry/venv/lib/python3.9/site-packages/django/forms/jinja2/django

The same thing also happens with non-root users. This issue does NOT happen with Poetry installed from the latest get-poetry.py NOR with poetry==1.1.5 installed from pip.

The text was updated successfully, but these errors were encountered:

sdispater · 2021-04-01T12:49:13Z

Thanks for testing the new bootstrap method and for reporting this issue!

I think I know where the issue is coming from (Poetry's own virtual environment is mistakenly seen as the system environment).

I'll try to come up with a fix and let you know.

byteszard · 2021-04-15T02:15:40Z

I have the some problem when i use poetry install packages in docker. This is my temporary solution.

WORKDIR app
COPY ./pyproject.toml ./poetry.lock*  ./
RUN poetry export -f requirements.txt --output requirements.txt && pip install --no-cache-dir -r requirements.txt

ClementWalter · 2021-04-20T17:10:32Z

I have a possibly related issue. I am working inside a venv that I created by hand not using poetry (with --system-site-package), and poetry while seeing the deps already existing, download and install it again, see below:

(.venv) clementwalter@tmp$ poetry config --list
cache-dir = "/home/clementwalter/.cache/pypoetry"
experimental.new-installer = true
installer.parallel = true
virtualenvs.create = false
virtualenvs.in-project = null
virtualenvs.path = "{cache-dir}/virtualenvs"  # /home/clementwalter/.cache/pypoetry/virtualenvs

(.venv) clementwalter@tmp$ poetry env info

Virtualenv
Python:         3.7.10
Implementation: CPython
Path:           /home/clementwalter/tmp/.venv
Valid:          True

System
Platform: linux
OS:       posix
Python:   /opt/miniconda

(.venv) clementwalter@tmp$ poetry add numpy==1.19.2

Updating dependencies
Resolving dependencies... (0.1s)

Writing lock file

Package operations: 0 installs, 1 update, 0 removals

  • Updating numpy (1.19.2 /opt/miniconda/lib/python3.7/site-packages -> 1.19.2)

(.venv) clementwalter@tmp$ python -c "import numpy; print(numpy.__file__)"
/home/clementwalter/tmp/.venv/lib/python3.7/site-packages/numpy/__init__.py

(.venv) clementwalter@tmp$ poetry remove numpy
Updating dependencies
Resolving dependencies... (0.1s)

Writing lock file

Package operations: 0 installs, 0 updates, 1 removal

  • Removing numpy (1.19.2)

(.venv) clementwalter@tmp$ pip install numpy==1.19.2
Requirement already satisfied: numpy==1.19.2 in /opt/miniconda/lib/python3.7/site-packages (1.19.2)

(.venv) clementwalter@tmp$ python -c "import numpy; print(numpy.__file__)"
/opt/miniconda/lib/python3.7/site-packages/numpy/__init__.py

So here poetry does an extra install while pip does not and poetry is not supposed to manage env, so why not keeping the version that already exists?

ruohola · 2021-04-21T13:30:01Z

I have a possibly related issue. I am working inside a venv that I created by hand not using poetry (with --system-site-package), and poetry while seeing the deps already existing, download and install it again, see below:

(.venv) clementwalter@tmp$ poetry config --list
cache-dir = "/home/clementwalter/.cache/pypoetry"
experimental.new-installer = true
installer.parallel = true
virtualenvs.create = false
virtualenvs.in-project = null
virtualenvs.path = "{cache-dir}/virtualenvs"  # /home/clementwalter/.cache/pypoetry/virtualenvs

(.venv) clementwalter@tmp$ poetry env info

Virtualenv
Python:         3.7.10
Implementation: CPython
Path:           /home/clementwalter/tmp/.venv
Valid:          True

System
Platform: linux
OS:       posix
Python:   /opt/miniconda

(.venv) clementwalter@tmp$ poetry add numpy==1.19.2

Updating dependencies
Resolving dependencies... (0.1s)

Writing lock file

Package operations: 0 installs, 1 update, 0 removals

  • Updating numpy (1.19.2 /opt/miniconda/lib/python3.7/site-packages -> 1.19.2)

(.venv) clementwalter@tmp$ python -c "import numpy; print(numpy.__file__)"
/home/clementwalter/tmp/.venv/lib/python3.7/site-packages/numpy/__init__.py

(.venv) clementwalter@tmp$ poetry remove numpy
Updating dependencies
Resolving dependencies... (0.1s)

Writing lock file

Package operations: 0 installs, 0 updates, 1 removal

  • Removing numpy (1.19.2)

(.venv) clementwalter@tmp$ pip install numpy==1.19.2
Requirement already satisfied: numpy==1.19.2 in /opt/miniconda/lib/python3.7/site-packages (1.19.2)

(.venv) clementwalter@tmp$ python -c "import numpy; print(numpy.__file__)"
/opt/miniconda/lib/python3.7/site-packages/numpy/__init__.py

So here poetry does an extra install while pip does not and poetry is not supposed to manage env, so why not keeping the version that already exists?

That seems like a separate issue, but I'm not sure.

JamesXNelson · 2021-04-29T22:02:40Z

a 🖐️ from me too. cannot use install-poetry.sh on relatively clean ubuntu 20.04, but get-poetry.sh works perfectly.

Until python-poetry/poetry#3870 is resolved. Original commit: 8c59f3e

Until python-poetry/poetry#3870 is resolved. Original commit: 8c59f3e2739c5e92c8279b9dfb7775fcf75c23d1 Original-Commit: TeoZosa/cookiecutter-cruft-poetry-tox-pre-commit-ci-cd@c578af1

SaulAryehKohn · 2021-06-09T19:04:05Z

I found that I was able to remove this issue on Ubuntu 20.04 by specifying a particular release of virtualenv in install-poetry.py:

517         with temporary_directory() as tmp_dir:
518             subprocess.call(
519                 [sys.executable, "-m", "pip", "install", "virtualenv==20.0.23", "-t", tmp_dir], # <-- changed from just "virtualenv"
520                 stdout=subprocess.PIPE,
521                 stderr=subprocess.STDOUT,
522             )

python-poetry/poetry#3706 python-poetry/poetry#3870 python-poetry/poetry#4056 Poetry has a new install script, added in python-poetry/poetry#3706. The old get-poetry.py install script is not compatible with Python 3.10, so the new install-poetry.py script will be used. Docker builds and GitHub Actions workflows will be updated to use `POETRY_HOME=/opt/poetry` consistently. As of Poetry 1.1.7, there may be complications in Docker when using install-poetry.py without venvs (`POETRY_VIRTUALENVS_CREATE=false`). While installing dependencies, the following error is frequently seen: ```text OSError Could not find a suitable TLS CA certificate bundle, invalid path: /opt/poetry/venv/lib/python3.9/site-packages/certifi/cacert.pem at /opt/poetry/venv/lib/python3.9/site-packages/requests/adapters.py:227 in cert_verify ``` Poetry may be incorrectly attempting to read from its virtualenv if it's not respecting `POETRY_VIRTUALENVS_CREATE` (python-poetry/poetry#3870). Downstream steps also do not respect `POETRY_VIRTUALENVS_CREATE`, so the application does not run.

python-poetry/poetry#3706 python-poetry/poetry#3870 python-poetry/poetry#4056 Poetry has a new install script, added in python-poetry/poetry#3706. The old get-poetry.py install script is not compatible with Python 3.10. This commit will update the GitHub Actions workflow to use the new install-poetry.py script, with `POETRY_HOME=/opt/poetry` for consistent installs independent of user account, and will also update the workflow to use a virtualenv and run commands with `poetry run` to avoid issues with `POETRY_VIRTUALENVS_CREATE=false`. As of Poetry 1.1.7, there may be complications with install-poetry.py run without venvs (`POETRY_VIRTUALENVS_CREATE=false`). An error is seen: ```text OSError Could not find a suitable TLS CA certificate bundle, invalid path: /opt/poetry/venv/lib/python3.9/site-packages/certifi/cacert.pem at /opt/poetry/venv/lib/python3.9/site-packages/requests/adapters.py:227 in cert_verify ``` Poetry may be incorrectly attempting to read from its virtualenv if it's not respecting `POETRY_VIRTUALENVS_CREATE` (python-poetry/poetry#3870). Downstream steps also do not respect `POETRY_VIRTUALENVS_CREATE`, so the application does not run. To avoid these issues, `poetry run` can be prepended to commands to prompt Poetry to use its virtualenv. Additionally, Poetry errors out with a `JSONDecodeError` when attempting to install packages with Python 3.10 (python-poetry/poetry#4210). Python 3.10 support will be postponed until Poetry is compatible.

github-actions · 2024-03-02T10:05:25Z

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

ruohola added kind/bug Something isn't working as expected status/triage This issue needs to be triaged labels Apr 1, 2021

sdispater self-assigned this Apr 1, 2021

sdispater added Installer area/venv Related to virtualenv management labels Apr 1, 2021

sdispater added this to the 1.2 milestone Apr 19, 2021

joeld1 mentioned this issue Apr 28, 2021

Update Poetry Installation Documentation for Dockerfile examples tiangolo/uvicorn-gunicorn-fastapi-docker#86

Closed

A-Telfer mentioned this issue May 16, 2021

Poetry can't install opencv but pip can (RPI) #4073

Closed

3 tasks

TeoZosa added a commit to TeoZosa/cookiecutter-cruft-poetry-tox-pre-commit-ci-cd that referenced this issue May 26, 2021

🐛 Rollback install-poetry.py migration

c578af1

Until python-poetry/poetry#3870 is resolved. Original commit: 8c59f3e

br3ndonland mentioned this issue Jul 5, 2021

Add Python 3.10 support br3ndonland/inboard#36

Merged

10 tasks

This was referenced Jul 30, 2021

Fix system environment detection #4329

Merged

[1.1] Fix system environment detection #4330

Merged

sdispater closed this as completed in #4329 Jul 30, 2021

br3ndonland mentioned this issue Sep 19, 2021

Pin and test Poetry version br3ndonland/inboard#44

Merged

1 task

abn removed the status/triage This issue needs to be triaged label Mar 3, 2022

github-actions bot locked as resolved and limited conversation to collaborators Mar 2, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Poetry from `install-poetry.py` does not install packages to system Python even with `POETRY_VIRTUALENVS_CREATE=0` #3870

Poetry from `install-poetry.py` does not install packages to system Python even with `POETRY_VIRTUALENVS_CREATE=0` #3870

ruohola commented Apr 1, 2021 •

edited

Loading

sdispater commented Apr 1, 2021

byteszard commented Apr 15, 2021 •

edited

Loading

ClementWalter commented Apr 20, 2021 •

edited

Loading

ruohola commented Apr 21, 2021

JamesXNelson commented Apr 29, 2021

SaulAryehKohn commented Jun 9, 2021

github-actions bot commented Mar 2, 2024

Poetry from install-poetry.py does not install packages to system Python even with POETRY_VIRTUALENVS_CREATE=0 #3870

Poetry from install-poetry.py does not install packages to system Python even with POETRY_VIRTUALENVS_CREATE=0 #3870

Comments

ruohola commented Apr 1, 2021 • edited Loading

Steps to reproduce:

sdispater commented Apr 1, 2021

byteszard commented Apr 15, 2021 • edited Loading

ClementWalter commented Apr 20, 2021 • edited Loading

ruohola commented Apr 21, 2021

JamesXNelson commented Apr 29, 2021

SaulAryehKohn commented Jun 9, 2021

github-actions bot commented Mar 2, 2024

Poetry from `install-poetry.py` does not install packages to system Python even with `POETRY_VIRTUALENVS_CREATE=0` #3870

Poetry from `install-poetry.py` does not install packages to system Python even with `POETRY_VIRTUALENVS_CREATE=0` #3870

ruohola commented Apr 1, 2021 •

edited

Loading

byteszard commented Apr 15, 2021 •

edited

Loading

ClementWalter commented Apr 20, 2021 •

edited

Loading