Poetry 1.8 change regarding virtualenvs behaviour (failing ReadTheDocs builds)

[ewjoachim]

• Poetry version: 1.8.0
• Python version: 3.10
• OS version and name: Ubuntu 22.04
• pyproject.toml: https://github.com/procrastinate-org/procrastinate/blob/ff9b7c3584f823e40d35544fd62c8e6b03f9c048/pyproject.toml

• I am on the latest stable Poetry version, installed using a recommended method.
• I have searched the issues of this repo and believe that this is not a duplicate.
• I have consulted the FAQ and blog for any relevant entries or release notes.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option) and have included the output below.

Issue

This issue relates to a problem with Poetry and ReadTheDocs (RTD).
The official RTD integration doc advocates for running the following commands:

$ pip install poetry
$ poetry config virtualenvs.create false
$ poetry install --with docs

In the environment this runs in, there is a virtualenv created with python -mvirtualenv $READTHEDOCS_VIRTUALENV_PATH and $READTHEDOCS_VIRTUALENV_PATH/bin is added in the PATH as the first element, though VIRTUAL_ENV is not set.

• In 1.7, poetry install added packages to the $READTHEDOCS_VIRTUALENV_PATH venv.
• In 1.8, packages are added to the underlying Python env, not in the virtualenv.

Consequently, packages added this way are not seen by subsequent steps, and the builds fail.

See Discord discussion at https://discord.com/channels/487711540787675139/487711540787675143/1211368925158183004

See RTD corresponding issue: readthedocs/readthedocs.org#11150

[dimbleby]

sounds like you have been relying on a bug

poetry config virtualenvs.create false

recommend not doing that.

Seems like you already have a virtual environment: activate it properly - including setting the environment variable - so that poetry can detect it

[abn]

As discussed on discord, I suspect the correct fix here is to activate the virtual environment correctly within the build environment as @dimbleby mentioned.

In particular, replace this.

poetry config virtualenvs.create false

With the following.

export VIRTUAL_ENV=$READTHEDOCS_VIRTUALENV_PATH

For my own reference here is a simplified reproducer.

podman run --rm -i --entrypoint bash docker.io/python:3.12 <<EOF
set -xe
export PATH="/root/.local/bin:\${PATH}"
OLD_PYTHON=\$(which python)

# not usually recommended
python -m pip install --disable-pip-version-check -q poetry

# do not do this unless there is a REAL reason for it
export POETRY_VIRTUALENVS_CREATE=false

# create a global virtual env
python -m venv global
NEW_PYTHON="\$(realpath global)/bin/python"

# fake activiate it (this is bad practice)
export PATH="\$(realpath global)/bin:\${PATH}"

poetry new demo
pushd demo
poetry add cowsay

# if active python is ignored and system python gets used the package
# appears here
\${OLD_PYTHON} -m pip show cowsay || :

# if active python is correctly detected the package should be here
\${NEW_PYTHON} -m pip show cowsay || :
EOF

[ewjoachim]

For the record, those are the new instructions in the RTD docs:

version: 2

build:
  os: "ubuntu-22.04"
  tools:
    python: "3.10"
  jobs:
    post_create_environment:
      - pip install poetry
    post_install:
      - VIRTUAL_ENV=$READTHEDOCS_VIRTUALENV_PATH poetry install --with docs

[ewjoachim]

Ok, the snippet above has been merged in the RTD docs as "this is the way". I'll let you decide whether you want to close this issue, leave it open for a few week for visibility or what, but I'm ok with the state of things so far.

[abn]

@ewjoachim great work. Thank you.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.