Upgrading Python can break Poetry

[TallGirlVanessa]

Steps to reproduce

1. Use this buildpack plus heroku/python to deploy a Python project to Heroku.
2. Use Heroku CI.
3. Include poetry commands in the CI checks:

.local/bin/poetry check
.local/bin/poetry lock --no-update --check

4. Upgrade to Python 3.11.1

[tool.poetry.dependencies]
python = "3.11.1"

Expected behavior

Poetry installs and generates a requirements.txt and runtime.txt, Heroku installs Python version 3.11.1 and the requirements.txt packages, Poetry's checks succeed.

Actual behavior

Poetry installs and generates a requirements.txt and runtime.txt, Heroku installs Python version 3.11.1 and the requirements.txt packages, Poetry's checks fail with an error like this:

Python path configuration:
  PYTHONHOME = '/app/.heroku/python'
  PYTHONPATH = '/app'
  program name = '/app/.local/share/pypoetry/venv/bin/python'
  isolated = 0
  environment = 1
  user site = 1
  import site = 1
  sys._base_executable = '/app/.local/share/pypoetry/venv/bin/python'
  sys.base_prefix = '/app/.heroku/python'
  sys.base_exec_prefix = '/app/.heroku/python'
  sys.platlibdir = 'lib'
  sys.executable = '/app/.local/share/pypoetry/venv/bin/python'
  sys.prefix = '/app/.heroku/python'
  sys.exec_prefix = '/app/.heroku/python'
  sys.path = [
    '/app',
    '/app/.heroku/python/lib/python310.zip',
    '/app/.heroku/python/lib/python3.10',
    '/app/.heroku/python/lib/python3.10/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00007ff6ee928480 (most recent call first):
  <no Python frame>

Hypothesis

Poetry is installed into its venv using the Heroku default Python for heroku-22 which is 3.10.9. Once heroku/python switches to 3.11.1, the 3.10.9 installation is deleted, which breaks the Poetry venv.

Suggested solution

In this buildpack, use pyenv to install an isolated Python that Poetry can use and rely on. Related to the discussion here: heroku/heroku-buildpack-python#796 (comment)

I'm not sure how to do this in a way that plays nicely with Heroku's caching and stuff. I haven't made a buildpack of my own before.

Workaround

Do not upgrade to 3.11.

[zyv]

In this buildpack, use pyenv to install an isolated Python that Poetry can use and rely on. Related to the discussion here: heroku/heroku-buildpack-python#796 (comment)

I'm not sure how to do this in a way that plays nicely with Heroku's caching and stuff. I haven't made a buildpack of my own before.

This buildpack wasn't designed with making Poetry useful after it's finished in mind, but I admit that it was never made explicit. It merely does an intermediate processing step of converting Poetry lock into requirements.txt & runtime.txt, so that Python applications using Poetry can be deployed to Heroku using standard machinery. We aren't using Heroku CI, so it actually never occurred to us, that someone might interpret the goal of the buildpack of providing a useful Poetry installation for CI purposes.

Solving this problem would indeed require a separate Python installation as you suggest, so that Heroku doesn't pull stuff under our feet. This is not something we are interested in addressing. But maybe we can update the README to make it clearer. Any suggestions?

To solve your problem I would actually suggest to follow the recommended Poetry installation steps inside the CI environment:

export POETRY_HOME=/app/poetry
curl -sSL https://install.python-poetry.org | python3 -
$POETRY_HOME/bin/poetry --version

https://python-poetry.org/docs/#installing-with-the-official-installer

This way you should be able to land with a useful Poetry installation for your CI purposes.

[TallGirlVanessa]

That makes a lot of sense, we'll do that! I would say something like

Note: this buildpack is only for generating requirements.txt and runtime.txt for subsequent use by heroku/python. Do not depend on the installed Poetry's location, or its venv remaining functional at app runtime. If you need Poetry for other purposes (for example CI checks), make your own separate Poetry installation for those purposes.

[zyv]

@vlmphipps thank you!