Support for Poetry

[mixxorz]

Poetry is a python package manager, similar to Pipenv.

At the moment, heroku-buildpack-python supports both vanilla pip (via requirements.txt) and Pipenv (via Pipfile, Pipfile.lock). Unfortunately, there's been a lot of backlash with regards to Pipenv in the Python community, and on the back of that, Poetry was born to serve as a better Pipenv.

That being said, it would be good if this buildpack can detect and install dependencies using Poetry. The package manager uses two files: pyproject.toml (which has been provisionally accepted in PEP 518) and poetry.lock.

[kennethreitz]

Perhaps a bin/install would be appropate (in the app repo), which would replace the entire pip install step with whatever logic you like!

[CaseyFaist]

Hey there! It's on the product roadmap to add experimental Poetry support - there's a user submitted PR #801 that I might be able to iterate on. I'll circle back in a few weeks with a progress update, but keep an eye on that pr for more details for now 👍

[bsdz]

I think one wouldn't need explicit poetry support but simply the latest version of pip (>19.). See python-poetry/poetry#321. Would be nice if one could easily configure which pip version is used from a buildpack runtime.txt or similar.

[rschuetzler]

It looks like #801 was closed by the user. Any update on the status of Poetry support?

[bsdz]

It looks like #801 was closed by the user. Any update on the status of Poetry support?

Not sure if helpful but another PR (by me) also adds Poetry Support

[b0g3r]

Looks like progress about feature is frozen :(

[cjolowicz]

Here is a series of PRs which implement support for Poetry and other PEP-517 compliant build backends:

• Upgrade to pip 19 #833
• Support pyproject.toml #834
• Support poetry.lock #835

[RobRoseKnows]

Curious as to the status of this. There seems to be numerous community PR to do this, but none of them are merged.

[CaseyFaist]

Thanks for checking in on the status y'all - the first of @cjolowicz 's prs (pip update to 20) has been merged and released, and fully plan on working through the last two of that trio to provide poetry support. I'll leave this issue open until resolved, but will close other Poetry feature requests so we can track progress and questions in one place.

[CaseyFaist]

Good news everyone! Prepare yourselves for a rambly update...

So, while I've been pushing out smaller improvements, this issue has been hanging around in the back of my mind. I think one of the lessons from the way pipenv support was added is that a tight coupling between pre-python install (path updates, necessary system packages, flags, more sysadmin-type setup) and post-python install (pip installs, package installs, any sort of venv setups) leads to a lot of user and maintenance friction. Especially as the package changes, grows and adapts.

We do some magic to reach into the pipenv config files and do the right thing with them in the sysadmin step, because it as a tool doesn't cause deterministic builds at the python install level, it just documents what the determined python version for the build should be. The proposed solution in #835 is good work, but follows the same convention - and, I'm concerned the convention sets us up for that same friction.

From Poetry's documentation, the solution to that is to rely on pyenv for pre-python setup, and then poetry for post-python setup, with a clearer divide between the two. This is the path forward here.

Poetry has a lot of promise and I'm very excited about it, but I don't believe Heroku should be restrictive - it should be possible to use the tooling of your choice, and it should be plug and play. So I want to be able to easily plug and play new systems as they gain ground and maturity (and may no one wait this long again 😂). That's the benefit of a strict pre/post python divide using pyenv (or at first, just a .python-version file) -- it's a simple and straightforward base to build on, whether it's for Poetry, standard pip workflow flow or other tools. The downside is that it's yet another file to be specified on Heroku, albeit a community-standard one. So while I hate to ask for duplicate specifiers or files, I think we need both.

With this decision formed, though, expect to see progress towards this speed up a bit in the next several weeks. Especially after I'm able to figure out how to record my PyCon talks from my home 😬

[alfondotnet]

Hi @CaseyFaist !!,

has there been any update regarding this?. Not trying to be pushy, just curious :-)

[edmorley]

@alfonsoperez Hi! I've just taken over from Casey as Python owner. Exploring support for alternative package managers is definitely on my list of things to do, though the overall list is pretty long at the moment, so not sure as to timeframe :-)

[nextmat]

Hi @edmorley, great to see all the things you are cleaning up!

Just wanted to check back and see if you have any sense of a likely timeline to support or if this is still a ways off?

[edmorley]

@nextmat I don't have a timeline at the moment, but this is definitely bubbling up closer to the top of the list, now some of the more urgent issues have been addressed :-)

[ipmb]

It'd be great if the buidpack could leverage pyproject.toml to make this determination. PEP-518 already provides a standard way to define a build system, but Heroku could leverage its own tool table for Heroku-specific configuration as well.

[nextmat]

In case this helps anyone else, this buildpack works as an interim solution if run first.

It exports a requirements.txt file from Poetry to allow the official buildpack to work afterwards.

[anentropic]

I agree with @ipmb ... this should be less about specific support for Poetry or Pipenv and more that any modern Python buildpack should detect a pyproject.toml file and support PEP-518 (this can replace both requirements.txt and runtime.txt)

But yes, I would like Poetry support

[edmorley]

Hi :-)

So, if I understood it correctly, the plan is to leave the classic buildpack as is for now, and provide official Poetry support for the CNB buildpack later, because it would compose well?

Yeah that's correct. Since the CNBs will eventually replace the classic buildpacks, we don't want to invest too much time in the latter - though once the CNBs exist there may be some amount of backporting to the classic buildpacks, to make the transition easier.

[Garrett-R]

Node.js and Java have a native CNB, Python is next.

I'm curious if there's any ETA on the Python native CNB. Sounds pretty cool BTW!

[edmorley]

Hi! We've been working on libcnb.rs, which will power our CNBs moving forwards. That library is now in a good shape (as of the last few weeks), so work on the Python CNB will be starting imminently.

[ulgens]

@edmorley Thanks for the update! Is there any resource-channnel we can track the progress of Python CNB implementation?

[edmorley]

I would repo watch:

• https://github.com/Malax/libcnb.rs
• https://github.com/heroku/buildpacks-python (currently empty, but once past design/research/local prototyping will start to have content added)

[edmorley]

Also, if you have opinions on ways to specify Python version, feel free to comment on:
#913 (comment)

[edmorley]

Thank you for everyone's patience here! I've finally had a chance to return to this buildpack after working on the Python CNB (the next-generation Cloud Native Buildpack, that's set to replace this one; see the Python CNB repo and the CNB preview announcement).

As of #1682 this buildpack now supports Poetry :-)

If you are using the moneymeets/python-poetry-buildpack buildpack you will need to remove that buildpack in order to use the native Poetry support, since otherwise the requirements.txt exported by that buildpack will take precedence (intentionally done this way for now for backwards compatibility).

You will also need to add a .python-version file to specify the Python version, since we don't support reading the version from poetry.lock (see the #1682 PR description for more details). Our newly added .python-version file support also supports specifying just the major Python version (eg 3.13 rather than 3.13.0) , meaning you can get automatic security updates without having to remember to bump the version every time an upstream CPython release occurs.

For all other details on the Poetry usage/behaviour, see the #1682 PR description.