Add support for environment variables at pyproject.toml

[loop0]

It would be handy to have support for using values from environment vars on pyproject.toml
The use case I have is that I need to use different repository urls depending on my environment.
Something like:

[[tool.poetry.source]]                                                          
name = "private"                                                          
url = "${PRIVATE_REPO_URL}"

I'm still new to poetry, but I could help implementing the feature with some guidance.

[jordanhamill]

I too would like to not check in a private repo url. Perhaps in this case, the value from config.toml could be used for the repository?

The docs already mention adding a repository url for pushing. If url was missing from source you could check for the following key:

f"repositories.{source.name}.url"

I pushed up a spike 290e886 to see if it would work and it looks like it does, I'm not sure if it is the correct solution though.

My suggestion becomes a little cumbersome when building docker images. Currently I'm doing everything on one line so nothing sensitive is cached in a layer:

ARG PYPI_USERNAME
ARG PYPI_PASSWORD
RUN poetry config http-basic.my-repo ${PYPI_USERNAME} ${PYPI_PASSWORD} && poetry install ${POETRY_INSTALL_OPTIONS} && poetry config --unset http-basic.my-repo

[kofron]

@loop0 did you take a stab at this? I'd love to see this implemented and have some ideas around how to do it.

[loop0]

Not yet, I was planning to tackle this next week as I am starting to use poetry again. Do you mind sharing your ideas?

[kofron]

Awesome, great to hear - let me know if I can help at all.

Basically what I was imagining is a new function (or class) that lives in poetry.util.helpers called environment_expander or similar.

The basic idea is that after pyproject.toml is read from disk, and after we know that it exists, we take a transformation pass over the TomlFile object, and applying a function to all of the values in the object that evaluate expressions of the form ${SOME_ENV_VARIABLE} by calling os.env.

I would want to apply this pass as early as often - at poetry/puzzle/provider.py:161 for example.

Not sure if you would want to raise if SOME_ENV_VARIABLE didn't exist...

Thoughts?

[loop0]

Sounds a great idea! And of course you can help, I will try to make some time tomorrow to write the first code for it. If you feel you want to write it first just mark me on the PR and I can also help you later.

[dimkirt]

Any update on this one?

[kofron]

@dimkirt #481 - was closed a couple of days ago by the author, who doesn't want to add this feature.

[Peque]

Pipenv implements this feature. As they mention, it is "quite useful if you need to authenticate to a private PyPI". That is my case too.

@sdispater Being ranked as the 11th most popular issue in this repository (out of 400+), would you reconsider accepting pull-requests for this? 😇

[metrofun]

For us it's also a blocker to finally migrate from Pipenv

[loop0]

We need this feature to migrate to poetry from pipenv as well, that's why I initially open this issue. Unfortunately I didn't have much time to work in the solution, but as far as I remember the code, it shouldn't be hard. We only need the bless from poetry's developers.

[kofron]

I did actually do this work in this PR: #481

However, @sdispater did not want to add this feature to the project.

[Peque]

Let us hope they eventually change their mind and reconsider adding this to Poetry. 😇

[absassi]

Although this solution would solve the problem, and it's the same solution as Pipenv, it's only good for CI, but terrible for humans. Once you put environment variable names in pyproject.toml, every developer must somehow export the variables to use Poetry, they can't just use poetry config http-basic anymore.

It would be much better if Poetry recognized specific environment variables as credentials (such as POETRY_USERNAME and POETRY_PASSWORD or maybe POETRY_REPONAME_USERNAME, to support different credentials for different repositories), like twine does, for instance. Then the environment variables are needed only for CI and humans can do whatever they want in their machines.

But this is not something you can do with an external program, it's something that needs to be implemented in Poetry.

[delphyne]

@absassi I disagree. Any large enterprise already has a set of consistent environment variables every developer must export. These are used across many languages and toolsets (pipenv, cake, gradle, maven, ant, node, yarn, docker, terraform, etc.). Adding a new set of environment variables is just as bad as using whatever people want.

Maybe just compromise and do both. look for some poetry-specific variable, but also allow environment variables to be interpolated like just about every other build tool that exists today.

[stale]

Is this still relevant? If so, what is blocking it? Is there anything you can do to help move it forward?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[Peque]

Still relevant.

Maybe @loop0 has some updates on this? 😊

[loop0]

No updates, sorry. We still want to migrate from pipenv to poetry here, but this issue is a blocker for most projects.

[loop0]

For instance, there is a way to use poetry with gemfury's repository, but the documentation is not clear, here are the steps:

poetry config repositories.your_repo https://pypi.fury.io/your_repo/
# at the auth configuration you just repeat your token twice
poetry config http-basic.your_repo YOUR_TOKEN YOUR_TOKEN

And then you add the following to your pyproject.toml (after the [tool.poetry] block)

[[tool.poetry.source]]
name = "your_repo"
url = "https://pypi.fury.io/your_repo/"

So with some scripting it is possible to have this running on your CI

[adawalli]

For instance, there is a way to use poetry with gemfury's repository, but the documentation is not clear, here are the steps:

poetry config repositories.your_repo https://pypi.fury.io/your_repo/
# at the auth configuration you just repeat your token twice
poetry config http-basic.your_repo YOUR_TOKEN YOUR_TOKEN

And then you add the following to your pyproject.toml (after the [tool.poetry] block)

[[tool.poetry.source]]
name = "your_repo"
url = "https://pypi.fury.io/your_repo/"

So with some scripting it is possible to have this running on your CI

While this is true, if you are using docker, you run the risk of caching these credentials, which is obviously not good. As it stands, it's not really possible (without jumping through some serious hoops) to write a 12 Factor App (https://12factor.net/) with Poetry + Private Repos without better Environment Variable support.

[sandroden]

I want o add another reason for using variable interpolation in pyproject.toml. When you add a dependency on locally stored projects as in:

[tool.poetry.dev-dependencies]
thx-sphinx = {path = "../thx-sphinx"}

you're making strong assumptions on where people place their files that is definitely not a nice behaviour. Much better to set a default that users can overwrite. A .env is a typical solution in docker environment and not only (ie: autoenv). When you enter a directory the .env is read and configured correctly. This would make poetry flexible enought for normal workflow.

[papagala]

This would be incredibly convenient for CI/CD pipelines and for reusing already available private repository urls in the form of environment variables.

[absassi]

@sandroden, depending on what you want to accomplish, you might be interested also in the feature request #1168, which can be an alternative to path dependencies.

[bubthegreat]

@sdispater - Any reconsideration here? This is difficult when I want to test anything that needs environment variables but I can't set and unset them without editing the virtualenv activate file directly, which is a huge hassle. It makes it hard to test in multiple packages or environments that might need environment variables set, like DB_HOST, etc.

[funkyfuture]

the system administrator in me likes that. but apart from it, it would lead to a situation where poetry.lock wouldn't be reproducible without sharing all environment variables. and then, what would be the point?

besides, i'm missing defaults being mentioned in this thread for which Docker-Compose has a good syntax, as an example.

[bubthegreat]

I'm doing my own workarounds by having an .env file and pulling them from there, but my ideal world would be setting them locally in the virtualenv and adding them as checks in poetry.lock with a warning if they're not set.

[jacebrowning]

Assuming the following environment variables are set:

export EXAMPLE_PYPI_URL=https://pypi.example.com/pypi/
export EXAMPLE_PYPI_USERNAME=example
export EXAMPLE_PYPI_PASSWORD=example

This is how I handle uploads to a private PyPI server:

poetry build
poetry config repositories.example ${EXAMPLE_PYPI_URL}
poetry config http-basic.example ${EXAMPLE_PYPI_USERNAME} ${EXAMPLE_PYPI_PASSWORD}
poetry publish --repository example

I can see why adding environment variable expansion to pyproject.toml could be too much of a scope creep for Poetry.

[alekseysavitskiy]

@Li-ReDBox check this:

poetry/poetry/config/config.py

Line 113 in 22c3aad

if self._use_environment:

[Li-ReDBox]

@alekseysavitskiy, my understand is that reading env variables are defined for class Config. Could you please help me to understand if there is no configuration has been set, say I do not have the setting_name in the call in my configuration, will get be called? In my current testing setup, I have not run any poetry config.

My gut feeling is that you can set a config key by running poetry config, then the value of this key can be overridden by an environmental variable later.

Without config, env variables have not worked in my tests. I may have missed something.

[reitzig]

FWIW, here's another use case that doesn't work with poetry config at all: injecting version numbers at build time.

This is popular in my current environment (Java shop; they have Maven do that). You'd also want to do that if you publish on tag builds.

[sisp]

@reitzig Specifically for your example use case "publish on tag builds", you can set version = "0.0.0" in pyproject.toml and when a CI job runs for a tagged commit, you can run poetry version $CI_COMMIT_TAG (or however your CI service provides the tag) in the CI job before building the package.

[akvadrako]

@reitzig Specifically for your example use case "publish on tag builds", you can set version = "0.0.0" in pyproject.toml and when a CI job runs for a tagged commit, you can run poetry version $CI_COMMIT_TAG (or however your CI service provides the tag) in the CI job before building the package.

That only makes sense if building as part of a CI workflow. In our case we use the git tag as the version but do those builds locally. This doesn't work very well with poetry since updating pyproject.toml after creating a tag then makes the checkout dirty.

[nikhiljohn10]

I have added one feature request #4458 where this issue is addressed in regards to the implementation part.

[Greedquest]

I would like this because mypy defines a $MYPY_CONFIG_FILE_DIR environment variable which I would like to access in my tooling setup.

[tool.mypy]
mypy_path = "$MYPY_CONFIG_FILE_DIR/stubs"

This works in mypy.ini but not pyproject.toml

[kraczak]

I am also interested what is best practice to install libraries from Arifactory using poetry. Is there any guide on that?

[timorkal]

For me this could help in the following scenario:
I want my pyproject.toml to behave differently in my ci pipe (dev vs prod).
For example I want for dev ci, the pyproject.toml will have this dependency:
xxxxxxxxx = { version = "^1", allow-prereleases = true }
While in the prod ci, the dependency will look like this:
xxxxxxxxx = { version = "^1", allow-prereleases = false }

So if I could use some ENV_VAR like $ALLOW_PRERELEASES configured in the toml file like this:
xxxxxxxxx = { version = "^1", allow-prereleases = $ALLOW_PRERELEASES }
I could easily handle it.
Now I need to use commands like sed, to overcome this obstacle.

[ghost]

For code artifact this most certainly doesn't work.

The entire conceit behind the environment variables is to avoid adding sensitive information such as one's account number or domain name. I've read the solutions above but they still stick that sensitive info in the pyproject.toml, which would compromise security upon commit to a repo.

[adamwojt]

This seems like a useful thing. For example it enables to have this:
company-common = {path = ${LOCAL_PATH_TO_REPO}, develop = true}
This can be left commented and in VCS and people in company can un-comment it to switch to development mode.

[neersighted]

This is explicitly out of scope for Poetry -- pyproject.toml is declarative, and it's not just Poetry that makes it that way. With support for PEP 621 this will be a hard constraint and not just a Poetry-enforced decision, but it's also worth noting that allowing for any sort of environment interpolation in pyproject.toml will break the ability of your project to be understood and installed by most other tools which likely will not have the correct variables available.

It's better to determine what specific functionality people are looking to compute dynamically and look at them individually -- many can be implemented as plugins or small standalone features.

[rossamurphy]

have the poetry developers seen sense yet and decided to implement this feature that everyone is crying out for? or are we still waiting for the pedantic arguments to end?

[bubthegreat]

This is explicitly out of scope for Poetry -- pyproject.toml is declarative, and it's not just Poetry that makes it that way. With support for PEP 621 this will be a hard constraint and not just a Poetry-enforced decision, but it's also worth noting that allowing for any sort of environment interpolation in pyproject.toml will break the ability of your project to be understood and installed by most other tools which likely will not have the correct variables available.

It's better to determine what specific functionality people are looking to compute dynamically and look at them individually -- many can be implemented as plugins or small standalone features.

How does that PEP block anything? There's a section that's specifically for configuring your tools:
https://packaging.python.org/en/latest/specifications/pyproject-toml/#arbitrary-tool-configuration-the-tool-table

The [tool] table is where any tool related to your Python project, not just build tools, can have users specify configuration data as long as they use a sub-table within [tool], e.g. the flit tool would store its configuration in [tool.flit].

Why would we not leverage the [tool] section for configurations like this? It supports arrays, lists, and other mixed values that can be dumped into the os.environ during poetry run for things like unit tests, or it could be injected into the virtual environment - it's been years that folks have been requesting this - are the devs at least open to PR proposals for this?

[rterbush]

Not understanding why this is closed...

Simple use case:

1. Team of poetry users wants to be able to install a dependency in a local directory and each team member has that local dependency at a different PATH on local disk.

Solution seems to be that each member sets an environment variable where that code is located on their local disk so that the pyproject.toml can just specify foobar = {path = $MODSRC}

[scottrbrtsn-ata-llc]

😩

The number of times I look for the thing I want and find something like this.

[scottrbrtsn-ata-llc]

.envrc and direnv seem like a good alternative tho.

[danb27]

please?

[neersighted]

Why would we not leverage the [tool] section for configurations like this? It supports arrays, lists, and other mixed values that can be dumped into the os.environ during poetry run for things like unit tests, or it could be injected into the virtual environment - it's been years that folks have been requesting this - are the devs at least open to PR proposals for this?

.envrc and direnv seem like a good alternative tho.

You both are misunderstanding the issue, which is asking for environment variable interpolation in pyproject.toml, and not environment variable declaration for e.g. poetry run.

In any case, PEP 621 does preclude this, unless we fall back to dynamic and the old project format, which I think we are unlikely to extend at this time.

[scottrbrtsn]

Ah, I see the distinction. Sorry for the spam.

[loop0]

Yeah, I opened this ticket in the past because I was trying to solve the authentication problem with private repositories during CI/CD. And I was able to achieve that in a different way, as I wrote in here:
#208 (comment)

[danb27]

Yeah, I opened this ticket in the past because I was trying to solve the authentication problem with private repositories during CI/CD. And I was able to achieve that in a different way, as I wrote in here:
#208 (comment)

Am I missing something, or does this still not end up with your private repository URL in both pyproject.toml and poetry.lock?

I have figured out how to push to and download from my private repository, but not without having the url committed in at least one of these files (but again, could totally be missing something).

I also understand why poetry might not want to implement this feature. It is just a shame.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.