Manually specifying the venv path

[viniciusd]

Is it possible to manually (via some config) specify which venv poetry is going to use?

Due to some requirements, I would like to have two projects sharing the same virtual environment.

Best case scenario would be setting the path via a config or environment variable

[thernstig]

@viniciusd Is this what you are looking for: https://poetry.eustace.io/docs/configuration/#settingsvirtualenvspath-string ?

[viniciusd]

Not exactly. This one defines where each project's virtual environment will be set.
For example, if I have settings.virtualenvs.path = /usr, and install two projects, A, and B, the first while will be located in /usr/A while the latter should be in /usr/B.
I want them to share the same virtual environment instead.

[ulgens]

pyenv solves this by .venv file. You can put your env path in it and it automatically uses it.

[finswimmer]

Hey @viniciusd ,

could you describe in which scenario two projects needs to share the same virtuell environment?

Thanks.

fin swimmer

[ulgens]

@finswimmer Other usecases are setting the name for venv or using an already existing venv for new project.

[marco-silva0000]

can this not be solved with https://poetry.eustace.io/docs/configuration/#settingsvirtualenvspath-string

[ulgens]

can this not be solved with https://poetry.eustace.io/docs/configuration/#settingsvirtualenvspath-string

Nope. It just gives a base dir for all venvs.

[finswimmer]

Is it possible to manually (via some config) specify which venv poetry is going to use?

To answer this first: AFAIK there is no way to do this (at the moment).

Due to some requirements, I would like to have two projects sharing the same virtual environment.

@finswimmer Other usecases are setting the name for venv or using an already existing venv for new project.

Mhh, I wouldn't do this. It looks to me like a bad practices. New projects should start with a fresh virtual environment to ensure only dependencies needed are installed. Different projects should have different environments to avoid any conflicts when (de)installing/updating/downgrading a package for one projects.

One use case for specifying the path to the venv I can imagine, is when you run out of space and one need to put the venv files to a different location. A workaround (I haven't tested) might be as follow:

When settings.virtualenvs.in-project (virtualenvs.in-project in poetry v1.0.0) is set to True, poetry expect the venv files in the .venv folder inside the project. One could create a symlink called .venv that point to any other location where the real files are located.

fin swimmer

[ulgens]

To be honest, i don't think this is not about a being good/bad practice. Personally, i don't want to see (foldername-8charhash-pyversion) for every venv i activated, in console prompt. Giving a user the chance to change this is about "giving the chance" :) . I still think pipenv's way to do it with .venv file is just okay.

[viniciusd]

In addition to what @ulgens mentioned, a use case would be having a shared project with shared modules and dependencies in order to guarantee the different services/projects are using the same versions of the shared dependencies.

[marco-silva0000]

on deployment jobs is also useful/common to setup the environment in stages and with more flexibility, currently I have to export to requirements.txt to then use the correct env with other tools.
It's not only about being good/bad practice, sometimes you want the minimum entropy change and gradually implement changes to the building or deploying, and having the choice is always good.

[finswimmer]

The more I think about this feature request and the more you told me about the use cases, the less I'm convinced that poetry should support is.

poetry is about managing python projects and not environments. poetry is also about best practices, as it always seeking for solutions that are already standardized or are respected as those. Also it limits its configuration to be able to have one way to do it and not many. This makes the projects highly compatible to another and on different platforms.

Being able to define the path to the virtual environment and so makes it possible that multiple projects uses it, introduces another problem: Who is responsible that the poetry.lock is always up-to-date in all projects? Who is responsible that the dependencies defined in pyproject.toml in each project are always valid.

The clean way would be, that you define a new environment on your system, where in the end all projects should play together, that have these projects as dependencies.

[ulgens]

poetry is about managing python projects and not environments

Poetry can create virtual environments, activate and deactivate them, use existing ones, but not allows to define a venv path. I'm not sure this is a "best practice".

Who is responsible that the poetry.lock is always up-to-date in all projects?

I think this is out of scope. For given usecases, it's not really important and people already have other solutions.

I feel like you're missing the point. It's not about changing the default behaviour. I can set the tool to put venv in the folder and i can create a symlink from venv folder to real environment, but what if i don't want to do it for every project? Is it a bad thing if we have that possibility?

There is some strong words about why Poetry shouldn't implement it but it's already here, just half baked.

[schinckel]

My workflow that I think could take advantage of this is using tox to run tests.

Currently, when you use poetry install inside a tox environment, it uses the virtualenv that poetry "owns" for that python interpreter. This means that it's not an isolated environment: it's probably shared with the development environment.

Being able to use a specific environment (or even the "active" environment) would remove this problem, and make running tests using tox isolated.

You can't change the settings on poetry to temporarily change the venv location, because those settings are always user-global: and there's no guarantee you are the only poetry install running at a given time, which can result in conflicts if you change settings like venv location.

[schinckel]

FWIW: I have a workaround: you can use poetry export -f requirements.txt, and then pip install from that. It's not flawless (poetry will output invalid requirements.txt at times, omitting python version markers), but that's another issue.

[JoeJasinski]

I think this would be a useful feature to specify the exact virtualenv you want to use. For example, I'm using Poetry inside of a Docker container and I'd like to specify the exact directory where the virtualenv should be created. And I don't want to use the "in-project" setting because while I'm developing with docker or docker-compose, I often like to mount in my source code into the docker container (so I can make live updates).... but I don't want to mount in the virtualenv necessarily. I'd like to exactly specify where in the container I'd like to create the virtualenv so all of my configuration and scripts can reference that location.

When I set ENV POETRY_VIRTUALENVS_PATH=/site/env/ in my Dockerfile, Poetry creates a virtualenv under that directory with a random name. That makes it hard to determine where my virtualenv is inside of the Dockerfile or with other scripts. I understand that a feature of Poetry is the ability to use more than one virtualenv (which is great), but it would be nice to specify exactly which environment to use when I know I want to use it. For example, it would be nice if there were a setting called POETRY_VIRTUALENV_PATH (no S) to specify the exact location.

Maybe there's a way to do this already - I'm still getting used to the new 1.0 features.

[garthk]

If you need a predictable PYTHONPATH for your Dockerfile, strikes me the return value of EnvManager.generate_env_name is stable for any combination of name and cwd. As long as the team keep it stable, for any particular source path you'll be able to poetry env list and derive, from the output, something like:

ENV PYTHONPATH=/opt/example/example-7rCCnWI7-py3.7/lib/python3.7/site-packages
RUN poetry env use `which python`
RUN test "$(poetry env list | cut -f 1 -d ' ')" == "example-cHSZFygF-py3.7"

The test will crash your docker build if the path drifts.

[schinckel]

Inside a docker container you can also prevent poetry from creating virtual environments: this will be local to the docker container.

RUN pip install poetry
WORKDIR /app
COPY poetry.lock pyproject.toml /app/
RUN poetry config virtualenvs.create false
RUN poetry install --no-dev -vvv

[geckon]

I'd also like to see a possibility to specify path to virtualenv manually. It's useful in docker and possibly in other use cases too.

I know that I can create the virtual env manually, activate it and then run poetry in it but it seems like unnecessary hassle considering how poetry makes my life easier in other areas.

[kkawabat]

As a novice in package management, this is my usecase where specifying the venv path would be useful. If I can install A inside B's venv then I would not need to install B twice.

https://stackoverflow.com/questions/60287564/how-to-manage-editable-shared-libraries-with-poetry

[caleb15]

@ulgens I don't see any documentation on a .venv file in https://github.com/pyenv/pyenv

How do I use it?

I tried creating a .venv file with contents /home/caleb/.cache/pypoetry/virtualenvs/fifteen5-deploy-GcwqD37l-py3.6 (the output of poetry env info under Path) but the poetry venv was not automatically loaded.

I can't find any info on how to automatically activate the poetry venv using pyenv :(

Edit: I avoided the issue alltogether by just configuring poetry to not create venvs

[CrossR]

Specifying just a specific name (without the hash) would be good enough for me, and I can't seem to find a nice way of doing that.

My use case is having a venv setup for neovim that contains some packages that my neovim plugins need. I used to just set the python_host_prog to /path/to/my/venvs/nvim-plugins-py3.7/bin/python and it worked across machines (assuming I set poetry to always install to the same location). Now I'm having to have awkwardly change it on each machine, and remember not to commit that change to my config.

I have similar issues in VSCode with Python paths.

[jagretti]

I'd also like to see a possibility to specify path to virtualenv manually. It's useful in docker and possibly in other use cases too.

I know that I can create the virtual env manually, activate it and then run poetry in it but it seems like unnecessary hassle considering how poetry makes my life easier in other areas.

Hey @geckon , I'm trying to do this exact thing (create a virtual env manually and then use poetry inside of it), but poetry for some reason is not detecting that virtual env. An important thing, I'm trying to do it in a Dockerfile. I activate the virtual env, and then I set the VIRTUAL_ENV var with the venv directory, and poetry is still creating a virtualenv on their own. Do you know how can I do this?

[geckon]

@jagretti, you should be able to do something like this...

# Create virtual environment.
RUN python -m venv "/path/in/the/container/venv"

# Activate the venv.
ENV VIRTUAL_ENV "/path/in/the/container/venv"
ENV PATH "$VIRTUAL_ENV:${PATH}"

# Install poetry into the venv.
RUN pip install poetry

# Install dependencies into the venv.
COPY poetry.lock pyproject.toml "/path/in/the/container/"
RUN cd "/path/in/the/container" && poetry install --no-dev

[geckon]

Or if that doesn't work for some reason, you can try something like:

# Create virtual environment.
RUN python -m venv "/path/in/the/container/venv"

# Install poetry into the venv.
RUN . /path/in/the/container/venv/bin/activate && pip install poetry

# Install dependencies into the venv.
COPY poetry.lock pyproject.toml "/path/in/the/container/"
RUN cd "/path/in/the/container" && . /path/in/the/container/venv/bin/activate && poetry install --no-dev

[JulianFerry]

@geckon

@jagretti, you should be able to do something like this...
...

How would you do this using the recommended poetry installation curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry. py | python?

I would still like to be able to have my poetry installation separate from my virtualenv in order to avoid installing any of poetry's dependencies without having specified them in pyproject.toml. However I would still like to be able to point to a specific virtual environment with the same setup as @JoeJasinski .

---

Edit: Actually I think I figured it out. This should install poetry as a separate installation, and then cause poetry add and poetry install to install packages to /opt/venv

RUN curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python - -y
RUN python -m venv /opt/venv
ENV PATH="/root/.poetry/bin:/opt/venv/bin:$PATH"
RUN poetry config virtualenvs.create false

[shughes-uk]

For people still wondering about use cases. I have two python projects, one is the central project and the other is a plugin for the project. I'd like have each be installed and editable in the same environment as I am often making changes to both in tandem.

In the past I would simply do pip install -e project1 project2 but that does not work without a setup.py

[schinckel]

@JulianFerry be aware that poetry settings can not be applied to just the shell or location: they are user-global.

[JulianFerry]

@JulianFerry be aware that poetry settings can not be applied to just the shell or location: they are user-global.

Thanks. I am just using this for docker builds, which means this is not an issue. But as you say, perhaps there could be a better solution for development environments.

[dylanfprice]

I have the same use case as @theirix. Here is the best workaround I've found so far to get poetry to use an explicit venv:

RUN python -m venv /venv
ENV PATH=/venv/bin:$PATH \
    VIRTUAL_ENV=/venv
COPY pyproject.toml poetry.lock /app/
# Will install into the /venv virtualenv
RUN poetry install 

[alexttx]

My use case: I have a python script (call it "driver.py") that uses its own non-poetry virtual environment to execute commands that themselves need to be run a poetry environment. I want driver.py to execute: "poetry run meson compile -C build". It fails because poetry picks up driver.py's virtual environment. As a workaround, I would like to be able to setup a named poetry virtual environment that I can refer to with poetry run. For example:

# create the named environment in  {cache-dir}/virtualenvs/my-special-env
poetry install --name my-special-env 

Then my driver script, which uses asyncio, could do something like this:

process = await asyncio.create_subprocess_shell(
    "poetry run --venv MY_SPECIAL_ENV meson compile -C build")

[dzieciou]

@geckon wrote:

I know that I can create the virtual env manually, activate it and then run poetry in it but it seems like unnecessary hassle considering how poetry makes my life easier in other areas.

This is how we solve the problem right now, but the downside of that approach that we need to maintain Python version in two places:

• when creating virtualenv, e.g. virtualenv -p python3.8 $ENV_DIR

• in pyproject.toml:

  [tool.poetry.dependencies]
  python = "~3.8"

[kukushking]

Like how @ulgens very well put it back in 2019 and there is still no resolution:

Poetry can create virtual environments, activate and deactivate them, use existing ones, but not allows to define a venv path. I'm not sure this is a "best practice".

I feel like you're missing the point. It's not about changing the default behaviour. I can set the tool to put venv in the folder and i can create a symlink from venv folder to real environment, but what if i don't want to do it for every project? Is it a bad thing if we have that possibility?

There is some strong words about why Poetry shouldn't implement it but it's already here, just half baked.

It disappoints me how such an integral feature is getting a push back from the community on no grounds.

[tybritten]

This is exactly my use case- multistage docker builds. I want to create a /venv env, use poetry to install into it, and then copy it to my final stage container. I was having poetry output the requirements.txt and installing that, but now that doesn't work due to the --require-hashes issue.

[ClayShoaf]

could you describe in which scenario two projects needs to share the same virtuell environment?

I am working with a program that allows plugins.
There is a separate project that contains a collection of plugins.
I want to run tests on those plugins using pytest or poetry run pytest (whichever one I can get to work).
The main project recently started using poetry, but the plugins collection contains plugins that still use requirements.txt files and assume the intalled dependencies of the main project.
I can't just use poetry to export a requirements.txt file because the dependencies are structured with relative paths (i.e. ../../dependency), which pip does not recognize.
It would be nice if there was some way that I could use the same venv, similar to the way pyenv has pyenv local [version number]

[havardox]

Another use case would be a Docker bind mount. When managing dependencies inside a Docker container I would want the pyproject.toml and poetry.lock files to be preserved, so I mount my project's root directory into the container. I don't want Poetry creating an environment in its current directory, because that would copy over the .venv folder to the host as well.

[kellerza]

I've used the following method with pipenv and it seems just as effective with poetry as well.

ENV VIRTUAL_ENV=/usr/venv
ENV PATH="$VIRTUAL_ENV/bin:$PATH"
RUN python3 -m venv $VIRTUAL_ENV \
  && poetry install --no-interaction --no-ansi --no-dev

Inspired by this article.

[couling]

Looks like most use cases for this (particularly docker) are covered by #108

FROM base_image as build
# I think this feature is coming in 1.2 so may not work with pip install yet
RUN --mount=type=cache,target=/root/.cache \
    curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python -

RUN --mount=type=cache,target=/root/.cache \
    PATH=/root/.poetry/bin:${PATH} poetry config virtualenvs.in-project true

WORKDIR /app
COPY /source /app/source
COPY pyproject.toml /app
COPY poetry.lock /app

# Creates a venv that with the stable path /app/.venv
RUN --mount=type=cache,target=/root/.cache \
    PATH=/root/.poetry/bin:${PATH} poetry install --no-dev


FROM base_image

COPY --from=build /app /app
ENV VIRTUAL_ENV /app/.venv
ENV PATH /app/.venv/bin:${PATH}

[johnthagen]

In case it is helpful, I have a full Dockerfile example in my example Python project that uses VIRTUAL_ENV to solve this issue, and multi-stage Docker builds to reduce the final Docker image size.

• https://github.com/johnthagen/python-blueprint/blob/master/Dockerfile

[neersighted]

(from @kellerza)

ENV VIRTUAL_ENV=/usr/venv
ENV PATH="$VIRTUAL_ENV/bin:$PATH"
RUN python3 -m venv $VIRTUAL_ENV \
  && poetry install --no-interaction --no-ansi --no-dev

(from @johnthagen)

In case it is helpful, I have a full Dockerfile example in my example Python project that uses VIRTUAL_ENV to solve this issue, and multi-stage Docker builds to reduce the final Docker image size.

* https://github.com/johnthagen/python-blueprint/blob/master/Dockerfile

This is the solution, and for those who don't follow, why this issue is considered out of scope for Poetry. Poetry already has a bail-out for users who need more complicated virtual environment management than our (intentionally minimally featured) built-in solution offers -- If Poetry detects an already activated virtual environment, it will use it instead of creating its own.

Use of VIRTUAL_ENV and PATH in a Dockerfile can 'activate' the environment for all subsequent commands, and is a very useful pattern.

[eddyg]

There are so many other use cases than Docker that have been thoughtfully explained in the comments above. (There are probably hundreds of 👍’s on comments not related to Docker.)

Clearly this feature is important to a lot of people, so it is very disappointing to see it closed.

Even if there are “workarounds”, having a built-in option to help deal with “more complicated virtual environment management” means that much less manual tooling, training, debugging, etc. for every user of Poetry in that situation.

[joshkarlin]

this would be a nice feature to have and clearly people want it. would be convenient to not have to use other tools for a workaround

in case anyone finds this useful [apologies if it has already been mentioned and i missed it above], here is the workaround I use for activating docker venvs:

RUN echo "source $(poetry env info --path)/bin/activate" >> ~/.bashrc

if you only want to store the path then you could export it as an environment variable in your Dockerfile:

RUN echo "export VIRTUAL_ENV=$(poetry env info --path)" >> ~/.bashrc

note that these commands need to be run from the same dir as your pyproject.toml to know which env you want.

main advantage of the above approach is sticking with just poetry rather than using venv directly too. arguable whether that's much of an advantage vs the python -m venv $VIRTUAL_ENV approach 🤷 end result is the same: you have an env var which tells you where your venv is and can activate it as and when you see fit

hope that's useful for someone

[neersighted]

You should not depend on .bashrc in a Dockerfile -- if your base image has a different shell (or sets defaults differently) it will not work. If you run a command without a shell (e.g. RUN [...] or CMD [...]) it will not work.

In any case, in a container, it makes much more sense to control paths and be explicit, which not only makes your image build more understandable to other contributors, but can improve your layer caching strategy.

---

There seem to be two schools of thought: Those that think that Poetry is a composable tool that tries to make Python development as easy as possible while being interoperable, and those who want to replace the entire Python tooling ecosystem with Poetry.

The thoughts of the maintainers tend towards the middle, leaning towards interoperability when there is a perceived conflict. Poetry is aiming to improve the common use case while not inventing a parallel universe where it makes decisions incompatible with other tools, or where you are forced to use Poetry exclusively and not integrate it into existing workflows.

I do think our unobtrusive virtual environment management is a good example of striking this balance -- the simple case is abstracted for users, and the complex cases are easily solved by the rich tools that focus on virtual environment management. Cases in the middle, like containers, benefit from the standard tooling shipped with/maintained by the core Python project that all Python developers should be familiar with.

[joshkarlin]

I agree that it would be nicer to do this in dockerland by controlling the path, which would be easier if poetry were to allow us to specify a venv path. as you said, the .bashrc approach does not activate the venv inside the Dockerfile so poetry run ... has to be used for any RUN/ENTRYPOINT/CMD commands which need the environment - not ideal but works. once the image is built the env is available as desired.

your general position outlined above is sensible - poetry can't be expected to do everything. however, i do think that turning down a feature request to allow users to specify a venv name/path is a little inconsistent with the other features poetry offers. why draw the line at providing a name for the directory where poetry will create a venv? poetry already creates and names venvs, why would this qualify as obtrusive?

the cli has the following option:

env use            Activates or creates a new virtualenv for the current project.

adding a --name option doesn't seem like it would rupture spacetime. what am i missing?

[cpbotha]

poetry 1.2.2 refuses using the existing virtual environment with the suggested solution:

ENV VIRTUAL_ENV=/opt/venv
ENV PATH="$VIRTUAL_ENV/bin:$PATH"
RUN python3 -m venv $VIRTUAL_ENV \
  && poetry install --no-interaction --no-ansi --no-dev

With virtualenvs.create true, it just goes and creates a virtualenv in its cache directory. With virtualenvs.create false it detects virtual environment /usr and then errors out because I'm doing this test as a normal user.

So far, I have found no way to get 1.2.2 to use an existing environment created by venv.

[clintonroy]

@cpbotha appending to a closed ticket is a good way of not getting help.

Your snippet is identical (in thrust) to what I do at work.

Step 10/10 : RUN python3 -m venv $VIRTUAL_ENV   && poetry install -vvv
 ---> Running in 4ee946229dc8
Using virtualenv: /opt/venv
Project environment contains an empty path in sys_path, ignoring.
Updating dependencies
Resolving dependencies...
   1: fact: test is 0.1.0
   1: derived: test
   1: selecting test (0.1.0)
   1: Version solving took 0.006 seconds.
   1: Tried 1 solutions.

Writing lock file

Please, always give outputs, error messages, or really clear descriptions of what goes wrong.

[cpbotha]

I am very curious to hear how you did that with 1.2.2.

See below for a small test, first inputs then outputs, with virtualenvs.create = false, where poetry 1.2.2 (installed via https://install.python-poetry.org - this might be part of my problem) did not want to use the venv:

➜  test-project poetry --version
poetry config --local virtualenvs.create false
VIRTUAL_ENV=/tmp/venv
PATH=$VIRTUAL_ENV/bin:$PATH
python3.10 -m venv $VIRTUAL_ENV
poetry env use $VIRTUAL_ENV/bin/python3.10
poetry install -vv

Poetry (version 1.2.2)
Skipping virtualenv creation, as specified in config file.
Using virtualenv: /usr
Skipping virtualenv creation, as specified in config file.

  Stack trace:
   1  ~/.local/share/pypoetry/venv/lib/python3.10/site-packages/poetry/core/version/pep440/parser.py:83 in parse_pep440
        return PEP440Parser.parse(value, version_class)

  InvalidVersion

  Invalid PEP 440 version: '0.23ubuntu1'

➜  test-project which python
/tmp/venv/bin/python

BTW, I posted here for others to find, and to discuss, because this is the ticket that comes up about this specific topic of manually specifying the venv path.

[finswimmer]

@cpbotha I would suggest that you join us on discord for further discussions on that topic. This issue here is closed.

[cpbotha]

In my case, installing poetry into the /opt/venv environment with pip, instead of using the self-contained installer, did the trick!

@finswimmer Thank you very much for the offer! I like discord, but when I searched for a solution to the issue I was seeing, THIS issue was the main and most important hit. My reasoning is that others who search for similar solutions will also find this thread, so it makes sense to have helpful info here as well.

[finswimmer]

My reasoning is that others who search for similar solutions will also find this thread, so it makes sense to have helpful info here as well.

I understand and appreciate it 🙏 But you came to the point where something doesn't work for you but works for @clintonroy . So finding out what's going on shouldn't be part of this (closed) issue here.That's better done on discord or a separate issue. Coming back after everything is resolved and provide a solution is fine than.

[ulgens]

@finswimmer If you consider how many people are requesting this feature with valid use cases and how weak the explanation about why it shouldn't be implemented is, having multiple people commenting on a "closed" issue may be an indicator of that issue shouldn't be closed.

[neersighted]

Discussion on this issue has gone wildly off topic -- the proposed feature of specifying the path Poetry uses for it's built-in environment management (both in-project and otherwise) has been declined for now. That could be revisited in the future, but the core team is currently opposed to increasing the surface area of our (already difficult to maintain/in need of rework) environment management, as there have been no use cases presented not equally achievable (or even more easily achievable) with standard Python tooling, aka the venv module.

This issue is going off topic, so I am going to lock it for now as it's attracting support questions. In general, if you have a support question, please create a Discussion or join Discord; if you have done through troubleshooting and think you have identified a bug, please open a new issue. Overloading issues with multiple discrete variations or "me too" comments rarely does anything but add noise -- if something is a unique bug, we need to be able to track it and reason about it separately.

Thanks all -- and again, feel free to open Discussions or ask on Discord about this topic. The goal here isn't to discourage discussion, but to make it clear that this is currently rejected as a feature -- you're free to talk about it elsewhere and advocate for it, or even propose a PR (though, unless it does something novel not already discussed, it's likely not going to be accepted at this time).