Add possibility to install dependencies to target directory. pip -t flag analogue

[mvoitko]

• I have searched the issues of this repo and believe that this is not a duplicate.
• I have searched the documentation and believe that my question is not covered.

Feature Request

Add possibility to install dependencies to target directory. pip -t flag analogue.

The proposed command example is

poetry install -t ./vendor

[finswimmer]

Hey @mvoitko ,

could you please add a use case, where you think this feature would be useful?

fin swimmer

[RobbieClarken]

could you please add a use case, where you think this feature would be useful?

I'm interested in this feature to be able to download and build python packages as part of deploying to AWS Lambda. In order to deploy to Lambda you need to zip up your source code and dependencies and upload them to an S3 bucket. My build currently involves doing:

poetry export -f requirements.txt --without-hashes > requirements.txt
pip install -r requirements.txt -t ./bundle/

The --without-hashes is needed or I get an error when I pip install:

ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    setuptools>=39.2.0 from https://files.pythonhosted.org/packages/a0/df/635cdb901ee4a8a42ec68e480c49f85f4c59e8816effbf57d9e6ee8b3588/setuptools-46.1.3-py3-none-any.whl#sha256=4fe404eec2738c20ab5841fa2d791902d2a645f32318a7850ef26f8d7215a8ee (from cairocffi==1.1.0->-r requirements.txt (line 1))

It would be nice to be able to run something like:

poetry install -t ./bundle/

This would have the advantage of verifying hashes, simplifying the build and removing the direct dependency on pip.

[autoferrit]

This could also help with caching dependencies in building containers as well I think. #1382

[beanaroo]

We would also like to bundle an application for:

• AWS Lambda
• Docker Containers

It looks like there is an open Pull Request for this feature here: #799

[KentonParton]

Similarly to @beanaroo, I too would like to be able to install packages to a specific directory for AWS Lambda deployments and Docker Containers. It would be awesome to have this added to Poetry.

[beanaroo]

It appears the latest relevant PR is #2682

[Kaisbn]

Any update on this issue?

The PR doesn't seem to be moving. As mentioned above, it would be nice to have this feature especially for Lambda deployments.

[invidian]

It also seems some packages in Arch Linux like https://archlinux.org/packages/community/any/yubikey-manager/ started using poetry as build tool, which makes it difficult to upgrade with lack of ability to install project files into the target directory while building package.

[StampixSMO]

Any update on the draft PR?

[RoelantStegmann]

Any updates on this?

[nhuray]

Clearly something helpful for bundling Lambda and containers.

One thing also important is to be able to bundle only production dependencies (not dev).

Any news ?

[ehiggs]

A lot of people are explaining it's useful for bundling lambda and containers but it's might be helpful to explain why that is.

Regarding containers, Docker has a concept of a multi-stage build. In a multistage build you can have a builder image and a live image in the same file. The builder image has compilers for cython and so on installed. The live one has none of these build dependencies installed and only has the resulting binaries.

So the motivating example for poetry is to install all the dependencies to a specific directory and then, copy the files from the build image into the live image. Without this feature, we need to do surgery on /usr/local/lib or we need to use a venv. A venv in a container is like a hat on a hat.

Thanks!

[tsz662]

Any update on this? The below may be the simplest way to bundle an application with all dependencies into an AWS Lambda package but if poetry natively support an equivalent feature, that would be great.

$ poetry build --format wheel
$ pip install ./dist/foobar.whl -t ldist
$ cd ldist && zip lambda * -r

[AntoninGrele]

I'd like to bring another user case. I have setup a git hook on post-merge to trigger a django server restart when code is merged to the repository's master branch. In the git hook, Things like django collectstatic and django migrate are run, so that new static files are collected, and new migrations are applied.

It would be useful to also be able to also automatically install new packages, and this requires targeting a different directory than the one the post-merge hook is on. I'd need to be able to install the new packages from /home/username/....

[yogevyuval]

Does anyone know what is the current status of this? It's the last item preventing many people from migrating to poetry

[Iain-S]

This would also be useful for deployment as an Azure Function app.

[clintonroy]

I'm not familiar with the target environments, but is using pip itself viable?
I can get pip install . -t /tmp/foobar installing everything to /tmp/foobar, in a pyproject.toml based project,
by leaning on the [build-system] stanza.

[alex-todorov]

I'm also looking to get that feature, specifically I'll be using it for AWS Lambda

[stevekrasinsky]

Another vote for the AWS Lambda use case.

[sprckt]

+1 For the Lambda use case

[Torwalt]

Another solution that worked for me is installing the dependencies directly into the project. This will create a .venv folder which can be used to create a zip file.

 poetry config virtualenvs.create true
 poetry config virtualenvs.in-project true
 poetry install --no-dev
 # function handler file for AWS lambda
 cp lambda_function.py .venv/lib/python3.9/site-packages/
 cp -r ./YOUR_APPLICATION_ROOT .venv/lib/python3.9/site-packages/
 # set permissions so the aws runtime can execute the files
 find .venv/lib/python3.9/site-packages -type f -print0 | xargs -0 chmod 644
 find .venv/lib/python3.9/site-packages -type f -print0 | xargs -0 chmod 755
 cd .venv/lib/python3.9/site-packages/
 zip -q  -r lambda.zip ./*

[camfulton]

+1

[Cajuteq]

Hi, having followed this issue and other related (#1174), I've found that related pr (#799 and #2682) have been closed in favor of an official bundle plugin (https://github.com/python-poetry/poetry-plugin-bundle)
I didn't tested it yet and don't known if it fixes all the problems raised in this issue but perhaps someone like @finswimmer or @sdispater could clarify and maybe close this issue.

edit: just saw plugins are coming with 1.2.0 which is not yet released

[nitsujri]

@Cajuteq thanks for linking the plugin bundle. Please forgive my lack of knowledge - pretty new to python.

It looks like it's designed to only work with virtual envs? Specifically there's no way to bundle directly into the root python environment? For deployment, I don't want to mess with venvs.

Based on this thread my current deployment is:

RUN curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/install-poetry.py | python -
RUN /root/.local/bin/poetry config virtualenvs.create false

# We copy just the requirements.txt first to leverage Docker cache
COPY poetry.lock pyproject.toml ./

# https://github.com/python-poetry/poetry/issues/1937
RUN /root/.local/bin/poetry export --format requirements.txt --without-hashes --output requirements.txt
RUN pip3 install -r requirements.txt --target "${LAMBDA_TASK_ROOT}"

Above skips venv making this mentally easy for deployment.

My question becomes - is using the plugin as simple as poetry bundle venv ${LAMBDA_TASK_ROOT} and I don't have to worry about venvs? Are venvs simply folders added to a scoped $PATH for easy searching or is there more that allows python to find packages?

[velovix]

I maintain a plugin that adds Poetry support for a build tool used in the ROS community. It requires that Python dependencies be installed to a specific location. The bundle plugin ended up being a perfect solution to this problem, and I was able to replace my old poetry export + pip install -r requirements.txt workflow with it. Thanks!

@nitsujri Without getting too into how virtual environments work, you may be able to get away with setting the PYTHONPATH environment variable to point to the libraries in the virtual environment. So it would be something like:

RUN poetry bundle venv /my_project/my_venv
ENV PYTHONPATH="/my_project/my_venv/lib/python3.x/site-packages:${PYTHONPATH}"

That should allow you to use the libraries in the virtual environment without having to enter it. Let me know how it works for you.

[neersighted]

Closing as out of scope for the main project / covered by the bundle plugin. bundle could easily be extended with a prefix type that simply installs to a prefix, pip install -t style.

[Duncan-Haywood]

+1 for lambda and for sagemaker containers. It would be great to not have to move out of the poetry ecosystem back into pip when moving to production.

[neersighted]

See https://github.com/python-poetry/poetry-plugin-bundle -- this is out of scope for the core of Poetry, but would make sense as a FR on the plugin.

[cabney927]

My use case is installing poetry dependencies in a python lambda container.

I agree being able to do poetry install -t ./vendor would be nice. I worked around this by installing poetry in my container and exporting the poetry dependencies to a requirements.txt file. Then installing the resulting requirements.txt to the target directory. Hope this helps.

e.g.

pip install --upgrade pip
pip install poetry
poetry export --without-hashes --format=requirements.txt > requirements-poetry.txt
pip install -r requirements-poetry.txt --target ./vendor --upgrade

[lunarmint]

Any updates for this? +1 for the AWS lambda use case, would be so so nice to have.

[pievalentin]

Same thing for Azure Function resulting to exit the poetry ecosystem

[jason-curtis]

Admins, can we lock this conversation?

The issue is closed as out of scope and there are perfectly good solutions above. The recent comments have all been "+1"s from folks who haven't bothered to read the thread.

For the record, the solutions include:

1. using the bundle plugin and prepending to the PYTHONPATH
2. exporting requirements from poetry
3. building a .whl with poetry
4. using the virtualenvs.project setting to install to a known directory
5. I don't think anyone's mentioned it above, but in some cases you can also use the ENTRYPOINT instruction to run commands with poetry run. In one project I have a custom image built on python:3.10-slim with the entrypoint instruction: ENTRYPOINT [ "poetry", "run", "python", "-m", "awslambdaric" ] to run AWS Lambda RIC.