Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build and install the root package as a wheel in one invocation #1382

Closed
geryogam opened this issue Sep 17, 2019 · 21 comments
Closed

Build and install the root package as a wheel in one invocation #1382

geryogam opened this issue Sep 17, 2019 · 21 comments
Labels
area/cli Related to the command line area/installer Related to the dependency installer kind/feature Feature requests/implementations

Comments

@geryogam
Copy link
Contributor

geryogam commented Sep 17, 2019

@sdispater, in #730 you said:

The install command installs the current project in editable mode by internally creating a temporary setup.py file and executing pip install -e .. It does it each time install is executed to ensure that any new elements like entrypoints are properly installed.

Could you allow non-editable installations by adding a --no-editable command-line parameter for these three commands:

  • poetry install --no-editable
  • poetry add --no-editable
  • poetry update --no-editable

The documentation states that it is already possible for local dependencies, but with a develop file parameter in the pyproject.toml file:

If you don't want the dependency to be installed in editable mode you can specify it in the pyproject.toml file:

[tool.poetry.dependencies]
my-package = {path = "../my/path", develop = false}

Here I think that a command-line parameter would be more appropriate than a file parameter, as it is something that you want to configure at the command-line level rather than at the project level (like with pip where sometimes you want to run pip install . and sometimes pip install -e .). So if you could also drop the develop file parameter that would be awesome.

Being able to install projects in non-editable mode is so important, especially for deployment on servers. Currently we cannot use the poetry install command exclusively for such installations, we still have to resort to the pip install command.

@kasteph kasteph added area/cli Related to the command line kind/feature Feature requests/implementations area/installer Related to the dependency installer labels Sep 24, 2019
@stale
Copy link

stale bot commented Nov 23, 2019

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the stale label Nov 23, 2019
@geryogam
Copy link
Contributor Author

Up.

@stale stale bot removed the stale label Nov 23, 2019
@cjolowicz
Copy link

This is an important feature for multi-stage Docker builds.

Multi-stage Docker builds for Python typically install the project into a virtualenv during the build stage, and then copy the virtualenv over into a slim final image. An editable install creates a .egg-link file that links to the source code, and this link would only be valid for the duration of the build stage.

Currently my build stage installs projects by exporting from Poetry to pip:

COPY pyproject.toml poetry.lock ./
RUN /root/.poetry/bin/poetry export -f requirements.txt | /venv/bin/pip install -r /dev/stdin 
COPY . ./
RUN /root/.poetry/bin/poetry build && /venv/bin/pip install dist/*.whl

The requested feature would allow us to use Poetry directly:

COPY pyproject.toml poetry.lock ./
RUN /root/.poetry/bin/poetry install --no-root
COPY . ./
RUN /root/.poetry/bin/poetry install --no-editable

@arbroen
Copy link

arbroen commented Jan 6, 2020

I am using the exact same scenario as @cjolowicz but with the added configuration of a private PIP server. I cannot use the PIP_EXTRA_INDEX_URL due to security issues and have this configured via the pyproject.yml instead. Thus the solution is:

RUN poetry install --no-dev
RUN rm .venv/lib/python3.7/site-packages/*.egg-link
RUN poetry build
RUN pip install dist/*.whl --target=.venv/lib/python3.7/site-packages/ --no-deps

A poetry install --no-dev --no-editable would be an improvement.

@geryogam
Copy link
Contributor Author

@sdispater @finswimmer According to the number of thumbs up, this is an important feature. Being able to install projects in non editable mode is so important. Without this, we cannot use poetry exclusively and still have to rely on pip (by calling pip install explicitly). Any chance this get prioritised?

@geryogam geryogam changed the title Allow non editable installations Allow non-editable installations Feb 18, 2020
@ghost
Copy link

ghost commented Feb 20, 2020

Currently the cleanest way I've found to do this is along the lines of

python3 -m venv /venv && \
. /venv/bin/activate && \
poetry install -n --no-dev --no-root && \
poetry build -f wheel -n && \
pip install --no-deps dist/*.whl && \
rm -rf dist *.egg-info

Adding this feature would cut the number of commands in half and I wouldn't have to worry about dist files getting left over in the event of a failure.

Edit: Actually, I realized you can use pip install --no-deps . (you can also skip --no-deps and install everything, though I'm not sure that uses poetry.lock so I'd avoid it) directly with newer versions of pip and it will pick up the configuration in the [build-system] section of pyproject.toml though this kept failing on my machine using pyenv. Worked in docker/when using system python, however.

@ghost
Copy link

ghost commented May 20, 2020

This is also an issue with path dependencies, which also only install as editable rather than actually installing to site packages, and seemingly aren't usable at all if you want pip compatibility. Unfortunately using git repositories isn't viable in a lot of situations. Most of the stuff I work on uses a chroot or container for the build process which has no access to the user's git credentials.

@thejcannon
Copy link
Contributor

I'd like to see this feature for a reason I haven't seen enumerated yet. I can't validate my wheel has all the files I expect when I run my tests with my package in editable mode. Since the files will correctly be there locally, my tests pass, however it could very well be the case that I'm not including those files in my wheel meaning the package is broken.

My solution would be that PR validation would install the package in non-editable mode. Today that's accomplished by building the wheel and installing that, but would lovee to see poetry install --bikeshed "just work".

@finswimmer
Copy link
Member

I can't validate my wheel has all the files I expect when I run my tests with my package in editable mode. Since the files will correctly be there locally, my tests pass, however it could very well be the case that I'm not including those files in my wheel meaning the package is broken.

@thejcannon Isn't this exactly why tox exists?

@thejcannon
Copy link
Contributor

@finswimmer It may/may not be, but at this point tox is certainly more than that feature.

I hope you aren't advocating for a testing solution that involves both poetry and tox. They overlap in responsibilities (virtual environment management and command running), and I would much prefer to stay just using poetry 😄 .

@ckp95
Copy link

ckp95 commented Dec 12, 2020

I'd like to see this feature for a reason I haven't seen enumerated yet. I can't validate my wheel has all the files I expect when I run my tests with my package in editable mode. Since the files will correctly be there locally, my tests pass, however it could very well be the case that I'm not including those files in my wheel meaning the package is broken.

My solution would be that PR validation would install the package in non-editable mode. Today that's accomplished by building the wheel and installing that, but would lovee to see poetry install --bikeshed "just work".

Echoing this post -- today I couldn't figure out why my CI test for whether py.typed is included in the package wasn't working right. I have to use pip install . to make sure it's not in editable mode, but I'd like to be able to use Poetry instead.

@nschad
Copy link

nschad commented May 11, 2021

Any Update?

@geryogam
Copy link
Contributor Author

@ShuzZzle: It is still in the To-do list: https://github.com/python-poetry/poetry/projects/3#card-31617262
@sdispater and @abn: According to the number of thumbs up, this feature request is highly anticipated, so is there any way to make it a priority? It has been two years now.

@rszamszur
Copy link

rszamszur commented Feb 10, 2022

This is an important feature for multi-stage Docker builds.

@cjolowicz I agree. Moreover, this applies to the build process, CI/CD as well.

@maggyero In PR I've implemented --no-editable option just for the install command.

For add if I understand correctly poetry add will by default install dependency in none editable mode unless specified otherwise in pyproject.toml.

Lastly, there is poetry update command. From the code, it seems it only updates the dependencies, not the root package (it makes sense, since it always was in editable mode). However, it'll probably require some logic to update the root package if the project is installed with --no-editable. Fortunately, this can be detected automaticaly based on if package_name.pth file is in project .venv/lib/python3.x/site-packages/ directory. If so, reinstall, else do nothing. But first I'll wait for the feedback from Poetry maintainers.

Have a good one!

@johnthagen
Copy link
Contributor

As @cjolowicz mentions, this feature is really essential for a cleanly working with multi-stage Docker builds purely within Poetry.

If both this and venv path configuration were implemented (#1579), Poetry would have a super great story for Docker ❤️. Currently, it's a bit clunky to have to essentially shell out to the venv stdlib module, an exported requirement.txt file, and pip to get the correct behavoir.

@neersighted neersighted changed the title Allow non-editable installations Build and install the root package as a wheel in one invocation Oct 4, 2022
@neersighted
Copy link
Member

Closing this for now as covered by the bundle plugin, which is intended for this use case. It's still very much in development/a little rough, but that is where this functionality is going to live:

https://github.com/python-poetry/poetry-plugin-bundle

@Keramblock
Copy link

Keramblock commented Oct 26, 2022

I don't think, that it should be designed as a plugin. Personally I was surprised,that it is not a core functionality of poetry. I suggest to reopen this issue =(

@smcoll
Copy link

smcoll commented Nov 9, 2022

@neersighted i don't believe the plugin you mentioned handles the case raised by this issue. python-poetry/poetry-plugin-bundle#37 (comment)

@neersighted
Copy link
Member

neersighted commented Nov 9, 2022

Editable mode is orthogonal to whether installing a package and its deps into an environment (with the root in non-editable mode) is possible; indeed, that is what bundle currently does. You want it to grow a --no-editable or similar flag; your issue on the other repo is the correct place to track that.

@link89
Copy link

link89 commented Nov 11, 2022

#1382 (comment) this workaround is great but I think it is a bug of poetry-plugin-export that didn't handle develop = true correctly. Once they fix the bug there will me no clear way to work around this problem.

Copy link

github-actions bot commented Mar 1, 2024

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Mar 1, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
area/cli Related to the command line area/installer Related to the dependency installer kind/feature Feature requests/implementations
Projects
None yet
Development

Successfully merging a pull request may close this issue.