Faster build and installation of packages #6205
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
sdispater
force-pushed
the
feature/wheel-installer-and-builder
branch
3 times, most recently
from
455e638
to
96c7640
Compare
sdispater
force-pushed
the
feature/wheel-installer-and-builder
branch
4 times, most recently
from
63ce58f
to
ca99889
Compare
sdispater
force-pushed
the
feature/wheel-installer-and-builder
branch
from
62fb172
to
fb2e022
Compare
sdispater
force-pushed
the
feature/wheel-installer-and-builder
branch
7 times, most recently
from
6b23b02
to
cdeb660
Compare
|
Deploy preview for website ready! ✅ Preview Built with commit 838f0ee. |
sdispater
force-pushed
the
feature/wheel-installer-and-builder
branch
from
cdeb660
to
c8f4468
Compare
4 tasks
radoering
reviewed
Sep 23, 2022
There was a problem hiding this comment.
Some first (mostly superficial) feedback.
It seems the commits are not cut optimally: The commit "Add a wheel builder" already requires WheelInstaller which is added only later in "Add a wheel installer". (At least if we keep the commits separated when merging it may be worth to fix that.)
radoering
added
area/installer
2 tasks
radoering
force-pushed
the
feature/wheel-installer-and-builder
branch
from
c8f4468
to
2e97898
Compare
|
Good point. It gave similar verbose output as pip but of course no note that this is probably not a problem with poetry. I added such a disclaimer. PTAL if the message can be improved. |
dimbleby
reviewed
Jan 28, 2023
radoering
force-pushed
the
feature/wheel-installer-and-builder
branch
from
3041a2b
to
bd35898
Compare
neersighted
reviewed
Jan 28, 2023
radoering
force-pushed
the
feature/wheel-installer-and-builder
branch
3 times, most recently
from
f59fbc0
to
7f159dc
Compare
radoering
force-pushed
the
feature/wheel-installer-and-builder
branch
from
7f159dc
to
82d0bde
Compare
Co-authored-by: Randy Döring <30527984+radoering@users.noreply.github.com>
Co-authored-by: Randy Döring <30527984+radoering@users.noreply.github.com>
…t behavior of old installer)
radoering
force-pushed
the
feature/wheel-installer-and-builder
branch
from
82d0bde
to
838f0ee
Compare
radoering
approved these changes
Feb 5, 2023
3 tasks
4 tasks
mwalbeck
pushed a commit
to mwalbeck/docker-python-poetry
that referenced
this pull request
Feb 28, 2023
This PR contains the following updates: | Package | Update | Change | |---|---|---| | [poetry](https://python-poetry.org/) ([source](https://github.com/python-poetry/poetry), [changelog](https://python-poetry.org/history/)) | minor | `1.3.2` -> `1.4.0` | --- ### Release Notes <details> <summary>python-poetry/poetry</summary> ### [`v1.4.0`](https://github.com/python-poetry/poetry/blob/HEAD/CHANGELOG.md#​140---2023-02-27) [Compare Source](python-poetry/poetry@1.3.2...1.4.0) ##### Added - **Add a modern installer (`installer.modern-installation`) for faster installation of packages and independence from pip** ([#​6205](python-poetry/poetry#6205)). - Add support for `Private ::` trove classifiers ([#​7271](python-poetry/poetry#7271)). - Add the version of poetry in the `@generated` comment at the beginning of the lock file ([#​7339](python-poetry/poetry#7339)). - Add support for `virtualenvs.prefer-active-python` when running `poetry new` and `poetry init` ([#​7100](python-poetry/poetry#7100)). ##### Changed - **Deprecate the old installer, i.e. setting `experimental.new-installer` to `false`** ([#​7358](python-poetry/poetry#7358)). - Remove unused `platform` field from cached package info and bump the cache version ([#​7304](python-poetry/poetry#7304)). - Extra dependencies of the root project are now sorted in the lock file ([#​7375](python-poetry/poetry#7375)). - Remove upper boundary for `importlib-metadata` dependency ([#​7434](python-poetry/poetry#7434)). - Validate path dependencies during use instead of during construction ([#​6844](python-poetry/poetry#6844)). - Remove the deprecated `repository` modules ([#​7468](python-poetry/poetry#7468)). ##### Fixed - Fix an issue where an unconditional dependency of an extra was not installed in specific environments ([#​7175](python-poetry/poetry#7175)). - Fix an issue where a pre-release of a dependency was chosen even if a stable release fulfilled the constraint ([#​7225](python-poetry/poetry#7225), [#​7236](python-poetry/poetry#7236)). - Fix an issue where HTTP redirects were not handled correctly during publishing ([#​7160](python-poetry/poetry#7160)). - Fix an issue where `poetry check` did not handle the `-C, --directory` option correctly ([#​7241](python-poetry/poetry#7241)). - Fix an issue where the subdirectory information of a git dependency was not written to the lock file ([#​7367](python-poetry/poetry#7367)). - Fix an issue where the wrong Python version was selected when creating an virtual environment ([#​7221](python-poetry/poetry#7221)). - Fix an issue where packages that should be kept were uninstalled when calling `poetry install --sync` ([#​7389](python-poetry/poetry#7389)). - Fix an issue where an incorrect value was set for `sys.argv[0]` when running installed scripts ([#​6737](python-poetry/poetry#6737)). - Fix an issue where hashes in `direct_url.json` files were not written according to the specification ([#​7475](python-poetry/poetry#7475)). - Fix an issue where poetry commands failed due to special characters in the path of the project or virtual environment ([#​7471](python-poetry/poetry#7471)). - Fix an issue where poetry crashed with a `JSONDecodeError` when running a Python script that produced certain warnings ([#​6665](python-poetry/poetry#6665)). ##### Docs - Add advice on how to maintain a poetry plugin ([#​6977](python-poetry/poetry#6977)). - Update tox examples to comply with the latest tox release ([#​7341](python-poetry/poetry#7341)). - Mention that the `poetry export` can export `constraints.txt` files ([#​7383](python-poetry/poetry#7383)). - Add clarifications for moving configuration files ([#​6864](python-poetry/poetry#6864)). - Mention the different types of exact version specifications ([#​7503](python-poetry/poetry#7503)). ##### poetry-core ([`1.5.1`](https://github.com/python-poetry/poetry-core/releases/tag/1.5.1)) - Improve marker handling ([#​528](python-poetry/poetry-core#528), [#​534](python-poetry/poetry-core#534), [#​530](python-poetry/poetry-core#530), [#​546](python-poetry/poetry-core#546), [#​547](python-poetry/poetry-core#547)). - Validate whether dependencies referenced in `extras` are defined in the main dependency group ([#​542](python-poetry/poetry-core#542)). - Poetry no longer generates a `setup.py` file in sdists by default ([#​318](python-poetry/poetry-core#318)). - Fix an issue where trailing newlines were allowed in `tool.poetry.description` ([#​505](python-poetry/poetry-core#505)). - Fix an issue where the name of the data folder in wheels was not normalized ([#​532](python-poetry/poetry-core#532)). - Fix an issue where the order of entries in the RECORD file was not deterministic ([#​545](python-poetry/poetry-core#545)). - Fix an issue where zero padding was not correctly handled in version comparisons ([#​540](python-poetry/poetry-core#540)). - Fix an issue where sdist builds did not support multiple READMEs ([#​486](python-poetry/poetry-core#486)). ##### poetry-plugin-export ([`^1.3.0`](https://github.com/python-poetry/poetry-plugin-export/releases/tag/1.3.0)) - Fix an issue where the export failed if there was a circular dependency on the root package ([#​118](python-poetry/poetry-plugin-export#118)). </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNC4xNTIuNSIsInVwZGF0ZWRJblZlciI6IjM0LjE1Mi41In0=--> Reviewed-on: https://git.walbeck.it/walbeck-it/docker-python-poetry/pulls/655 Co-authored-by: renovate-bot <bot@walbeck.it> Co-committed-by: renovate-bot <bot@walbeck.it>
4 tasks
4 tasks
2 tasks
|
This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs. |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR (which is an updated version of #4213) is the first step towards getting rid of subprocess calls to
pipfor package installations and removals.The wheel building is handled by https://github.com/pypa/build and the wheel installation is handled by https://github.com/pypa/installer (with modifications to be more lenient).
Note that the wheel building is not parallelized due to race conditions that I am yet to track down (it seems like the issue comes from
buildwhen calling the backend hooks). This is not a blocker though.Overall this speeds up quite a bit the installation times, especially once wheels are cached.
An example with the installation of 115 packages (warm cache and artifacts already downloaded):
Without the new builder/installer:
226.24s user 15.26s system 488% cpu 49.413 totalWith the new builder/installer:
11.52s user 3.08s system 117% cpu 12.452 totalNote that this can be deactivated by setting
experimental.installer.modern-installation.Pull Request Check List
Resolves: python-poetry/roadmap#1