Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Faster build and installation of packages #6205

Merged
merged 8 commits into from
Feb 5, 2023

Conversation

sdispater
Copy link
Member

@sdispater sdispater commented Aug 20, 2022

This PR (which is an updated version of #4213) is the first step towards getting rid of subprocess calls to pip for package installations and removals.

The wheel building is handled by https://github.com/pypa/build and the wheel installation is handled by https://github.com/pypa/installer (with modifications to be more lenient).

Note that the wheel building is not parallelized due to race conditions that I am yet to track down (it seems like the issue comes from build when calling the backend hooks). This is not a blocker though.

Overall this speeds up quite a bit the installation times, especially once wheels are cached.

An example with the installation of 115 packages (warm cache and artifacts already downloaded):

Without the new builder/installer: 226.24s user 15.26s system 488% cpu 49.413 total
With the new builder/installer: 11.52s user 3.08s system 117% cpu 12.452 total

Note that this can be deactivated by setting experimental.installer.modern-installation.

Pull Request Check List

Resolves: python-poetry/roadmap#1

  • Added tests for changed code.
  • Updated documentation for changed code.

@sdispater sdispater force-pushed the feature/wheel-installer-and-builder branch 3 times, most recently from 455e638 to 96c7640 Compare August 20, 2022 23:21
@sdispater sdispater force-pushed the feature/wheel-installer-and-builder branch 4 times, most recently from 63ce58f to ca99889 Compare August 23, 2022 19:12
@sdispater sdispater force-pushed the feature/wheel-installer-and-builder branch from 62fb172 to fb2e022 Compare September 6, 2022 23:20
@sdispater sdispater force-pushed the feature/wheel-installer-and-builder branch 7 times, most recently from 6b23b02 to cdeb660 Compare September 18, 2022 17:59
@github-actions
Copy link

github-actions bot commented Sep 18, 2022

Deploy preview for website ready!

✅ Preview
https://website-az7ykk3s4-python-poetry.vercel.app

Built with commit 838f0ee.
This pull request is being automatically deployed with vercel-action

@sdispater sdispater force-pushed the feature/wheel-installer-and-builder branch from cdeb660 to c8f4468 Compare September 18, 2022 18:22
@sdispater sdispater marked this pull request as ready for review September 18, 2022 18:22
Copy link
Member

@radoering radoering left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some first (mostly superficial) feedback.

It seems the commits are not cut optimally: The commit "Add a wheel builder" already requires WheelInstaller which is added only later in "Add a wheel installer". (At least if we keep the commits separated when merging it may be worth to fix that.)

src/poetry/factory.py Outdated Show resolved Hide resolved
src/poetry/factory.py Outdated Show resolved Hide resolved
src/poetry/installation/executor.py Outdated Show resolved Hide resolved
src/poetry/installation/chef.py Outdated Show resolved Hide resolved
src/poetry/installation/chef.py Outdated Show resolved Hide resolved
@radoering radoering added this to the 1.3 milestone Sep 24, 2022
@radoering radoering added area/installer Related to the dependency installer impact/changelog Requires a changelog entry labels Sep 24, 2022
@neersighted neersighted modified the milestones: 1.3, 1.4 Oct 9, 2022
@radoering radoering force-pushed the feature/wheel-installer-and-builder branch from c8f4468 to 2e97898 Compare November 5, 2022 13:05
@radoering
Copy link
Member

Good point. It gave similar verbose output as pip but of course no note that this is probably not a problem with poetry. I added such a disclaimer. PTAL if the message can be improved.

@radoering radoering force-pushed the feature/wheel-installer-and-builder branch from 3041a2b to bd35898 Compare January 28, 2023 16:24
@radoering radoering force-pushed the feature/wheel-installer-and-builder branch 3 times, most recently from f59fbc0 to 7f159dc Compare February 3, 2023 22:39
@radoering radoering force-pushed the feature/wheel-installer-and-builder branch from 7f159dc to 82d0bde Compare February 5, 2023 12:20
@radoering radoering force-pushed the feature/wheel-installer-and-builder branch from 82d0bde to 838f0ee Compare February 5, 2023 13:04
@radoering radoering merged commit 53cdce0 into master Feb 5, 2023
@radoering radoering deleted the feature/wheel-installer-and-builder branch February 5, 2023 13:15
mwalbeck pushed a commit to mwalbeck/docker-python-poetry that referenced this pull request Feb 28, 2023
This PR contains the following updates:

| Package | Update | Change |
|---|---|---|
| [poetry](https://python-poetry.org/) ([source](https://github.com/python-poetry/poetry), [changelog](https://python-poetry.org/history/)) | minor | `1.3.2` -> `1.4.0` |

---

### Release Notes

<details>
<summary>python-poetry/poetry</summary>

### [`v1.4.0`](https://github.com/python-poetry/poetry/blob/HEAD/CHANGELOG.md#&#8203;140---2023-02-27)

[Compare Source](python-poetry/poetry@1.3.2...1.4.0)

##### Added

-   **Add a modern installer (`installer.modern-installation`) for faster installation of packages and independence from pip** ([#&#8203;6205](python-poetry/poetry#6205)).
-   Add support for `Private ::` trove classifiers ([#&#8203;7271](python-poetry/poetry#7271)).
-   Add the version of poetry in the `@generated` comment at the beginning of the lock file ([#&#8203;7339](python-poetry/poetry#7339)).
-   Add support for `virtualenvs.prefer-active-python` when running `poetry new` and `poetry init` ([#&#8203;7100](python-poetry/poetry#7100)).

##### Changed

-   **Deprecate the old installer, i.e. setting `experimental.new-installer` to `false`** ([#&#8203;7358](python-poetry/poetry#7358)).
-   Remove unused `platform` field from cached package info and bump the cache version ([#&#8203;7304](python-poetry/poetry#7304)).
-   Extra dependencies of the root project are now sorted in the lock file ([#&#8203;7375](python-poetry/poetry#7375)).
-   Remove upper boundary for `importlib-metadata` dependency ([#&#8203;7434](python-poetry/poetry#7434)).
-   Validate path dependencies during use instead of during construction ([#&#8203;6844](python-poetry/poetry#6844)).
-   Remove the deprecated `repository` modules ([#&#8203;7468](python-poetry/poetry#7468)).

##### Fixed

-   Fix an issue where an unconditional dependency of an extra was not installed in specific environments ([#&#8203;7175](python-poetry/poetry#7175)).
-   Fix an issue where a pre-release of a dependency was chosen even if a stable release fulfilled the constraint ([#&#8203;7225](python-poetry/poetry#7225), [#&#8203;7236](python-poetry/poetry#7236)).
-   Fix an issue where HTTP redirects were not handled correctly during publishing ([#&#8203;7160](python-poetry/poetry#7160)).
-   Fix an issue where `poetry check` did not handle the `-C, --directory` option correctly ([#&#8203;7241](python-poetry/poetry#7241)).
-   Fix an issue where the subdirectory information of a git dependency was not written to the lock file ([#&#8203;7367](python-poetry/poetry#7367)).
-   Fix an issue where the wrong Python version was selected when creating an virtual environment ([#&#8203;7221](python-poetry/poetry#7221)).
-   Fix an issue where packages that should be kept were uninstalled when calling `poetry install --sync` ([#&#8203;7389](python-poetry/poetry#7389)).
-   Fix an issue where an incorrect value was set for `sys.argv[0]` when running installed scripts ([#&#8203;6737](python-poetry/poetry#6737)).
-   Fix an issue where hashes in `direct_url.json` files were not written according to the specification ([#&#8203;7475](python-poetry/poetry#7475)).
-   Fix an issue where poetry commands failed due to special characters in the path of the project or virtual environment ([#&#8203;7471](python-poetry/poetry#7471)).
-   Fix an issue where poetry crashed with a `JSONDecodeError` when running a Python script that produced certain warnings ([#&#8203;6665](python-poetry/poetry#6665)).

##### Docs

-   Add advice on how to maintain a poetry plugin ([#&#8203;6977](python-poetry/poetry#6977)).
-   Update tox examples to comply with the latest tox release ([#&#8203;7341](python-poetry/poetry#7341)).
-   Mention that the `poetry export` can export `constraints.txt` files ([#&#8203;7383](python-poetry/poetry#7383)).
-   Add clarifications for moving configuration files ([#&#8203;6864](python-poetry/poetry#6864)).
-   Mention the different types of exact version specifications ([#&#8203;7503](python-poetry/poetry#7503)).

##### poetry-core ([`1.5.1`](https://github.com/python-poetry/poetry-core/releases/tag/1.5.1))

-   Improve marker handling ([#&#8203;528](python-poetry/poetry-core#528),
    [#&#8203;534](python-poetry/poetry-core#534),
    [#&#8203;530](python-poetry/poetry-core#530),
    [#&#8203;546](python-poetry/poetry-core#546),
    [#&#8203;547](python-poetry/poetry-core#547)).
-   Validate whether dependencies referenced in `extras` are defined in the main dependency group ([#&#8203;542](python-poetry/poetry-core#542)).
-   Poetry no longer generates a `setup.py` file in sdists by default ([#&#8203;318](python-poetry/poetry-core#318)).
-   Fix an issue where trailing newlines were allowed in `tool.poetry.description` ([#&#8203;505](python-poetry/poetry-core#505)).
-   Fix an issue where the name of the data folder in wheels was not normalized ([#&#8203;532](python-poetry/poetry-core#532)).
-   Fix an issue where the order of entries in the RECORD file was not deterministic ([#&#8203;545](python-poetry/poetry-core#545)).
-   Fix an issue where zero padding was not correctly handled in version comparisons ([#&#8203;540](python-poetry/poetry-core#540)).
-   Fix an issue where sdist builds did not support multiple READMEs ([#&#8203;486](python-poetry/poetry-core#486)).

##### poetry-plugin-export ([`^1.3.0`](https://github.com/python-poetry/poetry-plugin-export/releases/tag/1.3.0))

-   Fix an issue where the export failed if there was a circular dependency on the root package ([#&#8203;118](python-poetry/poetry-plugin-export#118)).

</details>

---

### Configuration

📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box

---

This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNC4xNTIuNSIsInVwZGF0ZWRJblZlciI6IjM0LjE1Mi41In0=-->

Reviewed-on: https://git.walbeck.it/walbeck-it/docker-python-poetry/pulls/655
Co-authored-by: renovate-bot <bot@walbeck.it>
Co-committed-by: renovate-bot <bot@walbeck.it>
Copy link

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Feb 29, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
area/installer Related to the dependency installer impact/changelog Requires a changelog entry
Projects
None yet
Development

Successfully merging this pull request may close these issues.

No longer rely on pip for installation
7 participants