Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add pip install --dry-run or similar, to get resolution result #53

Closed
vbabiy opened this issue Mar 15, 2011 · 66 comments · Fixed by #10771
Closed

Add pip install --dry-run or similar, to get resolution result #53

vbabiy opened this issue Mar 15, 2011 · 66 comments · Fixed by #10771
Labels
C: dependency resolution About choosing which dependencies to install type: feature request Request for a new feature UX User experience related

Comments

@vbabiy
Copy link
Contributor

vbabiy commented Mar 15, 2011

What's the problem this feature will solve?

pip currently does not have any mechanism for users to get the result of pip's dependency resolution. This functionality is useful for users to be able to:

  • generate something like a "lockfile"
  • checking if installing a package would break the existing environment
  • checking for dependency conflicts among a set of packages
  • (more?)

All these can be performed today, but require installing packages to some environment and introspecting the environment for information. Since all the relevant information is available to pip install at run time, it would be useful to avoid hitting issues with this.

Describe the solution you'd like

#8032 proposes a pip install --dry-run option.
#7819 proposes a pip resolve command.
#1345 has more proposals. :)

There have likely more proposals in the issue tracker that I can't find.

Alternative Solutions

Let other not-pip tooling in the ecosystem provide this functionality to users. This is sub-optimal given that pip's resolver isn't exposed publicly in any way (pip's internals are not to be used like a library).

The most notable example is pip-tools project, which is the current best answer for any user who is seeking this functionality.


Note: This description has been edited by @pradyunsg in April 2020 (see edit history for details), and some really old, outdated comments have been hidden on this issue.

@vbabiy

This comment has been minimized.

@vbabiy

This comment has been minimized.

@vbabiy

This comment has been minimized.

@vbabiy

This comment has been minimized.

@vbabiy

This comment has been minimized.

@vbabiy

This comment has been minimized.

@vbabiy

This comment has been minimized.

@vbabiy

This comment has been minimized.

@vbabiy

This comment has been minimized.

@vbabiy

This comment has been minimized.

@vbabiy

This comment has been minimized.

@phuihock

This comment has been minimized.

@LucianU

This comment has been minimized.

@baconalot

This comment has been minimized.

@inoatech

This comment has been minimized.

@smarnv

This comment has been minimized.

@Ivoz

This comment has been minimized.

@Russell-Jones

This comment has been minimized.

@beaumartinez

This comment has been minimized.

wallrj added a commit to ClusterHQ/flocker that referenced this issue Oct 16, 2014
@oraant

This comment has been minimized.

@realknorke

This comment has been minimized.

@mrsiesta

This comment has been minimized.

@mdeguzis

This comment has been minimized.

@balzss

This comment has been minimized.

@bleepnetworks

This comment has been minimized.

@andreif

This comment has been minimized.

@t-animal

This comment has been minimized.

@andreif

This comment has been minimized.

@grothesque
Copy link

Please correct me if I'm wrong, but the following seems to be true:

  • Installing a Python package involves executing its setup.py.
  • Without a --dry-run option, there's no easy and reliable way to know which packages pip's resolver will choose to install.

Hence, it seems to me that running pip install means consenting to running code from a rather arbitrary selection of PyPI packages on one's machine without an easy and reliable way to audit it. That selection depends recursively on the dependency choices and security practices of individual package authors.

@merwok
Copy link

merwok commented Dec 3, 2020

It depends if the project and version to be installed only has a source distribution (sdist, contains setup.py) or also wheels (built distribution, contains a metadata text file, is installed by file copies without arbitrary code being run)

@pradyunsg
Copy link
Member

Even with --dry-run, it's likely that pip will need to run build backends for the packages (which for setuptools, involves running the setup.py) which do not have wheels.

@uranusjr
Copy link
Member

Have we decided on what to call this functionality yet?

@ofek
Copy link
Contributor

ofek commented Dec 29, 2020

resolve seems logical

@cosmicexplorer
Copy link
Contributor

I called it resolve in #7819. It would correspond for example to the resolve command used for the coursier jvm artifact fetcher: https://get-coursier.io/docs/1.1.0-M11-2/cli-resolve.

@pradyunsg
Copy link
Member

#11096 (comment) just landed, adding pip install --dry-run.

@sbidoul Do you wanna close this, or wait under #10771 lands?

@sbidoul
Copy link
Member

sbidoul commented Jun 26, 2022

I'd wait for #10771 as machine readable output is important for several use cases mentioned here or in other issues merged with this one.

@pradyunsg
Copy link
Member

It took a while, but we got there in the end! :)

@FlorinAndrei
Copy link

I don't see a milestone - which version of pip will get this code?

@sbidoul
Copy link
Member

sbidoul commented Jul 15, 2022

I don't see a milestone - which version of pip will get this code?

It's merged in main. It will be in 22.2 when I do the release, in a week or so.

inmantaci pushed a commit to inmanta/inmanta-core that referenced this issue Jul 21, 2022
Bumps [pip](https://github.com/pypa/pip) from 22.1.2 to 22.2.
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href="https://github.com/pypa/pip/blob/main/NEWS.rst">pip's changelog</a>.</em></p>
<blockquote>
<h1>22.2 (2022-07-21)</h1>
<h2>Deprecations and Removals</h2>
<ul>
<li>Remove the <code>html5lib</code> deprecated feature flag. (<code>[#10825](pypa/pip#10825) &lt;https://github.com/pypa/pip/issues/10825&gt;</code>_)</li>
<li>Remove <code>--use-deprecated=backtrack-on-build-failures</code>. (<code>[#11241](pypa/pip#11241) &lt;https://github.com/pypa/pip/issues/11241&gt;</code>_)</li>
</ul>
<h2>Features</h2>
<ul>
<li>
<p>Add support to use <code>truststore &lt;https://pypi.org/project/truststore/&gt;</code>_ as an
alternative SSL certificate verification backend. The backend can be enabled on Python
3.10 and later by installing <code>truststore</code> into the environment, and adding the
<code>--use-feature=truststore</code> flag to various pip commands.</p>
<p><code>truststore</code> differs from the current default verification backend (provided by
<code>certifi</code>) in it uses the operating system’s trust store, which can be better
controlled and augmented to better support non-standard certificates. Depending on
feedback, pip may switch to this as the default certificate verification backend in
the future. (<code>[#11082](pypa/pip#11082) &lt;https://github.com/pypa/pip/issues/11082&gt;</code>_)</p>
</li>
<li>
<p>Add <code>--dry-run</code> option to <code>pip install</code>, to let it print what it would install but
not actually change anything in the target environment. (<code>[#11096](pypa/pip#11096) &lt;https://github.com/pypa/pip/issues/11096&gt;</code>_)</p>
</li>
<li>
<p>Record in wheel cache entries the URL of the original artifact that was downloaded
to build the cached wheels. The record is named <code>origin.json</code> and uses the PEP 610
Direct URL format. (<code>[#11137](pypa/pip#11137) &lt;https://github.com/pypa/pip/issues/11137&gt;</code>_)</p>
</li>
<li>
<p>Support <code>PEP 691 &lt;https://peps.python.org/pep-0691/&gt;</code><em>. (<code>[#11158](pypa/pip#11158) &lt;https://github.com/pypa/pip/issues/11158&gt;</code></em>)</p>
</li>
<li>
<p>pip's deprecation warnings now subclass the built-in <code>DeprecationWarning</code>, and
can be suppressed by running the Python interpreter with
<code>-W ignore::DeprecationWarning</code>. (<code>[#11225](pypa/pip#11225) &lt;https://github.com/pypa/pip/issues/11225&gt;</code>_)</p>
</li>
<li>
<p>Add <code>pip inspect</code> command to obtain the list of installed distributions and other
information about the Python environment, in JSON format. (<code>[#11245](pypa/pip#11245) &lt;https://github.com/pypa/pip/issues/11245&gt;</code>_)</p>
</li>
<li>
<p>Significantly speed up isolated environment creation, by using the same
sources for pip instead of creating a standalone installation for each
environment. (<code>[#11257](pypa/pip#11257) &lt;https://github.com/pypa/pip/issues/11257&gt;</code>_)</p>
</li>
<li>
<p>Add an experimental <code>--report</code> option to the install command to generate a JSON report
of what was installed. In combination with <code>--dry-run</code> and <code>--ignore-installed</code> it
can be used to resolve the requirements. (<code>[#53](pypa/pip#53) &lt;https://github.com/pypa/pip/issues/53&gt;</code>_)</p>
</li>
</ul>
<h2>Bug Fixes</h2>
<ul>
<li>Fix <code>pip install --pre</code> for packages with pre-release build dependencies defined
both in <code>pyproject.toml</code>'s <code>build-system.requires</code> and <code>setup.py</code>'s
<code>setup_requires</code>. (<code>[#10222](pypa/pip#10222) &lt;https://github.com/pypa/pip/issues/10222&gt;</code>_)</li>
<li>When pip rewrites the shebang line in a script during wheel installation,
update the hash and size in the corresponding <code>RECORD</code> file entry. (<code>[#10744](pypa/pip#10744) &lt;https://github.com/pypa/pip/issues/10744&gt;</code>_)</li>
<li>Do not consider a <code>.dist-info</code> directory found inside a wheel-like zip file
as metadata for an installed distribution. A package in a wheel is (by</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href="https://github.com/pypa/pip/commit/8e7e76e60f4e115ea1201bee2f176377a718fce1"><code>8e7e76e</code></a> Bump for release</li>
<li><a href="https://github.com/pypa/pip/commit/b6f6a94e36f10a4535ea5bbdc6b351f62003eede"><code>b6f6a94</code></a> Update AUTHORS.txt</li>
<li><a href="https://github.com/pypa/pip/commit/790725aca3f60c745e33827a6079d9600da373d8"><code>790725a</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/pypa/pip/issues/11274">#11274</a> from sbidoul/install-report-note-sbi</li>
<li><a href="https://github.com/pypa/pip/commit/d4b9e187aa7cc5ab14b2339f6171f7f2ea6504e9"><code>d4b9e18</code></a> Add clarifications to the installation report documentation</li>
<li><a href="https://github.com/pypa/pip/commit/b1a01ef762a78af1194958a1c874015eaf81fd04"><code>b1a01ef</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/pypa/pip/issues/11265">#11265</a> from finnagin/main</li>
<li><a href="https://github.com/pypa/pip/commit/48bcb0a4ccd30a9d00e58fe58827772e307a7e39"><code>48bcb0a</code></a> reformat to pass pre-commit check</li>
<li><a href="https://github.com/pypa/pip/commit/a7c1fe3bff5655393018c53b448b669b3525515b"><code>a7c1fe3</code></a> Remove utc fixture from tests</li>
<li><a href="https://github.com/pypa/pip/commit/0c574f72905185d62bcca741c813df9bae1d9282"><code>0c574f7</code></a> Remove time import</li>
<li><a href="https://github.com/pypa/pip/commit/246fef19149eea893f1cf3efd53f9b17c94c952f"><code>246fef1</code></a> Remove utc fixture</li>
<li><a href="https://github.com/pypa/pip/commit/c9cb7f4629bdd8c61b792feff6dacb1d2e848d57"><code>c9cb7f4</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/pypa/pip/issues/11270">#11270</a> from uranusjr/upgrade-pre-commit-hooks</li>
<li>Additional commits viewable in <a href="https://github.com/pypa/pip/compare/22.1.2...22.2">compare view</a></li>
</ul>
</details>
<br />

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=pip&package-manager=pip&previous-version=22.1.2&new-version=22.2)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

</details>
inmantaci pushed a commit to inmanta/inmanta-core that referenced this issue Jul 21, 2022
Bumps [pip](https://github.com/pypa/pip) from 22.1.2 to 22.2.
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href="https://github.com/pypa/pip/blob/main/NEWS.rst">pip's changelog</a>.</em></p>
<blockquote>
<h1>22.2 (2022-07-21)</h1>
<h2>Deprecations and Removals</h2>
<ul>
<li>Remove the <code>html5lib</code> deprecated feature flag. (<code>[#10825](pypa/pip#10825) &lt;https://github.com/pypa/pip/issues/10825&gt;</code>_)</li>
<li>Remove <code>--use-deprecated=backtrack-on-build-failures</code>. (<code>[#11241](pypa/pip#11241) &lt;https://github.com/pypa/pip/issues/11241&gt;</code>_)</li>
</ul>
<h2>Features</h2>
<ul>
<li>
<p>Add support to use <code>truststore &lt;https://pypi.org/project/truststore/&gt;</code>_ as an
alternative SSL certificate verification backend. The backend can be enabled on Python
3.10 and later by installing <code>truststore</code> into the environment, and adding the
<code>--use-feature=truststore</code> flag to various pip commands.</p>
<p><code>truststore</code> differs from the current default verification backend (provided by
<code>certifi</code>) in it uses the operating system’s trust store, which can be better
controlled and augmented to better support non-standard certificates. Depending on
feedback, pip may switch to this as the default certificate verification backend in
the future. (<code>[#11082](pypa/pip#11082) &lt;https://github.com/pypa/pip/issues/11082&gt;</code>_)</p>
</li>
<li>
<p>Add <code>--dry-run</code> option to <code>pip install</code>, to let it print what it would install but
not actually change anything in the target environment. (<code>[#11096](pypa/pip#11096) &lt;https://github.com/pypa/pip/issues/11096&gt;</code>_)</p>
</li>
<li>
<p>Record in wheel cache entries the URL of the original artifact that was downloaded
to build the cached wheels. The record is named <code>origin.json</code> and uses the PEP 610
Direct URL format. (<code>[#11137](pypa/pip#11137) &lt;https://github.com/pypa/pip/issues/11137&gt;</code>_)</p>
</li>
<li>
<p>Support <code>PEP 691 &lt;https://peps.python.org/pep-0691/&gt;</code><em>. (<code>[#11158](pypa/pip#11158) &lt;https://github.com/pypa/pip/issues/11158&gt;</code></em>)</p>
</li>
<li>
<p>pip's deprecation warnings now subclass the built-in <code>DeprecationWarning</code>, and
can be suppressed by running the Python interpreter with
<code>-W ignore::DeprecationWarning</code>. (<code>[#11225](pypa/pip#11225) &lt;https://github.com/pypa/pip/issues/11225&gt;</code>_)</p>
</li>
<li>
<p>Add <code>pip inspect</code> command to obtain the list of installed distributions and other
information about the Python environment, in JSON format. (<code>[#11245](pypa/pip#11245) &lt;https://github.com/pypa/pip/issues/11245&gt;</code>_)</p>
</li>
<li>
<p>Significantly speed up isolated environment creation, by using the same
sources for pip instead of creating a standalone installation for each
environment. (<code>[#11257](pypa/pip#11257) &lt;https://github.com/pypa/pip/issues/11257&gt;</code>_)</p>
</li>
<li>
<p>Add an experimental <code>--report</code> option to the install command to generate a JSON report
of what was installed. In combination with <code>--dry-run</code> and <code>--ignore-installed</code> it
can be used to resolve the requirements. (<code>[#53](pypa/pip#53) &lt;https://github.com/pypa/pip/issues/53&gt;</code>_)</p>
</li>
</ul>
<h2>Bug Fixes</h2>
<ul>
<li>Fix <code>pip install --pre</code> for packages with pre-release build dependencies defined
both in <code>pyproject.toml</code>'s <code>build-system.requires</code> and <code>setup.py</code>'s
<code>setup_requires</code>. (<code>[#10222](pypa/pip#10222) &lt;https://github.com/pypa/pip/issues/10222&gt;</code>_)</li>
<li>When pip rewrites the shebang line in a script during wheel installation,
update the hash and size in the corresponding <code>RECORD</code> file entry. (<code>[#10744](pypa/pip#10744) &lt;https://github.com/pypa/pip/issues/10744&gt;</code>_)</li>
<li>Do not consider a <code>.dist-info</code> directory found inside a wheel-like zip file
as metadata for an installed distribution. A package in a wheel is (by</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href="https://github.com/pypa/pip/commit/8e7e76e60f4e115ea1201bee2f176377a718fce1"><code>8e7e76e</code></a> Bump for release</li>
<li><a href="https://github.com/pypa/pip/commit/b6f6a94e36f10a4535ea5bbdc6b351f62003eede"><code>b6f6a94</code></a> Update AUTHORS.txt</li>
<li><a href="https://github.com/pypa/pip/commit/790725aca3f60c745e33827a6079d9600da373d8"><code>790725a</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/pypa/pip/issues/11274">#11274</a> from sbidoul/install-report-note-sbi</li>
<li><a href="https://github.com/pypa/pip/commit/d4b9e187aa7cc5ab14b2339f6171f7f2ea6504e9"><code>d4b9e18</code></a> Add clarifications to the installation report documentation</li>
<li><a href="https://github.com/pypa/pip/commit/b1a01ef762a78af1194958a1c874015eaf81fd04"><code>b1a01ef</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/pypa/pip/issues/11265">#11265</a> from finnagin/main</li>
<li><a href="https://github.com/pypa/pip/commit/48bcb0a4ccd30a9d00e58fe58827772e307a7e39"><code>48bcb0a</code></a> reformat to pass pre-commit check</li>
<li><a href="https://github.com/pypa/pip/commit/a7c1fe3bff5655393018c53b448b669b3525515b"><code>a7c1fe3</code></a> Remove utc fixture from tests</li>
<li><a href="https://github.com/pypa/pip/commit/0c574f72905185d62bcca741c813df9bae1d9282"><code>0c574f7</code></a> Remove time import</li>
<li><a href="https://github.com/pypa/pip/commit/246fef19149eea893f1cf3efd53f9b17c94c952f"><code>246fef1</code></a> Remove utc fixture</li>
<li><a href="https://github.com/pypa/pip/commit/c9cb7f4629bdd8c61b792feff6dacb1d2e848d57"><code>c9cb7f4</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/pypa/pip/issues/11270">#11270</a> from uranusjr/upgrade-pre-commit-hooks</li>
<li>Additional commits viewable in <a href="https://github.com/pypa/pip/compare/22.1.2...22.2">compare view</a></li>
</ul>
</details>
<br />

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=pip&package-manager=pip&previous-version=22.1.2&new-version=22.2)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

</details>
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Aug 15, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
C: dependency resolution About choosing which dependencies to install type: feature request Request for a new feature UX User experience related
Projects
None yet
Development

Successfully merging a pull request may close this issue.