Instantiate a new accumulator `InstallRequirement` for `combine_install_requirements` output #1519

richafrank · 2021-10-28T01:13:38Z

Replace the copy+mutate behavior of combine_install_requirements with instantiation of a new InstallRequirement. Since the PyPIRepository caches dependencies keyed by InstallRequirement instances, we no longer need to worry about:

preparing to force discovery of the ireq's name, then needing to keep track of the the "prepared" copy's dependencies (via the copy_ireq_dependencies method)
preparing an ireq for its name, then copying and mutating and needing to get dependencies anew for the mutated copy, which has already been prepared, and it's not really safe to prepare again
avoiding the previous situation by preparing a copy of the ireq for its name (the "name_proxy")

i.e. a much simpler solution. We can prepare an ireq to get its name and not worry about later uses.

The inputs to the new instance include the link, as done in #1506, which solves the local file information loss.

Builds on and replaces #1507, #1385, and #1512. The tests from all PRs pass, including the xfail I added in #1507.

Fixes #1505 , #1511 , #1054

Contributor checklist

Provided the tests for the changes.
Assure PR title is short, clear, and good to be included in the user-oriented changelog

Maintainer checklist

Assure one of these labels is present: backwards incompatible, feature, enhancement, deprecation, bug, dependency, docs or skip-changelog as they determine changelog listing.
Assign the PR to an existing or new milestone for the target version (following Semantic Versioning).

tests/test_resolver.py

richafrank · 2021-10-28T01:16:19Z

piptools/resolver.py

+            # Don't just yield from above. Instead, use the same `markers`-stripping
+            # behavior as we have for cached dependencies below.
+            dependency_strings = sorted(str(ireq.req) for ireq in dependencies)
+            yield from self._ireqs_of_dependencies(ireq, dependency_strings)


Without this change to make this behavior consistent with the cached dependencies case, tests failed because the output included markers, e.g. fake-tensorboardx==0.5; extra == "tune".

richafrank · 2021-10-28T01:19:08Z

This was necessary to solve the "conflict" described in #1512 (comment), but also feels like the longer-term better solution.

richafrank · 2022-01-08T19:55:20Z

@atugushev @AndydeCleyre Thoughts on timing for a review? Looks like not much activity across the repo since October...

mmerickel · 2022-02-24T05:19:44Z

Kind of in a pickle right now stuck on 6.2.0 due to this issue but that version is busted with new pip's fancy new log handler. Several new versions appear to have shipped with this regression so it'd be great to see some movement on this. To help I can confirm I ran this PR on my repo with 30 local packages using the -e file:src/foo#egg=foo syntax in them and the only changes observed is that it normalized packages with . in the name to a -. For example zope.sqlalchemy is now zope-sqlalchemy in the requirements.txt but this is a valid name for the package. Without this fix in 6.5.1 I see many dependencies dropped from the resulting txt file as well as many references to local packages removed from dependencies that are kept.

Jorricks · 2022-03-11T11:24:02Z

First of all, @richafrank this PR looks in very good shape. Nice work!
Good comments as well :)

Kind of in a pickle right now stuck on 6.2.0 due to this issue but that version is busted with new pip's fancy new log handler. Several new versions appear to have shipped with this regression so it'd be great to see some movement on this. To help I can confirm I ran this PR on my repo with 30 local packages using the -e file:src/foo#egg=foo syntax in them and the only changes observed is that it normalized packages with . in the name to a -. For example zope.sqlalchemy is now zope-sqlalchemy in the requirements.txt but this is a valid name for the package. Without this fix in 6.5.1 I see many dependencies dropped from the resulting txt file as well as many references to local packages removed from dependencies that are kept.

I'd actually argue that that is not a valid package name. If you lookup pypi, there is no zope-sqlalchemy, there is only zope.sqlalchemy. I actually just closed a ticket over at zope which truly should be opened here: zopefoundation/zope.interface#255. I will open an issue here as that is not related to this PR and should not block this PR in any way.

frejonb · 2022-04-04T10:53:32Z

Thanks for this PR. I'm extremely blocked by #1505. I'm not really sure what is missing for approving this PR, but would it be possible that there is a shorter, easier to review changes that only address #1505?

atzannes · 2022-05-25T01:03:02Z

I think I speak on behalf of many when I say that I am rooting this PR to get merged and released ASAP

…ependency" (commit d720fe6) This brings back copy_ireq_dependencies and combines its use with that of the proxies-for-naming. Editable and url_requirement ireqs are not cached by the Repository, so when dependencies are queried for them once, no dependencies will be found later for the copy made by combine_install_requirements. copy_ireq_dependencies links those dependencies at the time of copy.

round of resolution. By this point, we've frozen the dependencies.

Fixes jazzband#1054 Co-authored-by: Andrew <harta5039@gmail.com>

when combining instances

…uirement instances in combine_install_requirements

tests/test_cli_compile.py

atugushev

Let's get that in 👍🏻

to ensure nothing else was emitted

richafrank force-pushed the combine-without-copy branch from 1c3def2 to 2bdbd6f Compare October 28, 2021 01:13

richafrank commented Oct 28, 2021

View reviewed changes

tests/test_resolver.py Outdated Show resolved Hide resolved

richafrank commented Oct 28, 2021

View reviewed changes

This was referenced Oct 29, 2021

Tarball URL erased when compiled #1518

Open

Re resolve combined ireq #1512

Closed

richafrank force-pushed the combine-without-copy branch 2 times, most recently from bd3caf8 to b7e5d16 Compare November 5, 2021 15:42

richafrank mentioned this pull request Dec 18, 2021

pip-compile doesn't cache local requirements correctly. #1545

Closed

richafrank force-pushed the combine-without-copy branch from b7e5d16 to b29d25c Compare January 8, 2022 19:47

richafrank added bug fix resolver Related to dependency resolver labels Jan 8, 2022

AndydeCleyre force-pushed the combine-without-copy branch from b29d25c to 95d1553 Compare February 8, 2022 18:28

This was referenced Feb 11, 2022

Dependency stops getting locked if switching to Git install #1575

Closed

Fixing issue with losing data about local files #1385

Closed

Track dependencies for local packages that get combined #1507

Closed

AndydeCleyre requested a review from atugushev February 24, 2022 05:26

AndydeCleyre mentioned this pull request Feb 26, 2022

Some dependencies are lost when compiling with pip-tools>=6.3.1 #1589

Closed

RedRoserade added a commit to RedRoserade/pip-tools-repro-repo-a that referenced this pull request Feb 26, 2022

Test jazzband/pip-tools#1519

80a89d6

AndydeCleyre mentioned this pull request Mar 17, 2022

Second level dependencies missing on for local file packages on >= 6.3.1 #1505

Closed

AndydeCleyre force-pushed the combine-without-copy branch from 95d1553 to ad3dc21 Compare March 31, 2022 19:41

AndydeCleyre mentioned this pull request Mar 31, 2022

Extra requirements aren't properly pulled in by pip-compile #1606

Closed

richafrank force-pushed the combine-without-copy branch from ad3dc21 to 8aff146 Compare May 8, 2022 18:04

richafrank added 2 commits May 24, 2022 21:18

Failing test for case of resolver combining extras after the first

46ec202

round of resolution. By this point, we've frozen the dependencies.

richafrank and others added 8 commits May 24, 2022 21:18

Integration test for jazzband#1505

34a2416

Integrate microcat49's attribute copying fix for lost local req info

41b4d37

Fixes jazzband#1054 Co-authored-by: Andrew <harta5039@gmail.com>

Added test for lost local req info

b11a5c1

re-resolve dependencies of combined install_requirement

a9e647a

Make a new InstallRequirement instance instead of mutating a copy

f35a7ba

when combining instances

Removed unused parameter

351e126

No longer need the name proxy, now that we're not mutating InstallReq…

b065118

…uirement instances in combine_install_requirements

Simplify unit test, so it's not so implementation-specific.

d8b9135

richafrank force-pushed the combine-without-copy branch from 8aff146 to d8b9135 Compare May 25, 2022 01:18

richafrank mentioned this pull request May 30, 2022

Support for the importlib.metadata metadata implementation #1632

Merged

4 tasks

atugushev reviewed May 30, 2022

View reviewed changes

tests/test_cli_compile.py Outdated Show resolved Hide resolved

atugushev approved these changes May 30, 2022

View reviewed changes

Test the entire resolver output

97471ea

to ensure nothing else was emitted

atugushev merged commit e2d0fae into jazzband:master May 31, 2022

richafrank deleted the combine-without-copy branch May 31, 2022 23:58

richafrank mentioned this pull request Jun 1, 2022

can pip-compile handle multiple extras of the same package properly #1511

Closed

atugushev changed the title ~~Instantiate a new accumulator InstallRequirement for combine_install_requirements output~~ Instantiate a new accumulator InstallRequirement for combine_install_requirements output Jun 7, 2022

atugushev added this to the 6.7.0 milestone Jun 26, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Instantiate a new accumulator `InstallRequirement` for `combine_install_requirements` output #1519

Instantiate a new accumulator `InstallRequirement` for `combine_install_requirements` output #1519

richafrank commented Oct 28, 2021 •

edited by atugushev

Loading

richafrank Oct 28, 2021

richafrank commented Oct 28, 2021

richafrank commented Jan 8, 2022

mmerickel commented Feb 24, 2022

Jorricks commented Mar 11, 2022 •

edited

Loading

frejonb commented Apr 4, 2022 •

edited

Loading

atzannes commented May 25, 2022

atugushev left a comment

Instantiate a new accumulator InstallRequirement for combine_install_requirements output #1519

Instantiate a new accumulator InstallRequirement for combine_install_requirements output #1519

Conversation

richafrank commented Oct 28, 2021 • edited by atugushev Loading

Contributor checklist

Maintainer checklist

richafrank Oct 28, 2021

Choose a reason for hiding this comment

richafrank commented Oct 28, 2021

richafrank commented Jan 8, 2022

mmerickel commented Feb 24, 2022

Jorricks commented Mar 11, 2022 • edited Loading

frejonb commented Apr 4, 2022 • edited Loading

atzannes commented May 25, 2022

atugushev left a comment

Choose a reason for hiding this comment

Instantiate a new accumulator `InstallRequirement` for `combine_install_requirements` output #1519

Instantiate a new accumulator `InstallRequirement` for `combine_install_requirements` output #1519

richafrank commented Oct 28, 2021 •

edited by atugushev

Loading

Jorricks commented Mar 11, 2022 •

edited

Loading

frejonb commented Apr 4, 2022 •

edited

Loading