Add options for including build dependencies in compiled output #1681
Conversation
|
@apljungquist could you also document the new feature? |
tests/test_cli_compile.py
Outdated
| "--no-annotate", | ||
| "--no-emit-options", | ||
| "--no-header", | ||
| "--build-system-requires", |
There was a problem hiding this comment.
We also need a test case showing how to only extract the build system requirements without the runtime requirements.
Oh, and how about being able to separately request sdist build requirements, wheel build requirements and editable install requirements?
There was a problem hiding this comment.
We also need a test case showing how to only extract the build system requirements without the runtime requirements.
Makes sense given that the CLI flag presents this as an option. Will fix.
Oh, and how about being able to separately request sdist build requirements, wheel build requirements and editable install requirements
Good catch, I had missed that build systems can state additional requirements for sdist and wheel. But I cannot find any information on editable installs, do you remember where you learned about it?
I am however unsure about the use case for locking sdist and wheel requirements separately. What do you have in mind?
If there is no clear use case, is it worth increasing the size of the UI and the test matrix?
There was a problem hiding this comment.
But I cannot find any information on editable installs, do you remember where you learned about it?
https://peps.python.org/pep-0660/#get-requires-for-build-editable
I am however unsure about the use case for locking sdist and wheel requirements separately. What do you have in mind?
The fact that they are dynamically calculated suggests that they may be different or even conflicting.
Also, building a wheel from a Git checkout may be different than building a wheel from an sdist due to possible inconsistencies in what files are bundles in a wheel.
I imagine the same may happen with the editable checkout. Also, I could see a case when the editable install deps include some development helpers that further restrict the constraints.
If there is no clear use case, is it worth increasing the size of the UI and the test matrix?
It's probably worth thinking about edge cases. One case that I just realized is that the build deps can have env markers (both direct and transitive). This means that there could be different constraints produced depending on which OS/arch/Python version is used to generate them. This is an interesting case to explore. I currently experiment with having a matrix of constraint files per env combo, for runtime. Having that for build deps will probably be useful for non pure Python projects...
There was a problem hiding this comment.
We also need a test case showing how to only extract the build system requirements without the runtime requirements.
Makes sense given that the CLI flag presents this as an option. Will fix.
Now that I read the code again I am not sure what I was referring to. Instead it seems to me that we would have to add another flag to accomplish this e.g. --no-install-requires. This opens up questions like how such an option should interact with the --extra argument and I am starting to think that this should be treated as a separate issue.
I am also thinking about what would be a good interface for telling pip-compile what build requirements to include. I'm leaning towards replacing the previously proposed --build-system-requires flag with an --all-build flag and a --build parameter that works similarly to the --all-extra flag and --extra parameter respectively but takes editable, sdist or wheel as arguments.
Whatever the shape of the interface I think I need to spend some time exploring how the hooks work and how they can be tested. I will try to make some time for this over the coming weeks.
There was a problem hiding this comment.
Extras are tightly coupled with the normal runtime deps. So if they aren't used, they shouldn't be pulled in...
There was a problem hiding this comment.
I have implemented support for this but I am not sure I like it because it adds complexity to both the public interface and the implementation and it is not clear to me that anyone will actually use the feature.
|
@apljungquist would you mind addressing the comments? |
|
Thanks for the reminder. I will take care of it this weekend.
…On Tue, Nov 22, 2022, 20:22 Sviatoslav Sydorenko ***@***.***> wrote:
@apljungquist <https://github.com/apljungquist> would you mind addressing
the comments?
—
Reply to this email directly, view it on GitHub
<#1681 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABQ7T76VKSB7JW5HEYTPIDLWJUMRBANCNFSM6AAAAAAQPNJLRQ>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
@apljungquist @webknjaz Just to let you know that I'm really looking forward to this - thanks a lot! Please let me know if there is anything I can help with. |
Thanks for the encouragement. Can I ask how you plan to use it? |
@apljungquist In nix-community/dream2nix#401 we're aiming for fully reproducible (build) environments. |
tests/test_data/packages/small_fake_with_build_deps/backend/backend.py
Outdated
Show resolved
Hide resolved
|
It was confused that GH was talking about some mysterious required statuses, which is why I was pushing all the buttons to closer/reopen the PR and such. At some point, it occurred to me that it doesn't track my prior approval for some reason (eventual consistency bugs?) and I re-approved, and then I saw GH doing what it should've done earlier — the PR got queued up. It should merge this, once the testing completes. |
|
Oh, and additionally, Jannis dropped the requirement for the branches to be up-to-date, which doesn't make sense with merge queues. |
|
🎉 @apljungquist thanks again for such an involved and well-thought contribution that took over a year to get polished! |
|
Thanks. And thank you for patiently helping me! |
|
@apljungquist Hm.. |
|
@apljungquist so it appears that using cc @chrysle @merwok — in case any of you feel like going down the rabbit hole, and tackling a rather tricky problem here. |
| env.install(builder.build_system_requires) | ||
| env.install(builder.get_requires_for_build("wheel")) |
There was a problem hiding this comment.
@apljungquist looks like these installs have to be called with os.environ['PIP_CONSTRAINT'] set to a generated file with -P values dumped in order to install the backend deps we desire.
There was a problem hiding this comment.
@webknjaz Checking I understand this comment as being related to this note in the README: "- pip must be used with the PIP_CONSTRAINT environment variable to lock dependencies in build environments as documented in #8439."
Rather than just installing the requirements as listed in the metadata, you're suggesting that:
- The current value of `os.environ["PIP_CONSTRAINT"] should be saved to avoid irrevocably clobbering it
- For each
env.installstep, in addition to the requirements being passed explicitly, they also need to be added to a temporary file referenced fromPIP_CONSTRAINTto force the dependencies to actually be respected - Once the installs are done,
PIP_CONSTRAINTshould be reverted back to its original value.
To try to get CI going, I'm going to try just setting PIP_CONSTRAINT in the failing test case, which would mean this discovery still needs its own bug report.
There was a problem hiding this comment.
Attempted workaround in bc3f17b still didn't get the affected test case passing when testing locally, but I'm trying it in CI anyway in case the local failure was specific to Python 3.12.
There was a problem hiding this comment.
I tried a different workaround, specifying a known version of setuptools in the test metadata: c2bd925
There was a problem hiding this comment.
Even with both attempted workarounds combined, the build dependency compilation test still fails: https://github.com/jazzband/pip-tools/pull/2105/files#diff-99b06d953a9978ee564df2e67547b8a76f3b1ab116ab7c89a9005b9239f2a904
This is with setuptools 70.0.0 explicitly requested in the test project config, and with a constraints file constraining both setuptools and wheel set via the pip-tools CLI option and via the PIP_CONSTRAINT environment variable.
There was a problem hiding this comment.
- For each
env.installstep, in addition to the requirements being passed explicitly, they also need to be added to a temporary file referenced fromPIP_CONSTRAINTto force the dependencies to actually be respected
I think that the [build-system]requires value is there already as an explicit install request. But the constraints coming from other places like the CLI command or other/additional input files are not taken into account. I think they should be dumped into a temporary file and that file would need to be set in PIP_CONSTRAINT around such invocations.
Yes to points (1) and (3), though.
There was a problem hiding this comment.
@webknjaz Your suggestion to set the PIP_CONSTRAINT environment variable appears to work, but another problem surfaced; the failing test in question doesn't use build isolation (it succeeds when that is explicitly requested) – what do you suggest should be done in such a case?
There was a problem hiding this comment.
@chrysle oh, I missed this. Could you clarify what you mean by requesting it? Are you talking about calling some tooling outside the test suite or making said change in the tests? Perhaps a PR would make it more obvious what you mean.
There was a problem hiding this comment.
No problem! I was referring to the --build-isolation boolean flag. The failing test explicitly turned that off:
pip-tools/tests/test_cli_compile.py
Line 3438 in 5330964
However, this means that _create_project_builder yields build.ProjectBuilder relying on the outside environment and the (latest) version of setuptools that was installed together with pip-tools - no doubt to improve performance - but the test naturally fails.
Lines 192 to 194 in 5330964
Using build isolation makes the test pass after setting PIP_CONSTRAINT as you suggested.
There was a problem hiding this comment.
@chrysle thanks for the explanation! Yes, you're right that this was a trade-off made in the name of performance. It sounds like we need a separate regression/acceptance test checking this corner case in isolation. It'll be slower but, hopefully, one test won't make a big difference.
|
Nice digging, and interesting finds. I won't have time to work on this for a while though, so 🤞 chrysle or Éric are interested. |
|
I’m not familiar with any resolver and haven’t contributed to pip-tools’ code, so I will be declining the offer. |
You mean, against a newer version ( |
|
@apljungquist thanks for the heads up. Hopefully, someone else will have time, maybe. I thought that I might but never got to it. @chrysle yeah, I suppose I made a typo. |
|
@webknjaz I'm at it; could you take a look at my above comment, please? |
Fixes #1396. Add options for including build dependencies in the output from pip-compile. This allows users to improve the reproducibility of their CI since they can ensure that the same versions of build dependencies are used to build their package every time.
Note that build dependencies of build dependencies are not included in the output. This is one reason why builds will still not be completely reproducible.
Contributor checklist
Maintainer checklist
backwards incompatible,feature,enhancement,deprecation,bug,dependency,docsorskip-changelogas they determine changelog listing.