[RFE] Output to constraints.txt by default

[webknjaz]

What's the problem this feature will solve?

Currently, pip-compile outputs to requirements.txt but I think that it's misleading as the output of the tool better maps to the concept of lockfiles (aka constraints in pip-speak).

Describe the solution you'd like

I'd like pip-tools to be consistent with its own recommendations to use pip install -c.

Alternative Solutions

..not doing so?

Additional context

Last year, I wrote this explanation about the misconceptions of pinning: #1326 (comment). And I figured that pip-tools itself is sending mixed messaging regarding what it does. I believe we can correct at least some of it.

[chrysle]

Please ignore my comment, I think I had a short-circuit. I forgot that --strip-extras will become the default soon (should we hold this until 8.0.0?), so I was anxious this might disallow specifying extras. And editable requirements will get unsupported with this, won't they?

[webknjaz]

Why? This FR is just about changing the default output filename. Literally a string in one variable somewhere.

[chrysle]

Surely I'm stupid, but I read the pip documentation on constraint files
somewhen and was sure they have some "feature losses" in comparison to requirements files, e.g. they don't support extra specification.

[chrysle]

This illustrates my case:

$ echo "pylint[testutils]" > requirements.in 
$ pip-compile -o constraints.txt requirements.in 
#
# This file is autogenerated by pip-compile with Python 3.9
# by the following command:
#
#    pip-compile --output-file=constraints.txt requirements.in
#
astroid==3.0.2
    # via pylint
dill==0.3.7
    # via pylint
gitdb==4.0.11
    # via gitpython
gitpython==3.1.41
    # via pylint
isort==5.13.2
    # via pylint
mccabe==0.7.0
    # via pylint
platformdirs==4.1.0
    # via pylint
pylint[testutils]==3.0.3
    # via -r requirements.in
smmap==5.0.1
    # via gitdb
tomli==2.0.1
    # via pylint
tomlkit==0.12.3
    # via pylint
typing-extensions==4.9.0
    # via
    #   astroid
    #   pylint
$ python -m pip install -c constraints.txt tomli
DEPRECATION: Constraints are only allowed to take the form of a package name and a version specifier. Other forms were originally permitted as an accident of the implementation, but were undocumented. The new implementation of the resolver no longer supports these forms. A possible replacement is replacing the constraint with a requirement. Discussion can be found at https://github.com/pypa/pip/issues/8210
ERROR: Constraints cannot have extras

[chrysle]

I'm fine with your suggestion. I understand that what I tried is exactly what you disapproved of in the linked comment.
Still, I think defaulting to this should go together with stripping extras by default. Thougts?

[webknjaz]

@chrysle I think that stripping extras by default is already being worked on (#1954 / #1613). This issue is merely about the default filename. And maybe, the filenames in the docs. Otherwise, there's a confusion of what's being output.

The thing is that the output of pip-tools is always constraints. Sometimes, they can have extras, yes (hopefully, the default would flip sooner rather than later). But it's still constraints, a lockfile, if you will. It's good that the use of extras has been deprecated, though, because it doesn't make sense in constraints that are meant to be taken into account by the dependency resolver only when a package is met during the resolution. I'm convinced that by using the requirements terminology in these places, we're spreading more misconceptions which is what I aim to address here.

As for the "editable requirements will be unsupported", I think there were a few corner cases with that and we should definitely not add -e into the output, encouraging the users to use both -r and -c. The former would be able to have -e but not the latter.

[AndydeCleyre]

I find it confusing that in some contexts "constraints" are required, while in others they are limits but not necessarily requirements. Usually I think of the term meaning the latter, as I don't know an alternative word for that.

[mmerickel]

Cross linking here #1755 I have never found a solution for the use case in my ticket. --strip-extras is only half the battle, because we are also finding the source dependencies via editable references to packages in the repo which cause issues.

It would really be helpful if pip-tools offered a way to generate something that was actually constraints compatible. From my original layering example in that ticket, there is no way to cascade through something that works right now if one of the .in files contains -e editable references.

# base.txt
pyramid == 2.0

# src.in
-c base.txt
-e file:.

# dev.in
-c base.txt
-c src.txt  # <------------ note this is the output pin from src.in and is not constraint compatible

pyramid-debugtoolbar
black
isort
flake8
debugpy

$ pip-compile --strip-extras src.in
$ pip-compile --strip-extras dev.in

I'd like to see something like --output-constraints src.constraints.txt and then I could have dev.in look like:

-c base.txt
-c src.constraints.txt  # <------------ new constraint-compatible file

pyramid-debugtoolbar
black
isort
flake8
debugpy

[mmerickel]

Even better, fwiw, is that --strip-extras only applies to the file output via --output-constraints and they are left in the default src.txt.

[webknjaz]

I find it confusing that in some contexts "constraints" are required, while in others they are limits but not necessarily requirements. Usually I think of the term meaning the latter, as I don't know an alternative word for that.

@AndydeCleyre both requirements and constraints are pip-specific contexts. The latter is less known and is a set of additional restrictions that the dependency resolver applies only if it sees the listed projects during the process of resolution. Requirements, OTOH, are direct user requests to install something. So the difference is between "I want these installed" and "if these are going to end up selected for installations, here's the limits". I believe that pip-tools contributed to confusion by substituting the concepts in the readme/docs, combined with people not knowing at all that constraints are a thing. And that's my primary motivation — disentangling the concepts and educating the users. The filenames don't matter to pip — the contents just need to be text in the right format. Pip will work with pip install -r constraints.txt or pip install pkg -c requirements.in — these are confusing, should not be used with such names and cause cognitive dissonance, but they would work for as long as their contents look right to pip. Pip-tools (and dependabot) often advertises requirements.in and requirements.txt, while suggesting people to use the later as constaints as in pip install -r requirements.in -c requirements.txt — and I think this is the only right way to use these file pairs but I despise that the filenames create such confusion. I know that it's easy to detect such pairs for Dependabot which is why it's the only supported use in the context of that tool, but outside of it, I'd really like to push for naming things right and then integrating them correctly. Some time ago I produced this rant related to a series of misconceptions around constraints/lockfiles — #1326 (comment) — and I think this misnaming is a contributing factor.

Technically, if organized right, sets/matrixes of constraint files are essentially a very much usable lockfile. In fact, that's how rye uses pip-tools today, per my understanding (I still haven't yet used rye myself, so this statement is probably a bit too simplistic).

So I want to reinforce the understanding that pip-tools' outputs are better treated as lockfiles while the inputs are for direct deps. Misusing constraints as requirements can lead to cases when the users are attempting to install things that are inherently incompatible with their environments, just because they were generated in (and for) other envs. But using them as constraints would not make pip error out if there's some entries that never hit the depresolver.

[webknjaz]

@mmerickel I think that in general, editables shouldn't hit constraints themselves, only their runtime deps. Also, some of the concerns around having dependencies that don't automatically pull in the package/wheel deps are going to be addressed by PEP 735 "Dependency Groups", I think.

[mmerickel]

@webknjaz I understand editables shouldn't hit the constraints but I'm not following your suggestion. The issue here is that pip-tools doesn't provide a way to generate constraint files properly if -e is used in the .in file (which is a reasonable thing to do). pip-tools really needs to provide a way to generate a real constraint file from the .in.

Today I have to take the output of pip-compile and create my own constraints via:

grep -v '^-e ' src.txt > src.constraints.txt

Despite using the rest of the recommended flags like --no-emit-find-links --no-emit-index-url --strip-extras.

[webknjaz]

@mmerickel yes, that's basically what I meant to communicate. I also had to post-process the output sometimes. I think, this should be fixed (perhaps with a CLI option to exclude such entries or something).

By the way, you can put the CLI flags into a config file since v6.14 so that's going to make the invocations easier to follow.

[mmerickel]

Thanks for the tip about the config file.

@webknjaz Is it the goal of pip-tools that the output is used directly with pip install -r <pip-compile-output> or as constraints via pip install -r my.in -c <pip-compile-output>? It kind of feels like the latter if y'all are moving to a default of --strip-extras and --no-emit-index-url but that's definitely a change in philosophy whereas historically it has been the former.

[webknjaz]

historically it has been the former.

I feel like historically, it's been a combination of both mentioned in different parts of readme and thus it's very confusing. I'd like it to be -r + -c by default because this is the most predictable and straightforward configuration that should resolve at least a part of the confusion that exists today.

P.S. I don't know what the original goal looked like as the original author started out. It probably evolved over time.

[mmerickel]

It's very helpful perspective to use pip-tools in that way. I've been using it for years without that in mind and that would put me very much in the camp of expecting --no-emit-index-url --no-emit-find-links --strip-extra --no-emit-editable etc etc all being the default behavior of pip-compile.

I will also point out that this philosophy doesn't fit with pip-sync as I see it right now. I don't use it, but there isn't even an option in pip-sync to pass a constraints file in the latest version 7.4.0. The marketing pitch for pip-sync is that you can just use the files output from pip-compile.

[webknjaz]

Fair. Honestly, I don't know if I ever used pip-sync myself. I always go for pip-compile + plain pip install. And I've been experimenting with a more involved integration with tox for maximizing reproducibility — #826 (comment).

I don't think that this change would necessarily break pip-sync irrevocably in all cases, but perhaps some. I'm hoping we can rely on the existing test coverage to reveal that. But that's for pointing it out!

[mmerickel]

If pip-tools goes all in on generating constraints files, it would just make me over the moon to have an option to ignore environment markers. Just generate a monster constraints file of the entire possible dependency tree. Yes it might fail for some people and they can deal with platform-specific constraint files or have some options to ignore certain dependencies but gosh I'd love to have the option for a single cross-platform constraints file. Right now I have to lock it once on my mac, then spin up the x86_64 docker container and lock it again and it's awful tedious.

Huge +1 from me to just make great constraints files.

[mmerickel]

I do use the compiled files for other purposes which is why it's nice to have both outputs though. This is my ideal flow in a nutshell:

1. Desire to have a large constraints file across my project: pip-compile -r src.in -r dev.in -r ... > constraints.txt
2. A frozen list of all dependencies to build and create wheels: pip wheel --find-links wheels -w wheels -r src.in -r dev.in -c constraints.txt
3. A way to install what I need in dev: pip install --find-links wheels -r src.in -r dev.in -c constraints.txt
4. A way to install what I need in prod: pip install --find-links wheels -r src.in -c constraints.txt

[mmerickel]

The main thing I lose from something like this is a list of "these are the exact packages to install for src.in" in a static form. I'm losing that info and kicking it to pip install to redo all of the resolution and figure out that list. I have steps in my pipeline where I'd like to have that list so that I can pick out those libraries and build wheels for them similar to the pip wheel command above but working around limitations like that pip wheel doesn't build wheels in parallel.

An equivalent in another world is the yarn.lock which can be used to install everything, without the input package.json. This is similar to how pip-tools operates in the -r philosophy.

[webknjaz]

Just generate a monster constraints file of the entire possible dependency tree. Yes it might fail for some people and they can deal with platform-specific constraint files or have some options to ignore certain dependencies but gosh I'd love to have the option for a single cross-platform constraints file.

In #826, I realized that this is basically unrealistic with the current ecosystem. And it's because it's not always the right thing to do (#1326 (comment)) but mostly about the impossibility to cross-compile (there's bits missing in CPython stdlib and other places in the ecosystem). For the time being, there is no way to compile sdists for some other environment. So no, it's not possible to just ignore env markers, it'd just explode during the dependency resolution, which is why I used GHA to generate an entire matrix of them (#826 (comment)).

It's not about adding env markers into the constraint file, it's about being unable to resolve deps for trees involving platform-specific projects when there's sdists in the tree. FWIW pip-tools piggy-backs on pip's own depresolver, not adding much on top.

Besides, we probably shouldn't be reinventing the wheel while the effort to standardize the lockfiles is being debated on discuss.python.org as we speak (you should watch that discussion as it unfolds!).

The main thing I lose from something like this is a list of "these are the exact packages to install for src.in" in a static form. I'm losing that info and kicking it to pip install to redo all of the resolution and figure out that list.

I don't think you lose the direct deps since they're listed in .in files that you pass via -r. And pip show will display if packages are deps of something else or pulled in directly. Also, passing -c lets pip's depresolver discard unusable options much faster, compared to installing without constraints and having to backtrack stuff.

An equivalent in another world is the yarn.lock which can be used to install everything, without the input package.json. This is similar to how pip-tools operates in the -r philosophy.

Sounds like you want PEP 735 "dependency groups" here, rather than a distributable project deps. Except the yarn example corresponds to just one group of deps with no flexibility.

[webknjaz]

FTR the pip docs mention pip-tools in the context of constraints/lockfiles already, just without any specific suggestions around invoking it with certain args: https://pip.pypa.io/en/stable/topics/dependency-resolution/#use-constraint-files-or-lockfiles