[internal] Add experimental ./pants lock to generate a lockfile with pip-compile

[Eric-Arellano]

With this goal, you can run ./pants lock 3rdparty/python:: to create a lockfile using pip-compile. For example:

❯ ./pants lock 3rdparty/python:humbug
18:31:35.96 [INFO] Completed: Generate lockfile for 1 requirements: humbug==0.1.9
18:31:35.96 [INFO] Wrote lockfile to 3rdparty/python/lockfile.txt

#
# This file is autogenerated by pip-compile with python 3.9
# To update, run:
#
#    pip-compile --allow-unsafe --generate-hashes --output-file=3rdparty/python/lockfile.txt requirements.in
#
bugout==0.1.12 \
    --hash=sha256:0dc968e520c19c5db824acb35997ecc0b0227b1bf4a1f86a9c2a8cdc6d040a64 \
    --hash=sha256:999ad28771cdcfdc0e1e54aa5f5bf478d67b9f1a1cbc4b9df03ad98d46887903
    # via humbug
certifi==2021.5.30 \
    --hash=sha256:2bbf76fd432960138b3ef6dda3dde0544f27cbf8546c458e60baf371917ba9ee \
    --hash=sha256:50b1e4f8446b06f41be7dd6338db18e0990601dce795c2b1686458aa7e8fa7d8
    # via requests
chardet==4.0.0 \
    --hash=sha256:0d6f53a15db4120f2b08c94f11e7d93d2c911ee118b6b30a04ec3ee8310179fa \
    --hash=sha256:f864054d66fd9118f2e67044ac8981a54775ec5b67aed0441892edb553d21da5
    # via requests
humbug==0.1.9 \
    --hash=sha256:8771b54783f7f702cb23605df7a4d6ae81cda9c8167eccd09ed05aff5e56d5ab \
    --hash=sha256:e4c4019af50b1e461eb62346b0e732f22632e0654d7602aada82dcca8ff61dd7
    # via -r requirements.in
idna==2.10 \
    --hash=sha256:b307872f855b18632ce0c21c5e45be78c0ea7ae4c15c828c20788b26921eb3f6 \
    --hash=sha256:b97d804b1e9b523befed77c48dacec60e6dcb0b5391d57af6a65a312a90648c0
    # via requests
...

This is still very experimental, and there are many unresolved questions. The intent with landing this is to have a foundation to start iterating on and answering design questions.

Why pip-compile

There are several known ways to generate a lockfile in Python:

1. Create a virtual env, then use pip freeze.
2. Poetry or Pyenv.
3. pip-compile.
4. We could teach Pex how to do it.

Pip-compile offers a robust out-of-the-box solution with several benefits:

• Stick to pip ecosystem and the pip resolver, unlike Poetry and Pyenv.
• Does not install wheels like pip freeze, only downloads and does the minimum work necessary.
• Explains in the lockfile where each requirement comes from, which is very useful.

Lockfiles vs. constraints files

Note that this generates a "lockfile", not a "constraints" file, which we have confusingly been calling the same thing. Among other differences, a lockfile can include extra like requests[security], which choke with a constraints file.

Next steps

• Teach Pants to consume a lockfile, rather than constraints file.
• Fix the specs semantics for ./pants lock so that the lockfile is based on what gets used by your code, rather than what reqs are defined in your universe.
  • This will be important when we have multiple user lockfiles.

[ci skip-rust]

[Eric-Arellano]

Pretty sure the comment's proposed semantics are more correct

EDIT: we should probably defer this discussion for the dedicated PR. I'm thinking this PR should mostly be about simply using pip-compile. That way, we can have a more robust discussion there where I can give the full context and design process.

[benjyw]

Yes, I tend to agree.

[stuhood]

+1

This probably doesn't need to be exposed as a goal at all (in this PR)...?

[Eric-Arellano]

It's helpful for me to have a goal to run when iterating. But feature gated behind the experimental backend

[Eric-Arellano]

I think this will come from the transitive closure of the code using the 3rd party reqs, with a default to [python-setup].interpreter_constraints.

[stuhood]

Correct. With one permutation (templated copy á la lock-%p-%i) per platform (%s) and interpreter constraint (%i).

[Eric-Arellano]

FYI @wilsonliam and @chrisjrn , who will likely work on some tickets related to lockfiles.

Also FYI @jsirois

[benjyw]

Yes, I tend to agree.

[stuhood]

Thanks!

[stuhood]

One approach would be to have a resolve(name=..) target type, with optional parameters for overriding some default lockfile templates. To use it, you would declare one in some directory, and then when it was actually consumed by a *_binary target, the lockfiles would be produced next to it at the templated paths (templated by platform * interpreter_constraint * etc).

A downside of that modeling though is that it means that whatever our "default" global resolve is would be a bit magical. We don't want folks to have to declare one of these just to get started with Pants. Not sure how to fix that mismatch... maybe with a setting instead...?: [resolves] resolve_mapping = "{'default': '3rdparty/python/lock%p-%i', 'custom_for_aws': '3rdparty/python/aws_lock%p-%i'}".

[stuhood]

+1

This probably doesn't need to be exposed as a goal at all (in this PR)...?

[stuhood]

Correct. With one permutation (templated copy á la lock-%p-%i) per platform (%s) and interpreter constraint (%i).

[stuhood]

Is this output format something that PEX can consume...? Should PEX be delegating to pip_tools in order to put the fetched stuff in its cache...?

[stuhood]

Oh, and speaking of caching: should this be configured to use a named_cache (until PEX is invoking it)?

[Eric-Arellano]

Interesting on caching. The --output-file is a literal requirements.txt file.

I haven't investigated yet how pip-tools is doing the fetching and where it gets cached, but fwict it delegates to Pip. That makes sense to try to share that cache with Pex, and at a minimum to use a dedicated named_cache. I'll add a TODO in the next PR.

[stuhood]

Fix the specs semantics for ./pants lock so that the lockfile is based on what gets used by your code, rather than what reqs are defined in your universe.

• This will be important when we have multiple user lockfiles.

I think that the lock goal should not have any particular spec semantics: there will need to be some underlying reusable @rules (which will be used by the lock goal, but also by any @(goal_)rules consuming the resolves) that do a dance like:

• Figure out which requirements are reachable from the specs, and which resolves are reachable from the specs.
• See whether any of the reachable requirements have changed since the lockfile(s) for the relevant resolves were generated; warn/error if so.
• Return an in-memory representation of the whole resolve (in the case of Python, a PEX to subset)

The first two points are the hand-waviest. Figuring out how to do this without full runs of dependees is going to be tricky.