Lockfiles: what resolve to use with MyPy and Pylint operating on python_library targets?

[Eric-Arellano]

Thus far, multiple user lockfiles has been following the approach of https://docs.google.com/document/d/1sr1O7W9517w8YbdzfE56xvDb72LafK94Fwxm0juIRFc/edit: only "root" targets like python_tests and pex_binary have a resolve field to choose which resolve/lockfile to use. Dependencies are irrelevant to choosing a resolve, e.g. if a python_tests target depends on another python_tests target (weird to do, but legal), we simply go with the resolve of the target currently being built with ./pants test.

This scheme works great for ./pants package / pex_binary and ./pants test / python_tests. But it's foiled by MyPy and Pylint, which both require resolving 3rdparty dependencies and can treat python_library targets as the root, e.g. ./pants typecheck src/python/pants/util/strutil.py.

Naive, unaccpetable solution: python_library has resolve field

It would be unambiguous what resolve to use. However, we'd need to validate that the entire closure is using a consistent resolve, meaning all dependencies and dependees. (It's only possible to use a single resolve in a build)

This means that a python_library can only ever work with a single resolve, which is extremely limiting. For example, if strutil.py is set to default, then anything that uses strutil.py transitively must use the same resolve. Especially given that we will require pex_binary targets w/ platforms set to use a platform-aware lockfile (#12612), that is a non-starter.

Possible solution? Use dependees roots

Find all root targets that depend on the python_library and use their resolve.

What to do if no roots

Use the default resolve.

What to do if >1 resolves

If >1 roots, and those use different resolves, then we "should" be able to use any of those resolves. All those resolves, by definition, should be compatible with the library and its deps. If not, our invalidation check will warn/error.

We probably want to avoid choosing platform-aware resolves (#12612) when possible, as those might only work with platforms different than the host.

Performance concern

Computing dependees can be slow, as it requires scanning the whole repository. I don't know of a way to improve this.

A simple hack is that if the # of resolves (from [python-setup]) is <= 1, then we can short circuit.

[jsirois]

Naive, unaccpetable solution: python_library has resolve field

For completeness, clearly you could just pluralize the field and then check any given resolve has all file dependencies whose resolves contain it. This reflects libraries in-practice. They generally need to work with many resolves when you ship to PyPI. Pants is a good anti-example since it ships an API alongside a binary and really only works with 1 resolve since its all pinned out on PyPI. Plugin authors are then forced into that 1 resolve FAPP.

[Eric-Arellano]

Oh, interesting! Something like a compatible_resolves field for python_library, which is a list. But pex_binary and python_tests still choose a single resolve via resolve field.

Thanks for that metaphor of libraries vs. binaries in general, that makes sense.

I want to think more about that, but this seems like a solid improvement from the "find dependees" proposal. Thank you!

[jsirois]

To be clear, the metaphor is not mine. This is standard parlance in all lockfile communities I've encountered and the language is uniformly, lockfiles are for binaries and never libraries. Here resolves ~= locks and the same considerations are in play.

[stuhood]

As discussed in the thread on this topic, there might be an alternative to the dependees case (or perhaps just an optimization?): if we have a list of known resolves for a repository, we might be able to skip the dependees search by filtering the resolves to see which are relevant. In effect: the lockfiles for resolves might contain lists of their member 3rdparty targets, and act like a cache for resolve lookups.

[Eric-Arellano]

In effect: the lockfiles for resolves might contain lists of their member 3rdparty targets, and act like a cache for resolve lookups.

Ah, yeah, that type of check is possible via #12610. Interesting.

It does require a lot of magic fwict, though, such as a user figuring out how to force a particular resolve to be used. For example, if the target doesn't have any roots and you don't want it to use default, you'd have to introduce a root purely for the sake of changing the resolve. That's not very intuitive.

--

So far, John's proposal of compatible_resolves makes a lot of sense to me as being conceptually simple, and thus easy to reason about in what can be a very confusing topic. It does risk more boilerplate, but that boilerplate may be justified for the clarity it brings.

(We can also reduce the boilerplate with the bigger project of re-envisioning targets, so it's easier to say e.g. "this whole source tree uses these resolves")

[jsirois]

Presumably you reduce boilerplate by having a default_compatible_resolves option for defining up in pants.toml. By default say that's ALL. Then you only need to ad metadata to library targets that are not compatible with all resolves. Presumably, that cuts down on boilerplate significantly in many cases?

[chrisjrn]

I definitely like the idea of specifying multiple compatible resolutions here.

I'm trying to wrap my head around how we deal with a library being used as a dependency for another target here. If the library has a requirement A>=3,<=5, and the dependent target has requirement A==4, how do we deal with the case where our supplied resolutions for the library do not match the dependent target? Is this a case where we just fail and have to generate another resolution and add the that to the list of compatible resolutions for the library?

[Eric-Arellano]

If the library has a requirement A>=3,<=5, and the dependent target has requirement A==4, how do we deal with the case where our supplied resolutions for the library do not match the dependent target? Is this a case where we just fail and have to generate another resolution and add the that to the list of compatible resolutions for the library?

Requirements are modeled as python_requirement_library targets. The lockfile would need to have been generated with the target for both A>=3,<=5 and the target for A==4. During lockfile generation, that will fail with "double requirement", unless they used env markers like ; python_version > '3' (think back to your PR that supports this in requirements.txt files).

That is, the lockfile would either need to include both targets; or if that's not possible with the double requirement, use two distinct resolves.

[Eric-Arellano]

I think compatible_resolves is compelling enough to implement. Its conceptual simplicity is the key highlight for me. After a proof of concept, we can confirm it does the job effectively enough.

Thanks again for the feedback everyone!