-
-
Notifications
You must be signed in to change notification settings - Fork 636
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Lockfiles: when checking for staleness, only use current context's interpreter constraints #12542
Comments
I'm trying to figure out what "compatible" means here. The definition I'm thinking of says:
Does that line up with your understanding of the constraints? |
I was thinking the mathematical definition of subset: the context's ICs must be completely contained by the lockfiles's ICs. That is, every patch version in the context must be in the lockfile's possible patch versions. It need not be a "proper subset", it's valid if the context ICs == lockfile ICs. |
@Eric-Arellano Thoughts on including platforms as part of the requirements here? Or is that for a later date? |
If we go with Poetry, then it doesn't matter what platform you generate from. The lockfile should handle windows, Linux, and macOS. No need to invalidate based on platform. (Still thinking about John's question on handling --platform when set in a pex_binary. Will put up some thoughts, but tldr is I think this issue will not need to do anything.) |
Cool, I'll ignore that for now. |
(Completion of this task is blocked on merging #12448) |
Background
For tool lockfiles, #12448 uses the rules to get the
PythonLockfileRequest
for the tool to check if the current lockfile has become stale.Some tool lockfile requests are tricky to set up because interpreter constraints depend on the user's code, such as how the ICs used for Flake8 and Pytest depend on what the user has. This has a severe performance impact: when running any of these tools, like running tests on a single file, we must first check if the lockfile is stale by consulting the entire repository to determine its interpreter constraints. That is slow to scan/compute, and means tests will be invalidated much more infrequently.
Proposal
Instead, when checking a lockfile for staleness, we can simply check that the current context's interpreter constraints are compatible with the lockfile's constraints, rather than identical. This would avoid entirely the performance concern.
Tips for implementation
The main challenge will be adding code to
interpreter_constraints.py
to compare if two interpreter constraints are compatible with each other or not: checking if one is a subset of the other.Likely the easiest way to implement this is to leverage that we have users define for us already what the finite universe of Python interpreters should be, e.g. Pythons 3.5-3.10. You can use this function to generate all possible versions:
pants/src/python/pants/backend/python/util_rules/interpreter_constraints.py
Lines 179 to 183 in ecf85d7
(Even better, a PR I'm about to put up to add
InterpreterConstraints.flatten
adds a PR that computes all valid Py2 and Py3 versions for the ICs. You can use that.)Then, use set methods to check that one is a subset of the other.
--
We'll also need to refactor #12448 to no longer use the rule to determine the lockfile request, but instead compute what the request would be for that current context (e.g. lockfile for just that test).
The lockfile header will also probably need to start preserving the original
interpreter_constraints
string, and we'll need a way to serialize/deserialize that.The text was updated successfully, but these errors were encountered: