Lockfiles: check if context's requirements are compatible with the lockfile's

[Eric-Arellano]

Currently, we expect that the consumer of a lockfile has identical input requirements to what was used to generate the lockfile. We do this by hashing the input requirements and saving that hash in the lockfile.

This works great for tool lockfiles, where the context's requirements should always be the same as what the tool lockfile was generated with. But it does not work well with user requirements, where often the context is a subset of a bigger universe of requirements. For example, a test only uses requests, which is 1 of 20 requirements in the lockfile. We should not error, so long as the version of requests is compatible with what was used to generate the lockfile.

We sort of do this right now with constraints files:

pants/src/python/pants/backend/python/util_rules/pex_from_targets.py

Lines 246 to 251 in c2f2eb0

	unconstrained_projects = name_req_projects - constraint_file_projects
	if unconstrained_projects:
	logger.warning(
	f"The constraints file {python_setup.requirement_constraints} does not contain "
	f"entries for the following requirements: {', '.join(unconstrained_projects)}"
	)

But this compatibility check is not robust enough. It's only checking that the project_name is contained in the lockfile, whereas we need to validate that the entire requirement string is compatible. In a world of multiple user lockfiles, it will be valid to have one lockfile with Django==2 and another lockfile with Django==3. We need to make sure the whole requirement is compatible.

--

To implement, we should probably start preserving the original input requirement strings in the Pants metadata header at lockfile generation time. We can then check that the context's set of Requirement objects is a set.subset() of the lockfile's Requirement objects.

[Eric-Arellano]

Counterpoint to this idea: sometimes removing a dependency does change the behavior of the build. For example, Flake8 plugins. When a user removes a Flake8 plugin from [flake8].extra_requriements, their lockfile will need to be regenerated for it to actually be removed and for Flake8 behavior to change. There, we do not want to check that the context's lockfile is compatible: we want an exact match.

For user requirements, I do think we need to use the less precise "is compatible" check. Even though it has the same risk of not handling removing dependencies, it's necessary for performance so that we don't have to compute the entire resolve's requirement strings. And there's a workaround that users can manually regenerate the lockfile, we only won't automate telling them to.

So I think the takeaway is: for tool lockfiles, stick to exact matching. For user requirements, switch to compatibility checking.

[jsirois]

Fwiw, if tool resolves also subsetted their locks, everything would work the same and be correct in the dep remove case.

[Eric-Arellano]

Ah, true. Thanks John!

[chrisjrn]

I am going to start approaching this work today. My current interpretation is that "compatible" requirements, for now, is that the context's requirement strings are a subset of the requirement strings specified in the lockfile.

In the future, we may be able to parse out version strings and do the requisite set math on requirements strings (which will also help simplify our interpreter constraints work), but right now that is out of scope.

I currently do not have plans to verify that the header is unmodified, but that seems like a reasonable fix too.

[Eric-Arellano]

Sounds good! I agree with exact matches, ideally impervious to white space differences etc

As discussed in DM, tool lockfiles should continue using a hash and exact matches.

[chrisjrn]

Here's what I intend to do then:

• Add a new field to the metadata header called tags. tags is a dict[str,str].
• tags is reasonably free-form, but for now will include keys purpose and version. purpose can be either tool or user, and version should be castable to int.
• We retain our right to deprecate specific (purpose, version) pairs
• Lockfiles without a tags field will be assumed to be version 0 purpose tool lockfiles, and will be deprecated at our earliest convinience per Lockfiles: confirm deprecation plan for lockfile header #12683

[Eric-Arellano]

What's the motivation for embedding tool vs user, along with version?

I suspect an even simpler modeling is something like this:

{
    "requirements": ["foo==1.2", "bar>=1"],
    "requirementes_hash": null,
    "interpreter_constraints": ["==3.6.*"],
    "platforms": null,
}

I know you also proposed some nesting:

{
    "requirements": ["foo==1.2", "bar>=1"],
    "requirementes_hash": null,
    "env": {
        "interpreter_constraints": ["==3.6.*"],
        "platforms": null,
     },
}

[chrisjrn]

The motivation for embedding tool vs user was to give us an expectation of what fields need to be present to validate the lockfile, which would be important if we add a verification field (to check that the rest of the data in the header is unmodified)

[Eric-Arellano]

was to give us an expectation of what fields need to be present to validate the lockfile

That expectation comes from our code itself. If it's a tool lockfile, we need requirementes_hash else requirements. Our code knows what the lockfile refers to, no need afaict for that to be embedded in the header.

[Eric-Arellano]

if we add a verification field

What do you mean? Like a schema number? Ad discussed in #12683, I think we possibly would benefit from that, but probably not necessary because of the deprecation plan to not support older schema than the prior release's schema. We can likely keep it simple and leave that off, rather than needing to add and test this new mechanism.

[chrisjrn]

I meant hashing the data in the header

[Eric-Arellano]

(to check that the rest of the data in the header is unmodified)

Hm, I'm not sure that is important to address right now. We already have a big warning to not change the metadata. If you do, that's at your own peril. And it's easy to fix by regenerating the lockfile.

I don't think we need an extra mechanism to validate they didn't tamper with the header.

[chrisjrn]

Seems reasonable

[Eric-Arellano]

Did you mean to close this @chrisjrn? This hasn't been merged to main.

[chrisjrn]

No, I must have accidentally clicked a button