Pex repository not created if [python].enable_resolves=true and the resolve does not use a PEX lockfile

[stuhood]

A pex repository is not created when using the combination of settings from the description, which results in skipping the --pex-repository subsetting operation.

[stuhood]

I've spent a few hours investigating this, but the code is fairly hairy. Will try to get it resolved by EOD Tuesday.

[thejcannon]

To be more specific here, this is only an issue if the user is using a non-PEX-lockfile right? I don't see a foo_resolver option in https://www.pantsbuild.org/v2.11/docs/reference-python

[thejcannon]

So I'm not sure this is an issue. There's only 2 reasons we need the repository.pex:

1. [python].resolve_all_constraints is True
   • "Requires [python].requirement_constraints to be set." which is "Mutually exclusive with [python].enable_resolves."
   • So this doesn't apply
2. [python]run_against_entire_lockfile is True
   • I don't see this in the description, and it default to False

I don't see a reason why enable_resolves needs to be (transitively) mutually exclusive with resolve_all_constraints though. That would "fix" this issue.

@Eric-Arellano thoughts?

[stuhood]

To be more specific here, this is only an issue if the user is using a non-PEX-lockfile right? I don't see a foo_resolver option in https://www.pantsbuild.org/v2.11/docs/reference-python

Correct. Sorry: I meant lockfile_generator.

So I'm not sure this is an issue. There's only 2 reasons we need the repository.pex:

@thejcannon : The --pex-repository option is an important optimization for any monolithic resolve which cannot subset a resolve the way that PEX lockfiles can. If you don't use it with enable_resolves and lockfile_generator=poetry, then the effect is that every single target re-runs the resolve from scratch, which will hit the network to check PIP for updates and etc (although it should hit the PIP cache for actual wheels). PEX's native subsetting is much, much more efficient than that.

The reason that resolve_all_constraints is mutually exclusive with multiple resolves is that in multiple resolves, resolve_all_constraints is the right strategy (...for a poetry/manual lockfile), not that it's the wrong one.

But most importantly: it causes repositories which are not using PEX lockfiles to fail to build (ours in particular), possibly because the PEX-repository resolution codepath is more lenient (?). So we'll need to fix it.

[thejcannon]

An important distinction here is that Pants can and will consume a PEX lockfile regardless of lockfile_generator's setting.

[Eric-Arellano]

Reminder that [python].lockfile_generator is not sufficient to know if it's a pex lockfile or not. a lot of work into that so you can do things like have some locks be pex and some manually generated. we need to call our helper function to inspect the lock and see if its json

[stuhood]

An important distinction here is that Pants can and will consume a PEX lockfile regardless of lockfile_generator's setting.

Yes, and that's part of why the code is so complicated, unfortunately. We're going to need to support both for a while, so I'm trying to push the detection further upfront without too much churn.

[thejcannon]

The reason that resolve_all_constraints is mutually exclusive with multiple resolves is that in multiple resolves, resolve_all_constraints is the right strategy (...for a poetry/manual lockfile), not that it's the wrong one.

Ah so really one way to think of this is it being implictly true in this case. That's what I missed in #14944

[Eric-Arellano]

i suspect pex_from_targets will need to inspect the lock and call our helper function to determine if repo.pex should be used. pex.py can hopefully stay unchanged from josh's pr