Python distribution task for user-defined setup.py + integration with ./pants {run/binary/test}

[CMLivingston]

Problem

There is no way to consume a python_library with c_extensions. Relates to #4740.

Solution

Create a new target, python_distribution, that specifies only a distribution name that maps to the directory the BUILD file lives in (a distribution's project directory). Pants will treat the directory that the python_distribution target is defined in aa a distribution project directory (containing a setup.py and sources) and run the pex command on that directory to build an inline wheel using the setup.py. The ./pants binary and ./pants run tasks will dump this distribution into the binary pex for consumption (or requirements pex for the ./pants run case). The setup.py takes care of linking the package sources and c/cpp code together as setuptools Extensions. We've run a similar example by our customer to validate that this meets their needs.

Result

Running ./pants binary or ./pants run on a python_binary target that depends on a python distribution will package the distribution as a wheel and include it in the created pex binary or requirements pex (for ./pants run) for consumption. This example has simple C code to demonstrate the ability to call C extensions from a package built by using this python_distribution target.

[wisechengyi]

Quoting myself from #4740

How about naming it python_c_extension_library? python_distribution is a bit confusing to me at first as I thought it would provide the version of python I wanted

@CMLivingston @kwlzn thoughts?

Edit: correct issue #

[kwlzn]

How about naming it python_c_extension_library? python_distribution is a bit confusing to me at first as I thought it would provide the version of python I wanted

@wisechengyi it's not specific to c-extensions tho, so that'd be framing it up wrong. the term Distribution is well known in the python community (and is the 'dist' in sdist/bdist et al), but is definitely an overloaded term in general.

maybe python_dist or python_sdist would be slightly more idiomatic and a little clearer?

[CMLivingston]

@wisechengyi I am pretty torn on it; on one hand I see the confusion that could be had there in terms of the name, however I agree with Kris that the convention of using Distribution in the community aligns with this use case. Abbreviating to python_dist sounds like a happy medium, but since we are building wheels here, I think python_bdist would be better than python_sdist?

[wisechengyi]

Thanks @kwlzn @CMLivingston for explanation. I think python_dist sounds good, and it leaves us the flexibility in case we want to build both sdist and bdist down the road.

[kwlzn]

looks like a good start - handful of initial thoughts.

[kwlzn]

once this or anything else goes in marked public you won't be able to remove or change it without a two minor release deprecation cycle. so it would probably make sense to revisit the marking once we've demonstrated good use of it and nothing else needs to change.

[CMLivingston]

✅

[kwlzn]

as-is, this example isn't the most readable because of how many files/dirs its split around. it would probably make sense to collapse this example into a single dir that had a singular BUILD file containing a python_dist + python_binary and all of the sources in one location.

[CMLivingston]

✅

[kwlzn]

there's nothing specific to c/cpp extensions here - it's just a python dist, full stop.

[CMLivingston]

✅

[kwlzn]

defining sources in the setup.py won't be enough to support things like changed or other features that rely on target->sources mappings.

since the Target impl subclasses PythonTarget, if you weren't overriding sources=[] in the super call it would have an implicit sources= field that would be partially correct in this example (i.e. only globs *.py at the moment, not the c files or other files). it would probably also make sense to 1) permit/require a definition of sources 2) explicitly define sources= in this example since the point is to be illustrative about how to use it 3) consider what the most appropriate default sources globs should be for python_dist.

[CMLivingston]

Along the lines of #3, what if we just used rglobs(*.h|*.py|*.c|*.cpp) as an implicit default? This way the user would not need to define the sources they care about in two places (setup.py and then have to also list them out in a sources param). I'm not sure I understand the value in allowing the user to define sources beyond restricting change detection to a subset of rglobs(*.h|*.py|*.c|*.cpp).

[kwlzn]

1. rglobs would prevent python_dist from being able to co-exist with other targets in a dir or any child dir, which I think would be bad in general. I'm fairly sure (but not positive) that all of the existing implicit sources globs you'd find would be regular globs for this reason.
2. globbing all of the .h/.c/.cpp to me, for python_dist, feels unusual/unexpected as a user. I could see *.py - but I'd argue that C extension dists are typically an exception vs the norm in the general case, so we should probably require that users explicitly declare those when needed.

I'm open to other perspectives and counter arguments tho, but from an adoption strategy perspective in my experience it's way easier to add shortcuts like this later than it would be to take them away once delivered if we find they lead to bad/unwanted behavior - so IMHO, starting with the least surprising implicit behavior would be ideal.

[CMLivingston]

✅

[kwlzn]

worth noting that subclassing PythonTarget is going to net you a handful of fields that may or may not make sense in the dist context, like provides, resources and resource_targets. should consider the ramifications of what would happen if a user defines these expecting them to Just Work - and potentially add tests to cover those cases or disallow their definition.

[CMLivingston]

✅

[kwlzn]

I would expect to see this keying all writes to some path under e.g. .pants.d/pyprep/local_dists/<target fingerprint>/... - ideally with that key computed outside of this function and handed in as a target dir vs being handed the workdir directly. it would also probably be a good idea to include some form of no-op reuse of previously built wheels on re-keys to the same fingerprint for build performance.

in order to ensure that sources are properly declared, it should likely also copy the declared sources for the target into a temporary dir in that path before invoking the setup.py in it. this will ensure that any temporary files that are created as part of that land in the workdir and thus don't need manual cleanup such as what you're doing for *.egg-info below.

[CMLivingston]

✅

[kwlzn]

would be good, in general, to add type info to all docstrings. e.g.

:param list dist_targets: A list of `PythonDistribution` targets to build.

[CMLivingston]

✅

[CMLivingston]

We noticed that docstrings in most of the files we have touched don't have types. Should we add them to this file alone or should we address it in a separate pull request?

[kwlzn]

it doesn't need to be uniform, but in general always adding type info going forward would be preferred.

[kwlzn]

not successfully building a dist here should probably be an error.

[CMLivingston]

✅

[kwlzn]

if two different python_dist targets here key to the same wheel, that should probably be an error because otherwise one will be silently ignored.

[CMLivingston]

✅

[kwlzn]

what happens here if the py_dist target happens to conflict or overwrite a resolved requirement?

seems to me like we should be dumping those prior to the resolve and then considering them in the full transitive resolve closure to reveal any resolution impact.

this also raises the question of how to handle transitive requirements of python_dist setup.py.. ideally we could construct the wheel intransitively and then let the resolve handle pulling in the transitive requirements as part of this.

there's also a question of how to handle setup_requires in a given setup.py, but that's something we can probably just punt on for now and revisit.

[CMLivingston]

That's a good point. Would this approach require anything beyond moving this logic up higher (above dump_requirements) and letting dump_requirements de-dupe transitive requirements?

How would we ensure that 3rd party requirements resolved as part of a python distribution (specified by 3rdparty/python:dep) and requirements specified in setup.py are not duplicated? It seems like this would necessitate handling setup_requires in the here and now as well.

[UnrememberMe]

Overall, looks promising. I have gave verbal feedback before and I will add those as comments here:

• I would suggest to give python_distribution a sources field, even if always uses as sources=rglob('**/*'). This should trigger test of depedent modules' test
• In long term, I would like to see some way to enable breaking down the C/C++ codes into libraries that we can create a build graph from. This should facilitate better modulation.

[CMLivingston]

Installing this task on pyprep seems like the best place to put it, however with this choice it will only work in ./pants run if the produced .whl is dumped into the leftmost runner pex, which seems like an anti-pattern. Thoughts?

[CMLivingston]

This is ready for another look when convenient. Unit/integration tests will follow after we solidify the task approach.

[kwlzn]

looking good mod a handful of questions and comments.

would also need some basic test coverage here before this can be merged.

[kwlzn]

indent

[lgvital]

✅

[kwlzn]

indent

[lgvital]

✅

[kwlzn]

indent

[lgvital]

Hmm, so I guess the linter catches only certain indent errors in *.py files, but not in BUILD files? Pretty sure it flagged indent issues in *py.

[lgvital]

✅

[kwlzn]

indent

[kwlzn]

this should just reference :superhello, since its in the same BUILD file

[lgvital]

✅

[kwlzn]

is there no way to use a FingerprintStrategy under care of a Task.invalidated() block to do this instead?

[kwlzn]

this task produces wheels, not pex files tho - which I wouldn't expect to write to dist (until consumed by a pex).

afaict, its possible for two python_dist targets to share the same base target name - but have totally different dist names based on their setup.py files. so I'm not sure this validation buys much here, fwict? thoughts?

[UnrememberMe]

We don't really know the dist_name will be created from the target definition currently. It seems like a reasonable approximation?

[lgvital]

Yeah, I think you're right that this doesn't really protect against much. Just removed.

[kwlzn]

this check is reasonable, but is this a case you've actually seen before? I wouldn't expect this to be possible without seriously non-standard setup.py.

[lgvital]

I think this was a check we added before we implemented the fingerprint strategy, so it may not be necessary anymore. Seems safe/reasonable enough to keep, but do you think we should remove?

[kwlzn]

nit: :returns:

[lgvital]

✅

[kwlzn]

seems to me like this is sidestepping the actual resolve, but ultimately I'm curious if this covers the case of:

1. user has a target that depends on 3rdparty/python:pytorch which provides e.g. pytorch==1.0.0
2. user also depends on a python_dist that provides pytorch==2.0.0
3. when the pex is built, it blows up indicating two incompatible versions of pytorch?

if so, great. if not, this could spell out trouble at runtime - particularly because of transitive deps that users might not be aware of. either way this case should probably get covered with an integration test.

[lgvital]

Great question. Let us get back to you on that, definitely need an integ test regardless.

[UnrememberMe]

that's a really good observation.

[UnrememberMe]

Does this issue get addressed here?

[CMLivingston]

This is covered in the same way a resolve would blow up on normal python targets (i.e. throwing an error indicating two incompatible versions of pytorch). I have an integration test covering this.

[UnrememberMe]

The codes for copying sources file should be taken care of with a invalidated block I believe. (https://github.com/pantsbuild/pants/blob/master/src/python/pants/task/task.py#L309)

[UnrememberMe]

that's a really good observation.

[UnrememberMe]

Also should check for multiple setup.py

[CMLivingston]

✅

[UnrememberMe]

We don't really know the dist_name will be created from the target definition currently. It seems like a reasonable approximation?

[UnrememberMe]

Just a thought: How long does a single setup runner run take? Is it anyway to run multiple setup runner simultaneously?

[CMLivingston]

Not sure at the moment, and this will likely be a follow up concern after this PR has landed.

[UnrememberMe]

Is this because of a style issue? If not, I would rather not submit a change here.

[CMLivingston]

This is a style fix.

[UnrememberMe]

4 spaces here vs. 2 below in dependencies

[CMLivingston]

✅

[UnrememberMe]

no need for local_dists_workdir any more.

[CMLivingston]

✅

[UnrememberMe]

indentation here is valid but a bit odd. I think we generally prefer to break earlier.

lambda tgt: isinstance(tgt, (
  PythonDidstribution, ...))

or

lambda tgt: isinstance(
  tgt,
  (PythonDistribution...)
)

seems to be a bit better.

[CMLivingston]

✅

[UnrememberMe]

2017

[CMLivingston]

✅

[UnrememberMe]

suggestion: instead of skip the test, if there is a wrong version of python installed, we should test that an error is generated appropriately.

[CMLivingston]

This is a pants convention to skip tests if the system does not have the target interpreter. The test util checks for the interpreter by making a subprocess.check_output call, so worrying about the error is out of the scope of these tests changes, I think. We can discuss further offline.

[UnrememberMe]

Does this issue get addressed here?

[UnrememberMe]

One small suggestion, feel free to take or leave.

[UnrememberMe]

SUGGESTION

  count_of_setup_py = [os.path.basename(s) for s in sources].count('setup.py')
  if count_of_setup_py == 0: 
    ...

[CMLivingston]

✅

[kwlzn]

handful of comments - fwict it seems like there's still a decent functional gap here in that a python_dist's transitive deps won't be considered during resolution. this feels like a feature customers will need from day 1.

[kwlzn]

this second check seems like an arbitrary restriction to me - it's not clear why a file named setup.py in e.g. a nested child dir rglob'd into a python_dist would matter as long as we can identify the main setup.py entrypoint target?

conversely, if that invalid, nested setup.py was the only one in a target then this wouldn't complain either.

checking only for the existence of "at least one "top-level" setup.py relative to the target definition" would probably be more correct, since thats the shape of a typical python dist.

[lgvital]

Makes sense. ✅

[kwlzn]

nit: docstring param ordering

[lgvital]

✅

[kwlzn]

this is good to test for, but I'm not sure I'd expect to see many outbound dependency edges from a python_dist other than for transitive pins. in fact, it could make sense to disallow dependencies= for this new target type altogether as a simplification measure since they don't make much sense in the dist building context (and pins could still be composed with a python_dist under a bare target).

but the test case I mentioned earlier (and the one I would be most worried about) would be if the python_dist's setup.py declared itself as pytorch==1.2.3 and then a target depended on that same requirement + e.g. python_requirement_library(pytorch==1.2.4) transitively. by just dumping the local dist into the pex chroot (vs considering that as part of the resolve), I believe this could lead to a pex that would be broken in subtle, weird ways.

I also don't see how any transitive deps of the python_dist would be resolved and included currently - and both of these problems appear to be somewhat intertwined at the impl level.

[lgvital]

These sound like fair concerns. Let me ask a couple clarifying questions to make sure I understand them correctly. I'm not entirely clear so this might be easier to clarify over chat / hangouts. Lmk if that's viable for you.

1. I assume disallowing dependencies= would be mainly a #simplify measure, but it wouldn't cover the test case that you mention?
2. Re: test case are you referring to a python_dist with a setup.py that has:

setup(
  name='pytorch',
  version='1.2.3'
)

and a target that depends on this python_dist and another target that depends on 1.2.4 transitively like:

python_binary(
  name='main',
  source='main.py',
  dependencies=[
    ':main2',
    ':pytorch',
  ]
)

python_library(
  name='main2',
  source='main2.py',
  requirements=[
    'python_requirement_library(pytorch==1.2.4)',
  ]
)

3. Hope I'm wrong here, but it seems like resolving transitive deps is a pretty complex/nontrivial undertaking. I understand the need to #BeRigorous, but I'm wondering if we can balance it with "Iterative, team-based development"? What if "this feels like a feature customers will need from day 1" weren't true for our internal customer that requested this feature? Assuming we address all other concerns, is there a path we can take to unblock our customer now and tackle transitive deps in a separate pull request?

[kwlzn]

1. correct, two separate things there.

2. that would be exactly it for the conflict case. for transitive dep resolution, the idea would be a python_binary that depended on a python_dist with a setup.py that declared a install_requires=['some_transitive_dep'] - which would be transitively imported from the python_binary's main.py during a test/run. I believe currently this would fail on the import of some_transitive_dep due to no resolver treatment for setup.py declared deps.

3. hmm, I don't get the sense it'd be super complex. the first bit would be extracting the injectable requirement spec from the built wheel or setup.py, but you can probably get away with simple post-build wheel filename parsing for that (and then e.g. '=='.join([name, version]). the second bit would be injecting that inferred spec in with the rest of the transitive requirement specs at build time before they're all passed to the resolver (and ensuring the dir containing the wheel is injected into the find-links paths with precedence). afaict, this would solve for both cases described in 2. but a TODO (ideally w/ github issue link) and iteration is fine also.

[lgvital]

Cool, thank you for the clarifications and guidance on #3. I'll take a stab and if it proves too complex / time consuming I may just TODO it.

So, related to #1 - the use case for including deps for a python_dist is that our internal customer wants to import tensorflow in setup.py so that they can load all the tensorflow includes into include_dirs. Would there be a way around this, if we weren't going to include deps inpython_dist?

[kwlzn]

...our internal customer wants to import tensorflow in setup.py so that they can load all the tensorflow includes into include_dirs. Would there be a way around this, if we weren't going to include deps inpython_dist?

the dependencies declared on this target would never be made available to the executing setup.py for use in the current model. they would only be pulled in transitively for any target roots and resolved into deps for a final pex product (pex, chroot or otherwise).

setuptools/setup.py et al already make an affordance for this case with setup_requires. this ensures the declared setup_requires deps are present at setup.py execution time vs install time. making it so that the user could declare "i need tensorflow during setup.py execution time" would, afaict, involve plumbing full support for setup_requires in SetupPyRunner via either native setup.py declarations or BUILD file declarations. I think from the pants/pex side, this would involve composing a setup-time pex/chroot and executing the setup.py inside of that where it'd be able to import tensorflow from a hermetic environment.

we typically don't deal with setup_requires much in pants/pex land because of the emphasis on bdists (wherein setup.py is pre-executed for us) for repeatability - but it'd certainly be a nice feature to have for python_dist consumption, since it'd bring us closer to parity with pip.

[kwlzn]

afaict, this happens in Target.__init__() so can kill that here.

[lgvital]

✅

[kwlzn]

could use set logic here:

>>> set([3,2,6]).isdisjoint(set([3,5,9]))
False
>>> set([3,2,6]).isdisjoint(set([1,5,9]))
True

[lgvital]

✅

[kwlzn]

can kill the else here and treat the if check as guard clause.

[lgvital]

✅

[kwlzn]

curious.. what is special about PythonDistribution targets vs others covered by the target_set_id path keying that necessitated this extra check?

[CMLivingston]

You were right, we didn't need that and removed it.

[kwlzn]

based on how python_dist_targets is used below, it seems like you could easily get at this with an inner local_python_dists = self.context.targets(is_local_python_dist) vs the new signature + changing the calling pattern of all callers. this would #simplify and reduce duplication.

[CMLivingston]

✅

[kwlzn]

ditto.

[CMLivingston]

✅

[kwlzn]

probably better to use the skipif decorator for these?

@unittest.skipIf(has_python_version(...), "no python ...")
def test_thing(...):
  pass

..but I would also expect python2.7 to be a strict requirement of pants' CI and test environments, so it's probably not even worth doing this check in the first place.

[CMLivingston]

I wrote it like this to leave room for extension when we want to test with Python 3 (similar to interpreter selection tests).

In this snippet, wouldn't we be doing not has_python_version(...) instead?

[CMLivingston]

✅

[UnrememberMe]

Reference in the error message to the target address will be nice.

[CMLivingston]

TargetDefinitionException contains this information.

[UnrememberMe]

If there are two local dists belong to two separate targets (say A and B). Running pants binary A B should generate .pex only contain the right local dist. Does this code handle the multiple targets scenario correctly?

[CMLivingston]

✅

[lgvital]

Remove? I assume this was for debugging.

[CMLivingston]

Nope. That returns a string which needs to be printed for verification in stdout.

[lgvital]

Since its assumed that pants is running 2.7, is it necessary to specify that in these variables? Or are you future-proofing a bit?

[CMLivingston]

Def not necessary, I can clean them up.

[kwlzn]

where is the 'conflicting dep' in this example case?

your setup.py provides a package called superhello. your python_dist and python_binary both depend on the same pex_lib @ pex==1.2.11.

to have a conflicting case here to exercise the core resolver consideration that we've been discussing, your setup.py would need to either stomp on the pex name as built into a dist with a differing version - or depend on one via install_requires - in order to create a conflict (ideally, tests for both).

and again, your python_dist should also not declare a dependencies= here because it is /providing/ a dep (and dragging its transitive dependencies along with it if/when the resolver is wired up correctly).

[CMLivingston]

This case was for declaring a dep in the binary below (3rdparty/pex which conflicts iirc), but I think this is all probably unnecessary because we know targets can handle this correctly already.
So you are proposing we kill of dependencies field of python dist all together and have internal teams list their deps (like tensorflow) in install_requires instead? Because writing python code that depends on TF in your py_dist and linking to 3rdparty/python:tensorflow in the dependencies field of a consuming binary target seems confusing.

[kwlzn]

producing >1 wheels should probably be an error condition, or at least a warning - vs the silent masking happening here now.

[CMLivingston]

✅

[kwlzn]

this was probably better before, as the one liner.

[kwlzn]

xx

[CMLivingston]

✅

[lgvital]

My previous suggestions have been addressed in previous commits. Just one minor nit here.

[lgvital]

We have a similar helper method above (is_distribution) and we have another call to isinstance in setup_py.py. Would it be useful to have one helper method, maybe in a util file?

[UnrememberMe]

Is there something we needed to do before we can land this change? The failed tests seem to be come over TravisCI infrastructure.

[stuhood]

I've just restarted them.

But the answer to your question is really "get a shipit from @kwlzn", probably.

[UnrememberMe]

The latest CI runs only failure comes from golang lint check, Exception message: No <meta name="go-import"> tag found at google.golang.org/grpc/codes, which shouldn't have anything to do with this branch.
Please instruct the next step.

[kwlzn]

I've restarted the failing shard - looks good. Will review tomorrow.

[UnrememberMe]

Ping.

[kwlzn]

looking good - on the test cases in particular. handful of comments.

[kwlzn]

this should use PythonDistribution.alias() vs 'python_dist'

[CMLivingston]

✅

[kwlzn]

nit: whitespace

[CMLivingston]

✅

[kwlzn]

I would expect this check for dependencies + TargetDefinitionException raising to happen in the PythonDistribution.__init__() vs at task execution time?

[CMLivingston]

AFAICT, this is not possible in the init method. My attempts have been blocked by AssertionError: Cannot retrieve dependencies of BuildFileAddress(testprojects/src/python/python_distribution/fasthello_with_install_requires/BUILD, fasthello) because it is not in the BuildGraph.

[stuhood]

Yea, not feasible elsewhere: target constructors don't see their dependencies currently.

[kwlzn]

ah, k

[CMLivingston]

✅

[kwlzn]

nit: pants repo style dictates '{}'.format(blah) form (vs pex's %s).

[CMLivingston]

✅

[kwlzn]

the use of the term 'build' here is misleading.. how about inject_synthetic_req_libs?

[kwlzn]

also, how about taking just a build_graph vs the entire context?

[CMLivingston]

✅

[kwlzn]

I'm not seeing how dist_tgts is used here?

..but maybe has_python_requirements should be expanded to also match PythonDistribution targets if they now provide a resolvable requirement on the surface?

[CMLivingston]

This would not be viable in the current implementation fwict, as PythonDistribution targets would get treated as requirements libraries elsewhere in the codebase and break things. I see the need for the distinction because tasks that operate on req libs need PythonRequirementsLibrary targets, and while these are provided by py dist targets, it appears to me that the current backend does not accept PythonDistribution targets as a has_python_requirements target as of right now.

[kwlzn]

I'd expect this to conform more to the existing calling convention - so ideally the caller passes in local_dist_targets as either a combined or separate param here? that would align better with how things are used below too, afaict?

[CMLivingston]

✅

[UnrememberMe]

tgts = req_libs + (local_dist_targets or []) works as well.

[kwlzn]

the class name feels like an odd Address to use here? it won't be unique, for one - so this likely needs to be generative into some synthetic namespace.

[CMLivingston]

✅

[kwlzn]

how are you ensuring alignment between the dump_requirements provided req->dists vs the dists explicitly added here?

i.e. how does dump_requirements know where to resolve custom dists from in the case where they're injected into req_tgts above?

[CMLivingston]

✅

[kwlzn]

might be good to add a test case for the case described here - did not see one below.

[CMLivingston]

✅

[kwlzn]

+ @benjyw and @jsirois for a second pair of eyes on the Task impl et al.

[kwlzn]

LGTM - one last important comment about synthetic address namespacing and a couple of readability/doc string things - but otherwise I think we're good to land this and iterate.

[kwlzn]

afaict, it doesn't actually include transitive dependencies here - just the surface requirement that aligns to the built wheel.

[kwlzn]

the logic here seems fine but I think this function would be more readable/maintainable if you decomposed it a bit - something along the lines of (note the renamed function name + docstring, too):

def inject_synthetic_dist_requirements(build_graph, local_built_dists, synthetic_address, binary_tgt=None):
  """Injects synthetic `PythonRequirement` objects from a `PythonDistribution` into the `BuildGraph`.
  ...
  """
  def should_create_req(bin_tgt, loc):
    if not bin_tgt:
      return True
    # <-- Add comment here about why this is necessary
    return any([tgt.id in loc for tgt in bin_tgt.closure()])

  def python_requirement_from_wheel(path):
    base = os.path.basename(path)
    whl_dir = os.path.dirname(path)
    whl_metadata = base.split('-')
    req_name = '=='.join([whl_metadata[0], whl_metadata[1]])
    return PythonRequirement(req_name, repository=whl_dir)

  local_whl_reqs = [
    python_requirement_from_wheel(whl_location)
    for whl_location in local_built_dists
    if should_create_req(binary_tgt, whl_location)
  ]

  if not local_whl_reqs:
    return []

  addr = Address.parse(synthetic_address)
  build_graph.inject_synthetic_target(addr, PythonRequirementLibrary, requirements=local_whl_reqs)
  return [build_graph.get_target(addr)]

[kwlzn]

generally speaking, synthetic addresses would map to the path hierarchy of a given target w/ some synthetic name (e.g. //path/to/the/dir:my_injected_dist_0).. this maps to e.g. //<some_sha_here> in the root.

at a minimum I think you would want to key this into the target's path namespace, e.g. ~:

':'.join((binary_tgt.address.rel_path, binary_tgt.invalidation_hash()))

..it'd also be good to break out the last two params here each onto its own line for readability while you're there.

[kwlzn]

and at that point, its probably just as safe/sane to use a static name key like:

':'.join((binary_tgt.address.rel_path, '{}_dist_reqs'.format(binary_tgt.address.target_name))

[benjyw]

Just noticed something: What happens if only these local dists change? The invalidated() block won't notice that, so the requirements pex won't be rebuilt. It seems like their originating targets should participate in the invalidation?

[benjyw]

Oh, I guess this is why the local_dist_targets arg exists? But they aren't passed in practice, as far as I can tell. And the duplication is a bit weird. Any reason not to do the inject_synthetic_dist_requirements() when the distribution is created, i.e., in build_local_python_distributions.py? That seems more idiomatic. Then downstream tasks like this one don't have to distinguish between the different provenances of requirement libraries, and can treat them uniformly.