Add an API to coarsen/partition Targets by their cycles #12251

stuhood · 2021-06-27T17:55:55Z

In order to recursively (such as for compilation) consume a graph of Targets containing cycles, we must "coarsen" the Targets in each cycle into a batch (called a strongly connected component).

This change adds a rule to compute the CoarsenedTargets for Addresses. To compile a graph of Targets with fine grained compiler invokes, you can "walk" the coarsened graph by requesting CoarsenedTargets for the roots, and then recursively requesting a compiler output per CoarsenedTarget dependency.

patricklaw · 2021-07-04T00:35:48Z

Hey Stu, I managed to get javac working against this branch and the coarsening/cycle-detection seems to work correctly. See #12275 for details.

I'm happy to consume this as-is, although you should check out the linked PR and in particular the coarsened_components rule in javac.py. Basically, my most immediate need was not for a single-target coarsening, but rather a set of unique coarse components from an input set of targets--and I feel like this likely belongs in the generic library code rather than special-cased in javac, though it's fine to leave it there for now if you want to wait until we find another motivating use-case.

[ci skip-build-wheels]

…dTargets. # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

# Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

…nd rename the coarsened unit to CoarsenedTarget. # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

stuhood · 2021-07-09T02:40:57Z

@patricklaw : I've applied the feedback from #12275 (which changed the API: sorry). See the top commit. CoarsenedTargets is now a collection of CoarsenedTarget instances, which each have 1) a set of members of the cycle, 2) the deduped dependencies of the cycle members.

Eric-Arellano

Cool stuff!

Eric-Arellano · 2021-07-09T16:17:25Z

src/rust/engine/src/externs/interface.rs

+fn strongly_connected_components(
+  py: Python,
+  adjacency_lists: Vec<(PyObject, Vec<PyObject>)>,
+) -> CPyResult<Vec<Vec<PyObject>>> {


It'd be great to do this w/ PyO3, but I see it depends on key_for which hasn't been ported yet. I can port this in a followup - it's more important we land this to unblock Patrick.

Yea. You can see in an earlier commit where I was attempting to do that too: the issue was that without hash/eq, I'd need to use PyDict rather than the rust native hashmap (not much of a hardship...maybe even more performant, but awkward).

What do you mean without hash/eq?

PyObject itself does not have Eq/Hash implementations, so you need wrapper types around it to add those (we use Value and Key to do that).

Eric-Arellano · 2021-07-09T16:19:35Z

src/rust/engine/src/externs/interface.rs

+  }
+
+  Ok(
+    petgraph::algo::tarjan_scc(&graph)


I love great libraries like this :)

Maybe add a brief comment motivating why we use Tarjan instead of Kosaraju, given that both are offered:

https://www.quora.com/Is-there-any-difference-or-advantages-or-disadvantages-between-Kosarajus-algorithm-and-Tarjans-algorithm-for-finding-strongly-connected-component-Which-one-is-most-useful?share=1

https://www.geeksforgeeks.org/comparision-between-tarjans-and-kosarajus-algorithm/

src/python/pants/engine/target.py

src/python/pants/engine/internals/graph_test.py

src/python/pants/engine/internals/graph.py

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

stuhood · 2021-07-12T23:56:05Z

I believe I've applied the feedback here: ready for another look.

patricklaw · 2021-07-13T00:21:21Z

This looks good to me at an API level. I've merged it into my local branch and I'm working on getting javac working against the new API, though I don't expect to finish tonight. Feel free to merge as-is and I'll keep working against it on main, otherwise I should be able to prove it out in javac within a day or two.

### Internal * [internal] Manually fix Black lockfile to handle interpreter constraints ([#12366](#12366)) * Revert "Prefix the entire setup.py chroot. (#12359)" ([#12370](#12370)) * [internal] Cache native client binary in CI ([#12355](#12355)) * Prefix the entire setup.py chroot. ([#12359](#12359)) * [internal] Fix AWS CLI breaking due to Python 2 usage ([#12364](#12364)) * [Internal] Add `git_url()` helper to `docutil.py` ([#12352](#12352)) * [Internal] Refactor how `PythonToolBase` exposes requirements and interpreter constraints ([#12356](#12356)) * [internal] Add experimental per-tool lockfiles with Docformatter as an example ([#12346](#12346)) * Remove Pants's dpeendency on `requests` ([#12348](#12348)) * [internal] Remove `TwoStepPex` abstraction ([#12343](#12343)) * Add psycopg2-binary to default module mapping ([#12339](#12339)) * [internal] Upgrade toolchain pants plugin to 0.13.1 ([#12338](#12338)) * Add an API to coarsen/partition Targets by their cycles ([#12251](#12251)) * Prepare 2.5.1 ([#12329](#12329)) * Bootstrap fewer JVM versions in Coursier/javac tests to hopefully reduce CI flakiness ([#12325](#12325)) * Native client respects `--concurrent`. ([#12324](#12324)) * Add client lib tests. ([#12322](#12322)) * Special case enum option parse failures. ([#12281](#12281))

…tion, including cycles created by subtargets. (#12231) Consume CoarsenedTargets (added in #12251) in javac for cycle resolution, including cycles created by subtargets.

stuhood mentioned this pull request Jul 6, 2021

WIP: Consume CoarsenedTargets in javac #12275

Closed

stuhood added 5 commits July 8, 2021 17:55

WIP: Add an initial implementation of the CoarsenedTargets.

15b2869

[ci skip-build-wheels]

Move scc implementation to rust-cpython.

207104b

[ci skip-build-wheels]

Add a test, and switch to using UnexpandedTargets to compute Coarsene…

653911a

…dTargets. # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

Squash with second commit.

a24f4dc

# Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

Move to batch computation of CoarsenedTargets, expose dependencies, a…

9a927bd

…nd rename the coarsened unit to CoarsenedTarget. # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

stuhood force-pushed the stuhood/coarsened-targets branch from 1e8e0c3 to 9a927bd Compare July 9, 2021 02:32

stuhood requested review from Eric-Arellano, benjyw and tdyas July 9, 2021 02:38

stuhood marked this pull request as ready for review July 9, 2021 02:38

stuhood requested review from patricklaw and removed request for benjyw July 9, 2021 02:41

Eric-Arellano reviewed Jul 9, 2021

View reviewed changes

Expand tests, fix dependencies, use/document include_special_cased_deps.

f33419d

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

Eric-Arellano approved these changes Jul 13, 2021

View reviewed changes

stuhood merged commit b0b7b0b into pantsbuild:main Jul 13, 2021

stuhood deleted the stuhood/coarsened-targets branch July 13, 2021 15:57

wisechengyi mentioned this pull request Jul 17, 2021

Prep 2.7.0.dev1 #12373

Merged

patricklaw mentioned this pull request Jul 25, 2021

Consume CoarsenedTargets (added in #12251) in javac for cycle resolution, including cycles created by subtargets. #12231

Merged

stuhood mentioned this pull request Aug 16, 2021

Build and install python_distribution deps for test, run, and repl #12573

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add an API to coarsen/partition Targets by their cycles #12251

Add an API to coarsen/partition Targets by their cycles #12251

stuhood commented Jun 27, 2021 •

edited

Loading

patricklaw commented Jul 4, 2021

stuhood commented Jul 9, 2021

Eric-Arellano left a comment

Eric-Arellano Jul 9, 2021

stuhood Jul 9, 2021

Eric-Arellano Jul 9, 2021

stuhood Jul 9, 2021

Eric-Arellano Jul 9, 2021

Eric-Arellano Jul 9, 2021

stuhood commented Jul 12, 2021

patricklaw commented Jul 13, 2021

Add an API to coarsen/partition Targets by their cycles #12251

Add an API to coarsen/partition Targets by their cycles #12251

Conversation

stuhood commented Jun 27, 2021 • edited Loading

patricklaw commented Jul 4, 2021

stuhood commented Jul 9, 2021

Eric-Arellano left a comment

Choose a reason for hiding this comment

Eric-Arellano Jul 9, 2021

Choose a reason for hiding this comment

stuhood Jul 9, 2021

Choose a reason for hiding this comment

Eric-Arellano Jul 9, 2021

Choose a reason for hiding this comment

stuhood Jul 9, 2021

Choose a reason for hiding this comment

Eric-Arellano Jul 9, 2021

Choose a reason for hiding this comment

Eric-Arellano Jul 9, 2021

Choose a reason for hiding this comment

stuhood commented Jul 12, 2021

patricklaw commented Jul 13, 2021

stuhood commented Jun 27, 2021 •

edited

Loading