Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MyPy and Pylint partition inputs via CoarsenedTarget (Cherry-pick of #15141) #15244

Merged
merged 1 commit into from
Apr 25, 2022

Conversation

stuhood
Copy link
Member

@stuhood stuhood commented Apr 25, 2022

CoarsenedTargets are structure shared, and because they preserve their internal structure, they can service requests for transitive targets for different roots from the same datastructure. Concretely: Mypy and Pylint can consume CoarsenedTargets to execute a single @rule-level graph walk, and then compute per-root closures from the resulting CoarsenedTarget instances.

This does not address #11270 in a general way (and it punts on #15241, which means that we still need per-root transitive walks), but it might provide a prototypical way to solve that problem on a case-by-case basis.

Performance wise, this moves cold check :: for ~1k files from:

  • main: 32s total, and 26s spent in partitioning
  • branch: 19s total, and 13s spent in partitioning

The rest of the time is wrapped up in #15241.

…d#15141)

`CoarsenedTarget`s are structure shared, and because they preserve their internal structure, they can service requests for transitive targets for different roots from the same datastructure. Concretely: Mypy and Pylint can consume `CoarsenedTargets` to execute a single `@rule`-level graph walk, and then compute per-root closures from the resulting `CoarsenedTarget` instances.

This does not address pantsbuild#11270 in a general way (and it punts on pantsbuild#15241, which means that we still need per-root transitive walks), but it might provide a prototypical way to solve that problem on a case-by-case basis.

Performance wise, this moves cold `check ::` for ~1k files from:
* `main`: 32s total, and 26s spent in partitioning
*  `branch`: 19s total, and 13s spent in partitioning

The rest of the time is wrapped up in pantsbuild#15241.
# Building wheels and fs_util will be skipped. Delete if not intended.
[ci skip-build-wheels]
@stuhood stuhood merged commit aee95c6 into pantsbuild:2.11.x Apr 25, 2022
@stuhood stuhood deleted the cherry-pick-15141-to-2.11.x branch April 25, 2022 20:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants