Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Shortcircuit local distribution source subsetting if there are no dists. #14564

Merged

Conversation

stuhood
Copy link
Member

@stuhood stuhood commented Feb 22, 2022

#14551 improved the performance of local dist building when local distributions are actually present. But there are cases (which @benjyw is pursuing) where the @rule takes a long time to run, even when no dists are actually present. This is likely to do with the source subtraction: either the calculation of the subset paths, or the execution of DigestSubset.

[ci skip-rust]
[ci skip-build-wheels]

# Rust tests and lints will be skipped. Delete if not intended.
[ci skip-rust]

# Building wheels and fs_util will be skipped. Delete if not intended.
[ci skip-build-wheels]
@@ -178,6 +178,12 @@ async def build_local_dists(
),
)

if not wheels:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we also shortcircuit at line 146? Other than an extra 2 lines of code, I see no reason not to.

Also I'm not sure this informative comment is necessary - short circuiting usually seems like the right thing to do!

Copy link
Member Author

@stuhood stuhood Feb 23, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we also shortcircuit at line 146? Other than an extra 2 lines of code, I see no reason not to.

That would require adjusting the signature of LocalDistsPex to make the pex optional, which affects a lot of callsites. I started down that path, but shortcircuiting here instead allowed for the null-object pattern.

Also I'm not sure this informative comment is necessary - short circuiting usually seems like the right thing to do!

Yeaaaa. @benjyw and I both felt that landing this was punting on determining what the actual performance issue was in the trailing code: he's following up on our best guess there. If he determines why then he can delete the comment perhaps, but in the meantime let's leave it.

@stuhood stuhood enabled auto-merge (squash) February 23, 2022 00:17
@stuhood stuhood merged commit 40bd5b3 into pantsbuild:main Feb 23, 2022
@stuhood stuhood deleted the stuhood/shortcircuit-local-dist-building branch February 23, 2022 00:45
stuhood added a commit to stuhood/pants that referenced this pull request Feb 23, 2022
…uild#14564)

pantsbuild#14551 improved the performance of local dist building when local distributions are actually present. But there are cases (which @benjyw is pursuing) where the `@rule` takes a long time to run, even when no dists are actually present. This is likely to do with the source subtraction: either the calculation of the subset paths, or the execution of `DigestSubset`.

[ci skip-rust]
[ci skip-build-wheels]
stuhood added a commit that referenced this pull request Feb 23, 2022
…pick of #14564) (#14566)

#14551 improved the performance of local dist building when local distributions are actually present. But there are cases (which @benjyw is pursuing) where the `@rule` takes a long time to run, even when no dists are actually present. This is likely to do with the source subtraction: either the calculation of the subset paths, or the execution of `DigestSubset`.

[ci skip-rust]
[ci skip-build-wheels]
alonsodomin pushed a commit to alonsodomin/pants that referenced this pull request Feb 25, 2022
…uild#14564)

pantsbuild#14551 improved the performance of local dist building when local distributions are actually present. But there are cases (which @benjyw is pursuing) where the `@rule` takes a long time to run, even when no dists are actually present. This is likely to do with the source subtraction: either the calculation of the subset paths, or the execution of `DigestSubset`.

[ci skip-rust]
[ci skip-build-wheels]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants