Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[internal] Refactor consumption of JVM tool lockfiles #14239

Merged
merged 5 commits into from
Jan 24, 2022

Conversation

Eric-Arellano
Copy link
Contributor

Three improvements:

  1. Rename MaterializedClasspath to ToolClasspath. The type is only being used to set up tools—whether those be external like Scalafmt or internal like our JVM parser. This rename makes that clear, which will allow us to start adding tool-specific functionality.
  2. Embrace that ToolClasspathRequest is 1-per-tool. This is how we've been using it de facto, including in two rules where there are 2 tools used in the same process. This allows us to simplify callsites / remove verbosity.
  3. Use the engine to read tool lockfiles, not open(). This is necessary to not break caching.

Tool lockfiles still have issues - a followup will fix these:

  • scalac_plugins.py is not using lockfile validation
  • callsites have more duplication than ideal

The type is only being used to set up tools—whether those be external like Scalafmt or internal like our JVM parser. This rename makes that clear, which will allow us to start adding tool-specific functionality.

# Rust tests and lints will be skipped. Delete if not intended.
[ci skip-rust]

# Building wheels and fs_util will be skipped. Delete if not intended.
[ci skip-build-wheels]
This is how we've been using it de facto, including in two rules where there are 2 tools used in the same process. This allows us to simplify callsites / remove verbosity

# Rust tests and lints will be skipped. Delete if not intended.
[ci skip-rust]

# Building wheels and fs_util will be skipped. Delete if not intended.
[ci skip-build-wheels]
This is necessary to not break caching.

# Rust tests and lints will be skipped. Delete if not intended.
[ci skip-rust]

# Building wheels and fs_util will be skipped. Delete if not intended.
[ci skip-build-wheels]
@chrisjrn
Copy link
Contributor

👀

# Rust tests and lints will be skipped. Delete if not intended.
[ci skip-rust]

# Building wheels and fs_util will be skipped. Delete if not intended.
[ci skip-build-wheels]
# Rust tests and lints will be skipped. Delete if not intended.
[ci skip-rust]

# Building wheels and fs_util will be skipped. Delete if not intended.
[ci skip-build-wheels]
Copy link
Contributor

@chrisjrn chrisjrn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Catching out the fact that we never have multiple lockfiles in a classpath request was good. This definitely simplifies the consuming code.

@@ -680,16 +681,13 @@ class MaterializedClasspathRequest:
"""

prefix: str | None = None
lockfiles: tuple[CoursierResolvedLockfile, ...] = ()
artifact_requirements: tuple[ArtifactRequirements, ...] = ()
lockfile: CoursierResolvedLockfile = CoursierResolvedLockfile(entries=())
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is SO much clearer. Nice.

@Eric-Arellano Eric-Arellano enabled auto-merge (squash) January 24, 2022 19:43
@Eric-Arellano Eric-Arellano merged commit 78f3a27 into pantsbuild:main Jan 24, 2022
@Eric-Arellano Eric-Arellano deleted the jvm-tool-consumption branch January 24, 2022 20:33
Copy link
Member

@stuhood stuhood left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lovely!

@Eric-Arellano
Copy link
Contributor Author

@stuhood @chrisjrn another simplification we can make: in practice, a ToolclassPathRequest only ever has a single artifact_requirements or lockfile, not both. There's no need to merge things. Does it seem safe to lean into this fact? It will make the code easier to follow in this next followup.

@stuhood
Copy link
Member

stuhood commented Jan 24, 2022

@stuhood @chrisjrn another simplification we can make: in practice, a ToolclassPathRequest only ever has a single artifact_requirements or lockfile, not both. There's no need to merge things. Does it seem safe to lean into this fact? It will make the code easier to follow in this next followup.

While there have been a lot of good cleanups, it would be good to connect further cleanup to the next feature we're trying to implement here. AFAICT, #13374 is finished, and we don't have any more lockfile stuff scheduled on the JVM side (https://github.com/pantsbuild/pants/projects/22?fullscreen=true).

@Eric-Arellano
Copy link
Contributor Author

Things are still broken:

  • scalac_plugins.py is not using lockfile validation

This is all in service of fixing that. Along with making things better than they were before - Chris and I collectively spent two weeks on this lockfile code because we were confused, so we're trying to clean things up so the next person in the future can approach it more easily.

@wisechengyi wisechengyi mentioned this pull request Jan 29, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants