Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[internal] Rename classes for generate_lockfiles.py for clarity #14218

Merged
merged 5 commits into from
Jan 20, 2022

Conversation

Eric-Arellano
Copy link
Contributor

There are two ways we "consume" lockfiles:

  • To generate it and save on disk, which requires doing a resolve.
  • To read an already-generated lockfile on disk.

@chrisjrn and I found that the current types are confusing to distinguish between these two uses. This is the first of multiple refactors to make more clear the types by renaming LockfileRequest to GenerateLockfile and Lockfile to GenerateLockfileResult.

# Rust tests and lints will be skipped. Delete if not intended.
[ci skip-rust]

# Building wheels and fs_util will be skipped. Delete if not intended.
[ci skip-build-wheels]
# Rust tests and lints will be skipped. Delete if not intended.
[ci skip-rust]

# Building wheels and fs_util will be skipped. Delete if not intended.
[ci skip-build-wheels]
Copy link
Contributor

@chrisjrn chrisjrn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Most of these changes are just renames, and those seem fine.

Please address the commented changes in jvm/../lockfile_metadata.py

@@ -57,15 +54,6 @@ def from_lockfile(
),
)

def _header_dict(self) -> dict[Any, Any]:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please don't delete this, it's defensive to ensure that future implementers don't override this incorrectly (primarily by making sure the instructions in the docstring are preserved in every file where LockfileMetadata is extended).

This is definitely a sign that the API needs refactoring (and I'll take that on), but in the meantime, this method should remain to prevent future foot-shooting while the current structure is in place.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wait actually I don't think I get the motivation. You'll have the docstring from core/util_rules/lockfile_metadata.py already. Plus the concrete examples from both JVM and Python for what to do.

This seems like a violation of DRY, and I don't understand what keeping these lines is worth the noise

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I preface this by saying that this code structure (which I wrote) is really bad, because it's impossible to validate that it's been used correctly, and I absolutely need to re-write it. Reasoning about it like it's good code is a bad idea.

The suggestion I'm making is absolutely a violation of DRY. The problem is that forgetting to call super() in this case creates an error because of how the versioning is implemented in the root class, and it's an error that I found difficult to test for back when I wrote the Python version.

Basically the suggestion, it's a defensive mechanism, to make sure the warning is front and centre to everyone who's editing the file, because it's easy to get it wrong. I think it's necessary because I wrote it and I found it difficult to work with.

My aim here is to give the next person to touch the code (presuming I don't fix the structure) every chance to succeed when they touch it next. It leaves a bad taste in my mouth, but that's how it is for the moment.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(I've added the refactor as a TODO for myself at #14219; I have an idea of what a better pattern would look like)

sorted(self.requirements) if self.requirements is not None else None
)
return out
return {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We had a discussion about this pattern back when implementing Python lockfiles, that if we were only changing one or two properties, it was better to amend a dictionary rather than construct a new one. This pattern is repeated over there.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't necessarily have a preference, but the inconsistency feels weird at this point :)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Our style guide says to prefer unpacking and expressions over mutation: https://www.pantsbuild.org/docs/style-guide#prefer-merging-collections-through-unpacking. We use this pattern abundantly throughout the codebase

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, I can't find the previous discussion where we decided on the pattern I copied over from the Python implementation, so happy to go with this version.

Is it worth changing the version python/lockfile_metadata to match?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, I can update it in followup changes, such as pending the other thread in this PR or the followup PRs I'm working on. I'd rather not burn a tree just for that change.

# Rust tests and lints will be skipped. Delete if not intended.
[ci skip-rust]

# Building wheels and fs_util will be skipped. Delete if not intended.
[ci skip-build-wheels]
@Eric-Arellano Eric-Arellano enabled auto-merge (squash) January 20, 2022 18:34
…le-request

[ci skip-rust]

[ci skip-build-wheels]
# Rust tests and lints will be skipped. Delete if not intended.
[ci skip-rust]

# Building wheels and fs_util will be skipped. Delete if not intended.
[ci skip-build-wheels]
@Eric-Arellano Eric-Arellano merged commit b13a150 into pantsbuild:main Jan 20, 2022
@Eric-Arellano Eric-Arellano deleted the rename-lockfile-request branch January 20, 2022 19:34
illicitonion added a commit to illicitonion/pants that referenced this pull request Jan 22, 2022
Internal changes:

* upgrade to Rust v1.58.0 ([pantsbuild#14174](pantsbuild#14174))

* [internal] fix typos in codegen registration ([pantsbuild#14172](pantsbuild#14172))

* Remove per-language indirection for formatter plugins. ([pantsbuild#14166](pantsbuild#14166))

* Pulls `Coordinate` and `ArtifactRequirements` out into a separate file to avoid a circuilar import later ([pantsbuild#14164](pantsbuild#14164))

* Factors lockfile metadata code into class that supports multiple languages as well as versions ([pantsbuild#14116](pantsbuild#14116))

* Adds `.env` file to make vscode auto-imports work correctly out of the box ([pantsbuild#14130](pantsbuild#14130))

* [internal] Rename classes for `generate_lockfiles.py` for clarity ([pantsbuild#14218](pantsbuild#14218))

* [internal] Fix bad Find+Replace ([pantsbuild#14213](pantsbuild#14213))

* [internal] Bump libc from 0.2.106 to 0.2.112 in /src/rust/engine ([pantsbuild#14206](pantsbuild#14206))

* [internal] Bump tempfile from 3.2.0 to 3.3.0 in /src/rust/engine ([pantsbuild#14202](pantsbuild#14202))

* [internal] Bump criterion from 0.3.3 to 0.3.5 in /src/rust/engine ([pantsbuild#14205](pantsbuild#14205))

* [internal] Bump walkdir from 2.3.1 to 2.3.2 in /src/rust/engine ([pantsbuild#14204](pantsbuild#14204))

* [internal] Bump generic-array from 0.14.4 to 0.14.5 in /src/rust/engine ([pantsbuild#14203](pantsbuild#14203))

* [internal] Remove the minimum bucket size of batching to improve stability. ([pantsbuild#14210](pantsbuild#14210))

* [internal] Make `JvmLockfileRequest` generic ([pantsbuild#14201](pantsbuild#14201))

* [internal] add comment re clippy issue ([pantsbuild#14188](pantsbuild#14188))

* [internal] rename some codegen scopes to put language first ([pantsbuild#14187](pantsbuild#14187))

* [internal] Check for ambiguity when running `generate-lockfiles` ([pantsbuild#14178](pantsbuild#14178))

* [internal] Change JVM lockfile format ([pantsbuild#14175](pantsbuild#14175))

* [internal] go: rewrite third party package analysis to not use `go list` ([pantsbuild#14157](pantsbuild#14157))

* [internal] Hotfix `fmt` crashing on non-formattable targets ([pantsbuild#14168](pantsbuild#14168))

* [internal] Simplify `core/goals/package.py` filesystem API usage ([pantsbuild#14162](pantsbuild#14162))

* [internal] java/thrift: register union that was not registered ([pantsbuild#14159](pantsbuild#14159))

* [internal] Bump arc-swap from 1.3.0 to 1.5.0 in /src/rust/engine ([pantsbuild#14148](pantsbuild#14148))

* [internals] Bump tokio-rustls from 0.22.0 to 0.23.2 in /src/rust/engine ([pantsbuild#14149](pantsbuild#14149))

* [internal] Bump num_cpus from 1.13.0 to 1.13.1 in /src/rust/engine ([pantsbuild#14150](pantsbuild#14150))

* [internal] Bump tokio-util from 0.6.7 to 0.6.9 in /src/rust/engine ([pantsbuild#14151](pantsbuild#14151))

* [internal] Bump os_pipe from 0.9.2 to 1.0.0 in /src/rust/engine ([pantsbuild#14152](pantsbuild#14152))

* [internal] Introduce new `BuiltinGoal` subsystem type. ([pantsbuild#13991](pantsbuild#13991))

* [internal] Upgrade python type stubs packages ([pantsbuild#14143](pantsbuild#14143))

* [internal] Bump strum_macros from 0.20.1 to 0.23.1 in /src/rust/engine ([pantsbuild#14141](pantsbuild#14141))

* [internal] Make `generate-lockfiles` goal generic ([pantsbuild#14122](pantsbuild#14122))

* [internal] Bump errno from 0.2.7 to 0.2.8 in /src/rust/engine ([pantsbuild#14137](pantsbuild#14137))

* [internal] Bump colored from 1.9.3 to 2.0.0 in /src/rust/engine ([pantsbuild#14138](pantsbuild#14138))

* [internal] Bump crossbeam-channel from 0.4.4 to 0.5.0 in /src/rust/engine ([pantsbuild#14139](pantsbuild#14139))

* [internal] Bump reqwest from 0.11.4 to 0.11.9 in /src/rust/engine ([pantsbuild#14135](pantsbuild#14135))

* [internal] Further tweak dependabot config ([pantsbuild#14132](pantsbuild#14132))

* [internal] python: use immutable_input_digests for protobuf codegen ([pantsbuild#13885](pantsbuild#13885))

* [internal] python: use immutable_input_digests for protobuf codegen ([pantsbuild#13885](pantsbuild#13885))

* [internal] No need for two f-strings/two strings. ([pantsbuild#14119](pantsbuild#14119))
illicitonion added a commit to illicitonion/pants that referenced this pull request Jan 22, 2022
Internal changes:

* upgrade to Rust v1.58.0 ([pantsbuild#14174](pantsbuild#14174))

* [internal] fix typos in codegen registration ([pantsbuild#14172](pantsbuild#14172))

* Remove per-language indirection for formatter plugins. ([pantsbuild#14166](pantsbuild#14166))

* Pulls `Coordinate` and `ArtifactRequirements` out into a separate file to avoid a circuilar import later ([pantsbuild#14164](pantsbuild#14164))

* Factors lockfile metadata code into class that supports multiple languages as well as versions ([pantsbuild#14116](pantsbuild#14116))

* Adds `.env` file to make vscode auto-imports work correctly out of the box ([pantsbuild#14130](pantsbuild#14130))

* [internal] Rename classes for `generate_lockfiles.py` for clarity ([pantsbuild#14218](pantsbuild#14218))

* [internal] Fix bad Find+Replace ([pantsbuild#14213](pantsbuild#14213))

* [internal] Bump libc from 0.2.106 to 0.2.112 in /src/rust/engine ([pantsbuild#14206](pantsbuild#14206))

* [internal] Bump tempfile from 3.2.0 to 3.3.0 in /src/rust/engine ([pantsbuild#14202](pantsbuild#14202))

* [internal] Bump criterion from 0.3.3 to 0.3.5 in /src/rust/engine ([pantsbuild#14205](pantsbuild#14205))

* [internal] Bump walkdir from 2.3.1 to 2.3.2 in /src/rust/engine ([pantsbuild#14204](pantsbuild#14204))

* [internal] Bump generic-array from 0.14.4 to 0.14.5 in /src/rust/engine ([pantsbuild#14203](pantsbuild#14203))

* [internal] Remove the minimum bucket size of batching to improve stability. ([pantsbuild#14210](pantsbuild#14210))

* [internal] Make `JvmLockfileRequest` generic ([pantsbuild#14201](pantsbuild#14201))

* [internal] add comment re clippy issue ([pantsbuild#14188](pantsbuild#14188))

* [internal] rename some codegen scopes to put language first ([pantsbuild#14187](pantsbuild#14187))

* [internal] Check for ambiguity when running `generate-lockfiles` ([pantsbuild#14178](pantsbuild#14178))

* [internal] Change JVM lockfile format ([pantsbuild#14175](pantsbuild#14175))

* [internal] go: rewrite third party package analysis to not use `go list` ([pantsbuild#14157](pantsbuild#14157))

* [internal] Hotfix `fmt` crashing on non-formattable targets ([pantsbuild#14168](pantsbuild#14168))

* [internal] Simplify `core/goals/package.py` filesystem API usage ([pantsbuild#14162](pantsbuild#14162))

* [internal] java/thrift: register union that was not registered ([pantsbuild#14159](pantsbuild#14159))

* [internal] Bump arc-swap from 1.3.0 to 1.5.0 in /src/rust/engine ([pantsbuild#14148](pantsbuild#14148))

* [internals] Bump tokio-rustls from 0.22.0 to 0.23.2 in /src/rust/engine ([pantsbuild#14149](pantsbuild#14149))

* [internal] Bump num_cpus from 1.13.0 to 1.13.1 in /src/rust/engine ([pantsbuild#14150](pantsbuild#14150))

* [internal] Bump tokio-util from 0.6.7 to 0.6.9 in /src/rust/engine ([pantsbuild#14151](pantsbuild#14151))

* [internal] Bump os_pipe from 0.9.2 to 1.0.0 in /src/rust/engine ([pantsbuild#14152](pantsbuild#14152))

* [internal] Introduce new `BuiltinGoal` subsystem type. ([pantsbuild#13991](pantsbuild#13991))

* [internal] Upgrade python type stubs packages ([pantsbuild#14143](pantsbuild#14143))

* [internal] Bump strum_macros from 0.20.1 to 0.23.1 in /src/rust/engine ([pantsbuild#14141](pantsbuild#14141))

* [internal] Make `generate-lockfiles` goal generic ([pantsbuild#14122](pantsbuild#14122))

* [internal] Bump errno from 0.2.7 to 0.2.8 in /src/rust/engine ([pantsbuild#14137](pantsbuild#14137))

* [internal] Bump colored from 1.9.3 to 2.0.0 in /src/rust/engine ([pantsbuild#14138](pantsbuild#14138))

* [internal] Bump crossbeam-channel from 0.4.4 to 0.5.0 in /src/rust/engine ([pantsbuild#14139](pantsbuild#14139))

* [internal] Bump reqwest from 0.11.4 to 0.11.9 in /src/rust/engine ([pantsbuild#14135](pantsbuild#14135))

* [internal] Further tweak dependabot config ([pantsbuild#14132](pantsbuild#14132))

* [internal] python: use immutable_input_digests for protobuf codegen ([pantsbuild#13885](pantsbuild#13885))

* [internal] python: use immutable_input_digests for protobuf codegen ([pantsbuild#13885](pantsbuild#13885))

* [internal] No need for two f-strings/two strings. ([pantsbuild#14119](pantsbuild#14119))
illicitonion added a commit to illicitonion/pants that referenced this pull request Jan 22, 2022
Internal changes:

* upgrade to Rust v1.58.0 ([pantsbuild#14174](pantsbuild#14174))

* [internal] fix typos in codegen registration ([pantsbuild#14172](pantsbuild#14172))

* Pulls `Coordinate` and `ArtifactRequirements` out into a separate file to avoid a circuilar import later ([pantsbuild#14164](pantsbuild#14164))

* Factors lockfile metadata code into class that supports multiple languages as well as versions ([pantsbuild#14116](pantsbuild#14116))

* Adds `.env` file to make vscode auto-imports work correctly out of the box ([pantsbuild#14130](pantsbuild#14130))

* [internal] Rename classes for `generate_lockfiles.py` for clarity ([pantsbuild#14218](pantsbuild#14218))

* [internal] Fix bad Find+Replace ([pantsbuild#14213](pantsbuild#14213))

* [internal] Bump libc from 0.2.106 to 0.2.112 in /src/rust/engine ([pantsbuild#14206](pantsbuild#14206))

* [internal] Bump tempfile from 3.2.0 to 3.3.0 in /src/rust/engine ([pantsbuild#14202](pantsbuild#14202))

* [internal] Bump criterion from 0.3.3 to 0.3.5 in /src/rust/engine ([pantsbuild#14205](pantsbuild#14205))

* [internal] Bump walkdir from 2.3.1 to 2.3.2 in /src/rust/engine ([pantsbuild#14204](pantsbuild#14204))

* [internal] Bump generic-array from 0.14.4 to 0.14.5 in /src/rust/engine ([pantsbuild#14203](pantsbuild#14203))

* [internal] Remove the minimum bucket size of batching to improve stability. ([pantsbuild#14210](pantsbuild#14210))

* [internal] Make `JvmLockfileRequest` generic ([pantsbuild#14201](pantsbuild#14201))

* [internal] add comment re clippy issue ([pantsbuild#14188](pantsbuild#14188))

* [internal] rename some codegen scopes to put language first ([pantsbuild#14187](pantsbuild#14187))

* [internal] Check for ambiguity when running `generate-lockfiles` ([pantsbuild#14178](pantsbuild#14178))

* [internal] Change JVM lockfile format ([pantsbuild#14175](pantsbuild#14175))

* [internal] go: rewrite third party package analysis to not use `go list` ([pantsbuild#14157](pantsbuild#14157))

* [internal] Hotfix `fmt` crashing on non-formattable targets ([pantsbuild#14168](pantsbuild#14168))

* [internal] Simplify `core/goals/package.py` filesystem API usage ([pantsbuild#14162](pantsbuild#14162))

* [internal] java/thrift: register union that was not registered ([pantsbuild#14159](pantsbuild#14159))

* [internal] Bump arc-swap from 1.3.0 to 1.5.0 in /src/rust/engine ([pantsbuild#14148](pantsbuild#14148))

* [internals] Bump tokio-rustls from 0.22.0 to 0.23.2 in /src/rust/engine ([pantsbuild#14149](pantsbuild#14149))

* [internal] Bump num_cpus from 1.13.0 to 1.13.1 in /src/rust/engine ([pantsbuild#14150](pantsbuild#14150))

* [internal] Bump tokio-util from 0.6.7 to 0.6.9 in /src/rust/engine ([pantsbuild#14151](pantsbuild#14151))

* [internal] Bump os_pipe from 0.9.2 to 1.0.0 in /src/rust/engine ([pantsbuild#14152](pantsbuild#14152))

* [internal] Introduce new `BuiltinGoal` subsystem type. ([pantsbuild#13991](pantsbuild#13991))

* [internal] Upgrade python type stubs packages ([pantsbuild#14143](pantsbuild#14143))

* [internal] Bump strum_macros from 0.20.1 to 0.23.1 in /src/rust/engine ([pantsbuild#14141](pantsbuild#14141))

* [internal] Make `generate-lockfiles` goal generic ([pantsbuild#14122](pantsbuild#14122))

* [internal] Bump errno from 0.2.7 to 0.2.8 in /src/rust/engine ([pantsbuild#14137](pantsbuild#14137))

* [internal] Bump colored from 1.9.3 to 2.0.0 in /src/rust/engine ([pantsbuild#14138](pantsbuild#14138))

* [internal] Bump crossbeam-channel from 0.4.4 to 0.5.0 in /src/rust/engine ([pantsbuild#14139](pantsbuild#14139))

* [internal] Bump reqwest from 0.11.4 to 0.11.9 in /src/rust/engine ([pantsbuild#14135](pantsbuild#14135))

* [internal] Further tweak dependabot config ([pantsbuild#14132](pantsbuild#14132))

* [internal] python: use immutable_input_digests for protobuf codegen ([pantsbuild#13885](pantsbuild#13885))

* [internal] python: use immutable_input_digests for protobuf codegen ([pantsbuild#13885](pantsbuild#13885))

* [internal] No need for two f-strings/two strings. ([pantsbuild#14119](pantsbuild#14119))

* `LockfileMetadata` - Replace flaky `_header_dict()` mechanism with more robust `header_attrs` mechanism ([pantsbuild#14229](pantsbuild#14229))
illicitonion added a commit that referenced this pull request Jan 22, 2022
Internal changes:

* upgrade to Rust v1.58.0 ([#14174](#14174))

* [internal] fix typos in codegen registration ([#14172](#14172))

* Pulls `Coordinate` and `ArtifactRequirements` out into a separate file to avoid a circuilar import later ([#14164](#14164))

* Factors lockfile metadata code into class that supports multiple languages as well as versions ([#14116](#14116))

* Adds `.env` file to make vscode auto-imports work correctly out of the box ([#14130](#14130))

* [internal] Rename classes for `generate_lockfiles.py` for clarity ([#14218](#14218))

* [internal] Fix bad Find+Replace ([#14213](#14213))

* [internal] Bump libc from 0.2.106 to 0.2.112 in /src/rust/engine ([#14206](#14206))

* [internal] Bump tempfile from 3.2.0 to 3.3.0 in /src/rust/engine ([#14202](#14202))

* [internal] Bump criterion from 0.3.3 to 0.3.5 in /src/rust/engine ([#14205](#14205))

* [internal] Bump walkdir from 2.3.1 to 2.3.2 in /src/rust/engine ([#14204](#14204))

* [internal] Bump generic-array from 0.14.4 to 0.14.5 in /src/rust/engine ([#14203](#14203))

* [internal] Remove the minimum bucket size of batching to improve stability. ([#14210](#14210))

* [internal] Make `JvmLockfileRequest` generic ([#14201](#14201))

* [internal] add comment re clippy issue ([#14188](#14188))

* [internal] rename some codegen scopes to put language first ([#14187](#14187))

* [internal] Check for ambiguity when running `generate-lockfiles` ([#14178](#14178))

* [internal] Change JVM lockfile format ([#14175](#14175))

* [internal] go: rewrite third party package analysis to not use `go list` ([#14157](#14157))

* [internal] Hotfix `fmt` crashing on non-formattable targets ([#14168](#14168))

* [internal] Simplify `core/goals/package.py` filesystem API usage ([#14162](#14162))

* [internal] java/thrift: register union that was not registered ([#14159](#14159))

* [internal] Bump arc-swap from 1.3.0 to 1.5.0 in /src/rust/engine ([#14148](#14148))

* [internals] Bump tokio-rustls from 0.22.0 to 0.23.2 in /src/rust/engine ([#14149](#14149))

* [internal] Bump num_cpus from 1.13.0 to 1.13.1 in /src/rust/engine ([#14150](#14150))

* [internal] Bump tokio-util from 0.6.7 to 0.6.9 in /src/rust/engine ([#14151](#14151))

* [internal] Bump os_pipe from 0.9.2 to 1.0.0 in /src/rust/engine ([#14152](#14152))

* [internal] Introduce new `BuiltinGoal` subsystem type. ([#13991](#13991))

* [internal] Upgrade python type stubs packages ([#14143](#14143))

* [internal] Bump strum_macros from 0.20.1 to 0.23.1 in /src/rust/engine ([#14141](#14141))

* [internal] Make `generate-lockfiles` goal generic ([#14122](#14122))

* [internal] Bump errno from 0.2.7 to 0.2.8 in /src/rust/engine ([#14137](#14137))

* [internal] Bump colored from 1.9.3 to 2.0.0 in /src/rust/engine ([#14138](#14138))

* [internal] Bump crossbeam-channel from 0.4.4 to 0.5.0 in /src/rust/engine ([#14139](#14139))

* [internal] Bump reqwest from 0.11.4 to 0.11.9 in /src/rust/engine ([#14135](#14135))

* [internal] Further tweak dependabot config ([#14132](#14132))

* [internal] python: use immutable_input_digests for protobuf codegen ([#13885](#13885))

* [internal] python: use immutable_input_digests for protobuf codegen ([#13885](#13885))

* [internal] No need for two f-strings/two strings. ([#14119](#14119))

* `LockfileMetadata` - Replace flaky `_header_dict()` mechanism with more robust `header_attrs` mechanism ([#14229](#14229))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants