-
Notifications
You must be signed in to change notification settings - Fork 25
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
refactor: package_index #93
Open
jjmaestro
wants to merge
14
commits into
GoogleContainerTools:main
Choose a base branch
from
jjmaestro:refactor-package_index
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
refactor: package_index #93
jjmaestro
wants to merge
14
commits into
GoogleContainerTools:main
from
jjmaestro:refactor-package_index
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
See bazelbuild/bazel#20369 While working on refactoring, I kept hitting rebase conflicts due to issues with the MODULE lock. I guess it's because the Bazel version in e2e tests is much lower than the current one with the fix (7.2) but still, I don't think it adds much to have the lock in e2e testing.
PR GoogleContainerTools#73 added the `_resolve` and this breaks the buildozer fix / autofix
It seemed like 75afff9 in GoogleContainerTools#47 added the new locks but as new files, that is, the old ones were left behind.
Add support for MODULE.bazel to the lock script and avoid printing an unnecessary (and annoying 😅) error when building in a "modern repo".
Add a `nolock` attribute to avoid getting annoying DEBUG messages for repos that we explicitly want to run without a lock.
* make apt/tests more readable by factoring out the parameters * add a "test suite macro" in each test file that group all of the unit tests in the file and prepends a "test suite prefix". IMHO this is better than using `unittest.suite` because we provide better naming than the automated `_test<NUMBER>` plus these better names are actual targets that can be executed one-by-one by name.
Add other nolock tests to exercise the package repos (the templates, etc).
* separate the script into a template file so it's easier to shellcheck and syntax highlight in editors. * shellcheck the script and remove all SC2086 warnings ("Double quote to prevent globbing and word splitting") * improve the buildozer help messages in the copy.sh script: * reduce duplication of buildozer command * add a more clear autofix bazel run command that can be easily copy-pasted * change some of the variable names in the copy.sh template for longer, easier to understand names (repo_name >> name, lock_label >> label) * move repo_name and workspace_relative_path into variables to reduce line length and improve readability
While working with some flaky mirrors and trying to figure out why they were failing I found the _fetch_package_index code a bit hard to follow so here's my attempt at streamlining it a bit: * Change the general flow of the for-loop so that we can directly set the reasons for failure in failed_attempts * Remove both integrity as an argument and as a return value since neither is ever used. * return the content of the Packages index instead of the path, since we already have the repository_context. This way we don't need rctx anywhere else. * Reword failure messages adding more context and debug information * Shorter lines and templated strings, trying to make the code easier to read and follow.
* remove the `state` "intermediary `struct`" in `package_resolution.bzl` since it wasn't used / needed. * refactor and move _set_dict from `util.bzl` to a `_package_set` method * use `dict .get()` with default values instead of the "nested `if`s" * renamed `package()` to `package_get` and make it return all package versions when the version is not specified so we can remove `_package_versions` * reordeder `(name, version, arch)` args to match the order of the index keys `(arch, name, version)`
Add testing for package_index mocking the external / side effects (downloads, decompression, etc).
The package resolution debugging that e.g. checks the package dependencies should all be within package_resolution.bzl _resolve_all() and not "leak out" returning the information that's only needed for debugging / logging. Also: * reduce the verbosity of the optional dependencies warning by just printing one message per root package instead of one per package. * break up the long lines to build the error messages and remove the "# buildifier: disable=print".
* move version constraint parsing from package_resolution to its own _parse_version_and_constraint method in version.bzl * refactor _version_relop into a compare method in version.bzl plus a VERSION_OPERATORS dict so that (1) we use the operator strings everywhere and (2) we can use the keys to validate the operators.
Previously we had: ```starlark pkgindex = package_index.new(rctx, sources = sources, archs = manifest["archs"]) pkgresolution = package_resolution.new(index = pkgindex) ``` And none of the code of package_resolution was used anywhere but in resolve.bzl and after initializing the `pkgindex`. Also, it makes sense since we are building the index from the manifest and once we have the index we use it to resolve the packages and populate the lock.
jjmaestro
force-pushed
the
refactor-package_index
branch
from
September 22, 2024 18:43
95da3ce
to
88623f1
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Note
Stacked on top of #92
refactor: package_index _fetch_package_index
While working with some flaky mirrors and trying to figure out why they were failing I found the _fetch_package_index code a bit hard to follow so here's my attempt at streamlining it a bit:
Change the general flow of the for-loop so that we can directly set the reasons for failure in failed_attempts
Remove both integrity as an argument and as a return value since neither is ever used.
return the content of the Packages index instead of the path, since we already have the repository_context. This way we don't need rctx anywhere else.
Reword failure messages adding more context and debug information
Shorter lines and templated strings, trying to make the code easier to read and follow.
refactor: package_index struct
remove the
state
"intermediarystruct
" inpackage_resolution.bzl
since it wasn't used / needed.refactor and move _set_dict from
util.bzl
to a_package_set
methoduse
dict .get()
with default values instead of the "nestedif
s"renamed
package()
topackage_get
and make it return all package versions when the version is not specified so we can remove_package_versions
reordeder
(name, version, arch)
args to match the order of the index keys(arch, name, version)
tests: package_index_test
Add testing for package_index mocking the external / side effects (downloads, decompression, etc).
refactor: centralize and improve package_resolution logging
The package resolution debugging that e.g. checks the package dependencies should all be within
package_resolution.bzl
_resolve_all()
and not "leak out" returning the information that's only needed for debugging / logging.Also:
reduce the verbosity of the optional dependencies warning by just printing one message per root package instead of one per package.
break up the long lines to build the error messages and remove the
# buildifier: disable=print
.refactor: version
move version constraint parsing from
package_resolution
to its own_parse_version_and_constraint
method inversion.bzl
refactor
_version_relop
into a compare method inversion.bzl
plus aVERSION_OPERATORS
dict
so that (1) we use the operator strings everywhere and (2) we can use the keys to validate the operators.refactor: combine package_resolution and package_index
Previously we had:
And none of the code of
package_resolution
was used anywhere but inresolve.bzl
and after initializing thepkgindex
.Also, it makes sense since we are building the index from the manifest and once we have the index we use it to resolve the packages and populate the lock.