[7.2.0] Add a git merge driver for MODULE.bazel.lock
#22650
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Adds a `jq` script to `scripts/` that merges any number of `MODULE.bazel.lock` files without using Bazel or reading the corresponding `MODULE.bazel` files.
The lockfile docs now have a section explaining the steps needed to set up this script as a custom merger driver for Git, which means that merge conflicts in `MODULE.bazel.lock` files will always be resolved automatically. Note that resolution may emit lockfiles with redundant information that will be dropped by subsequent Bazel invocations.
When Bazel encounters an error during lockfile parsing that could be caused by a merge conflict, it emits a different error message with a link to the docs. This required fixing the following kind of server crash when a conflict marker occurs inside a `recordedFileInputs` object:
```
FATAL: bazel crashed due to an internal error. Printing stack trace:
java.lang.RuntimeException: Unrecoverable error while evaluating node 'com.google.devtools.build.lib.bazel.bzlmod.BazelLockFileValue$$Lambda/0x000000f8011da998@314cd9ee' (requested by nodes 'RegistryKey{url=https://bcr.bazel.build/}')
at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:557)
at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:426)
at java.base/java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(ForkJoinTask.java:1403)
at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:387)
at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1312)
at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1843)
at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1808)
at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:188)
Caused by: java.lang.IllegalArgumentException: the provided path should be absolute in the filesystem
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:143)
at com.google.devtools.build.lib.rules.repository.RepoRecordedInput$RepoCacheFriendlyPath.createOutsideWorkspace(RepoRecordedInput.java:202)
at com.google.devtools.build.lib.rules.repository.RepoRecordedInput$RepoCacheFriendlyPath.parse(RepoRecordedInput.java:222)
at com.google.devtools.build.lib.rules.repository.RepoRecordedInput$File$1.parse(RepoRecordedInput.java:265)
at com.google.devtools.build.lib.bazel.bzlmod.GsonTypeAdapterUtil$11.read(GsonTypeAdapterUtil.java:376)
at com.google.devtools.build.lib.bazel.bzlmod.GsonTypeAdapterUtil$11.read(GsonTypeAdapterUtil.java:367)
at com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.read(TypeAdapterRuntimeTypeWrapper.java:41)
at com.google.gson.internal.bind.MapTypeAdapterFactory$Adapter.read(MapTypeAdapterFactory.java:186)
at com.google.gson.internal.bind.MapTypeAdapterFactory$Adapter.read(MapTypeAdapterFactory.java:145)
at com.google.devtools.build.lib.bazel.bzlmod.DelegateTypeAdapterFactory$1.read(DelegateTypeAdapterFactory.java:133)
at com.google.devtools.build.lib.bazel.bzlmod.LockFileModuleExtension_GsonTypeAdapter.read(LockFileModuleExtension_GsonTypeAdapter.java:171)
at com.google.devtools.build.lib.bazel.bzlmod.LockFileModuleExtension_GsonTypeAdapter.read(LockFileModuleExtension_GsonTypeAdapter.java:17)
at com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.read(TypeAdapterRuntimeTypeWrapper.java:41)
at com.google.gson.internal.bind.MapTypeAdapterFactory$Adapter.read(MapTypeAdapterFactory.java:187)
at com.google.gson.internal.bind.MapTypeAdapterFactory$Adapter.read(MapTypeAdapterFactory.java:145)
at com.google.devtools.build.lib.bazel.bzlmod.DelegateTypeAdapterFactory$1.read(DelegateTypeAdapterFactory.java:133)
at com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.read(TypeAdapterRuntimeTypeWrapper.java:41)
at com.google.gson.internal.bind.MapTypeAdapterFactory$Adapter.read(MapTypeAdapterFactory.java:187)
at com.google.gson.internal.bind.MapTypeAdapterFactory$Adapter.read(MapTypeAdapterFactory.java:145)
at com.google.devtools.build.lib.bazel.bzlmod.DelegateTypeAdapterFactory$1.read(DelegateTypeAdapterFactory.java:133)
at com.google.devtools.build.lib.bazel.bzlmod.BazelLockFileValue_GsonTypeAdapter.read(BazelLockFileValue_GsonTypeAdapter.java:129)
at com.google.devtools.build.lib.bazel.bzlmod.BazelLockFileValue_GsonTypeAdapter.read(BazelLockFileValue_GsonTypeAdapter.java:15)
at com.google.gson.Gson.fromJson(Gson.java:991)
at com.google.gson.Gson.fromJson(Gson.java:956)
at com.google.gson.Gson.fromJson(Gson.java:905)
at com.google.gson.Gson.fromJson(Gson.java:876)
at com.google.devtools.build.lib.bazel.bzlmod.BazelLockFileFunction.getLockfileValue(BazelLockFileFunction.java:93)
at com.google.devtools.build.lib.bazel.bzlmod.BazelLockFileFunction.compute(BazelLockFileFunction.java:73)
at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:468)
... 7 more
```
Alternatives considered:
* Letting Bazel resolve the conflict would require building knowledge about particular version control systems and their conflict style into Bazel. It would also either require the user to resolve conflicts in `MODULE.bazel` first or deviate from the current behavior that the lockfile is not updated when any Bzlmod error is encountered. The jq script can be used as is by every VCS with merge driver support and resolves the conflict in `MODULE.bazel.lock` independently of `MODULE.bazel`.
* Implementing the git merge driver as a `bazel mod` subcommand. This could be the source of intransparent slowdowns during regular git operations, which may even be triggered by other tools such as IDEs. The jq script is very fast.
* Implementing the merger as a Go binary in buildtools would replace the ubiquitous jq tool with a special purpose binary while also not solving the problem that per-user action is required once to register a custom merge driver.
Implements https://docs.google.com/document/d/1TjA7-M5njkI1F38IC0pm305S9EOmxcUwaCIvaSmansg/edit#heading=h.5mcn15i0e1ch
RELNOTES: Git merge conflicts in `MODULE.bazel.lock` files can be resolved automatically. See https://bazel.build/external/lockfile#automatic-resolution for the required setup.
Closes bazelbuild#22428.
PiperOrigin-RevId: 640596606
Change-Id: I20659e3e53a7d8f2529f2ad5a3e7f258d7af026d
bazel-io
added
team-ExternalDeps
Wyverald
approved these changes
Jun 5, 2024
github-actions
bot
removed
the
awaiting-review
keertk
pushed a commit
that referenced
this pull request
Jun 10, 2024
Release Notes: Configurability: + aquery: `//foo:bar` now means "all configured targets with label `//foo:bar`" instead of "choose an arbitrary configured target with label `//foo:bar`". This is in line with cquery behavior. (#22135) + Added a new flag `--incompatible_disable_native_repo_rules` to disable native repo rule usage in WORKSPACE. All native repo rules now have a Starlark counterpart that can be used in both WORKSPACE and Bzlmod; see #22080 for more details. (#22203) + Starlark command-line flags can now be referred to through `alias` targets. (#22212) ExternalDeps: + bzlmod `git_repository` now accepts the `strip_prefix` arg and passes it to the underlying `git_repository` call. (#22137) + Added a new `include()` directive to `MODULE.bazel` files, which allows the root module file to be divided into multiple segments. (#22204) + Fixed certain deadlocks in repo fetching with worker threads (`--experimental_worker_for_repo_fetching=auto`). (#22261) + `print` statements in module files are now only executed for the root module and modules subject to non-registry overrides (e.g. `local_path_override`). (#22263) + The new `refresh` value for `--lockfile_mode` behaves like the `update` mode, but additionally forces a refresh of mutable registry content (yanked versions and missing module versions) when switched to or from time to time while enabled. (#22371) + `Label` instances passed to `print` or `fail` as positional arguments are now formatted with apparent repository names (optimized for human readability). (#22460) + Changes to environment variables read via `getenv` now correctly invalidate module extensions. (#22541) + Git merge conflicts in `MODULE.bazel.lock` files can be resolved automatically. See https://bazel.build/external/lockfile#automatic-resolution for the required setup. (#22650) OSS: + Bazel on Linux and BSD now respects the XDG_CACHE_HOME environment variable instead of assuming that ~/.cache/bazel is writable. (#21817) Performance: + Paths in the Linux sandbox are now again identical to those outside the sandbox, even with `--incompatible_sandbox_hermetic_tmp`. (#22407) Remote-Exec: + The combined coverage report produced via `--combined_report=lcov` is now announced on the BES via the new `CoverageReport` event. (#22327) + The compact and full execution logs now contain start times for spawns (if available). (#22341) Rules-CPP: + The default Unix C++ toolchain now supports the `parse_headers` feature to validate header files with `--process_headers_in_dependencies`. (#22369) Starlark-Interpreter: + Starlark `min` and `max` buitins now allow a `key` callback, similarly to `sorted`. (#21960) Acknowledgements: This release contains contributions from many people at Google, as well as bazel.build machine account, Brentley Jones, Cameron Martin, Daniel Wagner-Hall, Douglas Thor, Fabian Meumertzheim, George Gensure, hvd, Isaac Torres, Keith Smiley, Mark Elliot, oquenchil, Romain Chossart, Son Luong Ngoc, Spencer Putt, Thomas Weischuh, Xdng Yng, Xùdōng Yáng, Zheng Wei Tan.
copybara-service bot
pushed a commit
that referenced
this pull request
Jun 10, 2024
Release Notes: Configurability: + aquery: `//foo:bar` now means "all configured targets with label `//foo:bar`" instead of "choose an arbitrary configured target with label `//foo:bar`". This is in line with cquery behavior. (#22135) + Added a new flag `--incompatible_disable_native_repo_rules` to disable native repo rule usage in WORKSPACE. All native repo rules now have a Starlark counterpart that can be used in both WORKSPACE and Bzlmod; see #22080 for more details. (#22203) + Starlark command-line flags can now be referred to through `alias` targets. (#22212) ExternalDeps: + bzlmod `git_repository` now accepts the `strip_prefix` arg and passes it to the underlying `git_repository` call. (#22137) + Added a new `include()` directive to `MODULE.bazel` files, which allows the root module file to be divided into multiple segments. (#22204) + Fixed certain deadlocks in repo fetching with worker threads (`--experimental_worker_for_repo_fetching=auto`). (#22261) + `print` statements in module files are now only executed for the root module and modules subject to non-registry overrides (e.g. `local_path_override`). (#22263) + The new `refresh` value for `--lockfile_mode` behaves like the `update` mode, but additionally forces a refresh of mutable registry content (yanked versions and missing module versions) when switched to or from time to time while enabled. (#22371) + `Label` instances passed to `print` or `fail` as positional arguments are now formatted with apparent repository names (optimized for human readability). (#22460) + Changes to environment variables read via `getenv` now correctly invalidate module extensions. (#22541) + Git merge conflicts in `MODULE.bazel.lock` files can be resolved automatically. See https://bazel.build/external/lockfile#automatic-resolution for the required setup. (#22650) OSS: + Bazel on Linux and BSD now respects the XDG_CACHE_HOME environment variable instead of assuming that ~/.cache/bazel is writable. (#21817) Performance: + Paths in the Linux sandbox are now again identical to those outside the sandbox, even with `--incompatible_sandbox_hermetic_tmp`. (#22407) Remote-Exec: + The combined coverage report produced via `--combined_report=lcov` is now announced on the BES via the new `CoverageReport` event. (#22327) + The compact and full execution logs now contain start times for spawns (if available). (#22341) Rules-CPP: + The default Unix C++ toolchain now supports the `parse_headers` feature to validate header files with `--process_headers_in_dependencies`. (#22369) Starlark-Interpreter: + Starlark `min` and `max` buitins now allow a `key` callback, similarly to `sorted`. (#21960) Acknowledgements: This release contains contributions from many people at Google, as well as bazel.build machine account, Brentley Jones, Cameron Martin, Daniel Wagner-Hall, Douglas Thor, Fabian Meumertzheim, George Gensure, hvd, Isaac Torres, Keith Smiley, Mark Elliot, oquenchil, Romain Chossart, Son Luong Ngoc, Spencer Putt, Thomas Weischuh, Xdng Yng, Xùdōng Yáng, Zheng Wei Tan.
rdeushane
pushed a commit
to SibrosTech/bazel
that referenced
this pull request
Jun 19, 2024
Release Notes: Configurability: + aquery: `//foo:bar` now means "all configured targets with label `//foo:bar`" instead of "choose an arbitrary configured target with label `//foo:bar`". This is in line with cquery behavior. (bazelbuild#22135) + Added a new flag `--incompatible_disable_native_repo_rules` to disable native repo rule usage in WORKSPACE. All native repo rules now have a Starlark counterpart that can be used in both WORKSPACE and Bzlmod; see bazelbuild#22080 for more details. (bazelbuild#22203) + Starlark command-line flags can now be referred to through `alias` targets. (bazelbuild#22212) ExternalDeps: + bzlmod `git_repository` now accepts the `strip_prefix` arg and passes it to the underlying `git_repository` call. (bazelbuild#22137) + Added a new `include()` directive to `MODULE.bazel` files, which allows the root module file to be divided into multiple segments. (bazelbuild#22204) + Fixed certain deadlocks in repo fetching with worker threads (`--experimental_worker_for_repo_fetching=auto`). (bazelbuild#22261) + `print` statements in module files are now only executed for the root module and modules subject to non-registry overrides (e.g. `local_path_override`). (bazelbuild#22263) + The new `refresh` value for `--lockfile_mode` behaves like the `update` mode, but additionally forces a refresh of mutable registry content (yanked versions and missing module versions) when switched to or from time to time while enabled. (bazelbuild#22371) + `Label` instances passed to `print` or `fail` as positional arguments are now formatted with apparent repository names (optimized for human readability). (bazelbuild#22460) + Changes to environment variables read via `getenv` now correctly invalidate module extensions. (bazelbuild#22541) + Git merge conflicts in `MODULE.bazel.lock` files can be resolved automatically. See https://bazel.build/external/lockfile#automatic-resolution for the required setup. (bazelbuild#22650) OSS: + Bazel on Linux and BSD now respects the XDG_CACHE_HOME environment variable instead of assuming that ~/.cache/bazel is writable. (bazelbuild#21817) Performance: + Paths in the Linux sandbox are now again identical to those outside the sandbox, even with `--incompatible_sandbox_hermetic_tmp`. (bazelbuild#22407) Remote-Exec: + The combined coverage report produced via `--combined_report=lcov` is now announced on the BES via the new `CoverageReport` event. (bazelbuild#22327) + The compact and full execution logs now contain start times for spawns (if available). (bazelbuild#22341) Rules-CPP: + The default Unix C++ toolchain now supports the `parse_headers` feature to validate header files with `--process_headers_in_dependencies`. (bazelbuild#22369) Starlark-Interpreter: + Starlark `min` and `max` buitins now allow a `key` callback, similarly to `sorted`. (bazelbuild#21960) Acknowledgements: This release contains contributions from many people at Google, as well as bazel.build machine account, Brentley Jones, Cameron Martin, Daniel Wagner-Hall, Douglas Thor, Fabian Meumertzheim, George Gensure, hvd, Isaac Torres, Keith Smiley, Mark Elliot, oquenchil, Romain Chossart, Son Luong Ngoc, Spencer Putt, Thomas Weischuh, Xdng Yng, Xùdōng Yáng, Zheng Wei Tan.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adds a
jqscript toscripts/that merges any number ofMODULE.bazel.lockfiles without using Bazel or reading the correspondingMODULE.bazelfiles.The lockfile docs now have a section explaining the steps needed to set up this script as a custom merger driver for Git, which means that merge conflicts in
MODULE.bazel.lockfiles will always be resolved automatically. Note that resolution may emit lockfiles with redundant information that will be dropped by subsequent Bazel invocations.When Bazel encounters an error during lockfile parsing that could be caused by a merge conflict, it emits a different error message with a link to the docs. This required fixing the following kind of server crash when a conflict marker occurs inside a
recordedFileInputsobject:Alternatives considered:
MODULE.bazelfirst or deviate from the current behavior that the lockfile is not updated when any Bzlmod error is encountered. The jq script can be used as is by every VCS with merge driver support and resolves the conflict inMODULE.bazel.lockindependently ofMODULE.bazel.bazel modsubcommand. This could be the source of intransparent slowdowns during regular git operations, which may even be triggered by other tools such as IDEs. The jq script is very fast.Implements https://docs.google.com/document/d/1TjA7-M5njkI1F38IC0pm305S9EOmxcUwaCIvaSmansg/edit#heading=h.5mcn15i0e1ch
RELNOTES: Git merge conflicts in
MODULE.bazel.lockfiles can be resolved automatically. See https://bazel.build/external/lockfile#automatic-resolution for the required setup.Closes #22428.
PiperOrigin-RevId: 640596606
Change-Id: I20659e3e53a7d8f2529f2ad5a3e7f258d7af026d
Commit 3187250