Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update bazelversion #439

Open
wants to merge 90 commits into
base: release-7.0.1
Choose a base branch
from
Open

Update bazelversion #439

wants to merge 90 commits into from

Conversation

iancha1992
Copy link
Owner

No description provided.

fmeum and others added 30 commits December 21, 2023 19:22
bazelbuild#20636)

… exist.

First, we removed the suggestion to "use `query`" as it was misleading
to some users.

Second, we fine-tuned the suggested target behaviors. The existing
method retrieves only the closest target within a reasonable distance.
We updated this to return potentially more targets, with a preference
for those that are particularly close.

Performance-wise, this should be not diverge much from the status quo.
The existing behavior already checks edit-distance for each target in
the package against the requested target. This adds minimal behavior on
top of that to identify the most likely candidate(s).

RELNOTES: None.
PiperOrigin-RevId: 591302621
Change-Id: I9bd272b68def2f9a75db01061287697d23936bca

Cherry-picked from ef98ef9
Hopefully this will fix our presubmits!

---------

This is a copy of bazelbuild#20733 for the
7.1.0 branch.

Co-authored-by: Googler <pcloudy@google.com>
Co-authored-by: Googler <hvd@google.com>
Co-authored-by: John Cater <jcater@google.com>
Co-authored-by: David Ostrovsky <David.Ostrovsky@gmail.com>
Co-authored-by: Justin Horvitz <jhorvitz@google.com>
Previously, if a config_setting referenced a Starlark setting through an
alias, it was always evaluated as if the setting had its default value.
Now, it is evaluated correctly.

This is done by looking up the value of the build setting in the
configuration based on its own label as specified in
BuildSettingProvider.

Fixes bazelbuild#13463 .

RELNOTES: None.
Commit
bazelbuild@d44c3f9

PiperOrigin-RevId: 585658985
Change-Id: Id534cd95282355f1143302bf703145bb53708a41

Co-authored-by: Googler <lberki@google.com>
Co-authored-by: Ian (Hee) Cha <heec@google.com>
…ory_rule` (bazelbuild#20732)

Makes the error message less confusing when referencing an existing
symbol that happens to be a macro, not a repository rule.

Closes bazelbuild#20657.

Commit
bazelbuild@69fa6cb

PiperOrigin-RevId: 595434847
Change-Id: Ifc37a65685c0196301d79a439f3245037cf39e21

Co-authored-by: Fabian Meumertzheim <fabian@meumertzhe.im>
Co-authored-by: Xùdōng Yáng <wyverald@gmail.com>
…elbuild#20633)

During `bazel query`, `Label#getDisplayForm(mainRepoMapping)` might be
called many many times. We can optimize for that case without
sacrificing too much memory by computing a reverse mapping for the main
repo mapping only.

Fixes bazelbuild#20613.

Closes bazelbuild#20617.

Commit
bazelbuild@d9169ab

PiperOrigin-RevId: 592607904
Change-Id: Iaaa709a51fe39556f03408080c1fe5e73689b761

Co-authored-by: Googler <wyv@google.com>
Co-authored-by: Ian (Hee) Cha <heec@google.com>
Co-authored-by: Xùdōng Yáng <wyverald@gmail.com>
Fixes bazelbuild#20743

Commit
bazelbuild@1a0b3a0

PiperOrigin-RevId: 595935153
Change-Id: I0409552aa92f3886c5abf3bd3ce50d67594dab7e

Co-authored-by: Googler <pcloudy@google.com>
…ug` (bazelbuild#20769)

The `--sandbox_debug` output for the Linux sandbox now also contains a
copyable command that drops the user into an interactive shell in an
identical sandboxed environment. This is in particular meant to address
the increased complexity of the bind mount structure incurred by the
flip of `--incompatible_sandbox_hermetic_tmp`.

Closes bazelbuild#20708.

Commit
bazelbuild@48ce49b

PiperOrigin-RevId: 595457357
Change-Id: I6dfd410895b93fce67b9666c76c5f7757e77dc3a

Co-authored-by: Fabian Meumertzheim <fabian@meumertzhe.im>
…manifested themselves when the output base was under /tmp (bazelbuild#20766)

1. Virtual action inputs didn't work. This was a fairly simple omission
in the path transformation logic.
2. Artifacts which are resolved symlinks (i.e. declared using
`declare_file`) did not work. This is fixed by keeping track of the
realpath of the symlink in `FileArtifactValue` / `TreeArtifactValue`.

Fixes bazelbuild#20515 and bazelbuild#20518.

RELNOTES: None.
Commit
bazelbuild@fb6658c

PiperOrigin-RevId: 595145191
Change-Id: I833025928374c78bc719d8e3be1efb2b2950b9f1

Co-authored-by: Googler <lberki@google.com>
Use a pre-allocated array to hold the intermediate transfers to avoid
allocations. Replace some of RxJava code with Futures to avoid RxJava
overheads.

This improves the perfromance of prefetchInputs on a large set of inputs
from ~400ms to ~16ms.

Fixes bazelbuild#20555.

Closes bazelbuild#20557.

Commit
bazelbuild@915fb3e

PiperOrigin-RevId: 595226013
Change-Id: If5296fa6b3c0166b95cfca4281255e523724a41f

Co-authored-by: Chi Wang <chiwang@google.com>
Co-authored-by: Xùdōng Yáng <wyverald@gmail.com>
Co-authored-by: Ian (Hee) Cha <heec@google.com>
Co-authored-by: Yun Peng <pcloudy@google.com>
`bazel mod show_extension @foo//:extensions.bzl%go_sdk`resulted in the
crash:
```
java.util.IllegalFormatConversionException: g != java.lang.String
```

Also makes an error more readable by swapping a `:`.

Closes bazelbuild#20627.

Commit
bazelbuild@6e6e6f8

PiperOrigin-RevId: 592942775
Change-Id: Ida5c234413c1647f81d3702bb9deeedcdd93df12

Co-authored-by: Fabian Meumertzheim <fabian@meumertzhe.im>
Co-authored-by: Xùdōng Yáng <wyverald@gmail.com>
Co-authored-by: Ian (Hee) Cha <heec@google.com>
…on (bazelbuild#20630)

The previous logic missed to normalize cases such as `"extension.bzl"`
and `"//extension.bzl"`, thus resulting in crashes if these styles are
mixed as well as invalid buildozer commands for `use_repo` fixing.

Instead of enumerating cases, parse the label and emit it in unambiguous
canonical form with a leading `@` stripped.

Closes bazelbuild#20482.

Commit
bazelbuild@71787cf

PiperOrigin-RevId: 592666970
Change-Id: Ieea34b27a187545a11107a334bbae14fef974ae8

Co-authored-by: Fabian Meumertzheim <fabian@meumertzhe.im>
Co-authored-by: Ian (Hee) Cha <heec@google.com>
Co-authored-by: Xùdōng Yáng <wyverald@gmail.com>
…zelbuild#20662)

When the user interrupts a build with Ctrl+C during the download or
extraction of a file, no Starlark error should be printed. This is
achieved by propagating `InterruptedException`s instead of failing the
repository rule with an `IOException`.

Closes bazelbuild#20023.

Commit
bazelbuild@a2d3f20

PiperOrigin-RevId: 578869315
Change-Id: I9dd901ed87ed00c239599877816c6836688c1e16

Co-authored-by: Fabian Meumertzheim <fabian@meumertzhe.im>
Co-authored-by: Xùdōng Yáng <wyverald@gmail.com>
Co-authored-by: Ian (Hee) Cha <heec@google.com>
…onment (bazelbuild#20643)

When a repository rule is fetch attributes are iterated over in
`enforceLabelAttributes` to prepopulate the environment, restarting the
fetch each time a new dependency is discovered.

This is faster than calling `repository_ctx.path(...)` early in the
repository rule implementation function but still has considerable
overhead.

This PR defers throwing `NeedsSkyframeRestartException` till after every
attribute has been processed, greatly reducing the number of restarts
and iterations.

In an internal project these optimisations are particularly noticeable.
Before: 2min 8s, 2min 7s
After: 1min 35s, 1min 27s

Closes bazelbuild#20434.

Commit
bazelbuild@e8ac96a

PiperOrigin-RevId: 588090528
Change-Id: I7917b137d6e60b6d6a73189cf396418a85b3ec28

Co-authored-by: Jordan Mele <mele@canva.com>
Co-authored-by: Xùdōng Yáng <wyverald@gmail.com>
…d#20803)

To make it easier to debug
bazelbuild#20748.

Closes bazelbuild#20783.

Commit
bazelbuild@18c8839

PiperOrigin-RevId: 596824058
Change-Id: I1dd6b940c30bd6c4b99904d19667f48a389b3a41

Co-authored-by: Chi Wang <chiwang@google.com>
)

Both the `lcov_merger` and the `collect_cc_coverage` script are obtained
from "well-known" implicit attributes of Starlark rules and do not
require any further special handling.

Removing this special handling also fixes a fringe bug: When a `cc_test`
is wrapped in another test target that applies a transition to it, the
removed logic set the `LCOV_MERGER` variable to the untransitioned path
of the `lcov_merger`, which will then not be available in the test
sandbox since it is only added as a runfile, but the test action
references it via its exec path. This resulted in a "file not found"
error that broke coverage.

Closes bazelbuild#20349.

Commit
bazelbuild@591d125

PiperOrigin-RevId: 589252093
Change-Id: I810f1b414dcfedcc8930a74390ef5c9bfd2cd030

Co-authored-by: Fabian Meumertzheim <fabian@meumertzhe.im>
Co-authored-by: Ian (Hee) Cha <heec@google.com>
Speculative fix for this crash:

```
FATAL: bazel crashed due to an internal error. Printing stack trace:
java.lang.RuntimeException: Unrecoverable error while evaluating node 'BZLMOD_REPO_RULE:@@_main~remote_jdk8_repos~remote_jdk8_macos_aarch64_toolchain_config_repo' (requested by nodes 'REPOSITORY_DIRECTORY:@@_main~remote_jdk8_repos~remote_jdk8_macos_aarch64_toolchain_config_repo')
	at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:550)
	at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:414)
	at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinPool.scan(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source)
Caused by: java.lang.NullPointerException: Cannot invoke "com.google.devtools.build.lib.bazel.bzlmod.BzlmodRepoRuleValue.getRule()" because the return value of "com.google.devtools.build.lib.skyframe.BzlmodRepoRuleFunction.createRuleFromSpec(com.google.devtools.build.lib.bazel.bzlmod.RepoSpec, net.starlark.java.eval.StarlarkSemantics, com.google.devtools.build.skyframe.SkyFunction$Environment)" is null
	at com.google.devtools.build.lib.skyframe.BzlmodRepoRuleFunction.compute(BzlmodRepoRuleFunction.java:151)
	at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:461)
	... 7 more
```

Closes bazelbuild#20807.

Commit
bazelbuild@35ba396

PiperOrigin-RevId: 597050428
Change-Id: Ie8e5c3800be1a7adbacab4ac115acfd308c0f59e

Co-authored-by: Fabian Meumertzheim <fabian@meumertzhe.im>
…t methods. (bazelbuild#20788)

Avoids an eager string conversion (and associated memory allocation)
when in a memory-sensitive context.

PiperOrigin-RevId: 585042874
Change-Id: I060303c844188653846c6849964bdd73aec0560b
For now, the sole implementation is ExpandedSpawnLogContext, which
subsumes the old SpawnLogContext class, as well as some of the logic
around log formats currently in SpawnLogModule.

In a followup, a separate CompactSpawnLogContext implementation will be
introduced. This will let us experiment with a new log format while
minimizing the chance of accidentally breaking the existing formats.

PiperOrigin-RevId: 588444807
Change-Id: I75bc2a577ec45202baf95c950a257eaf420cbeb0
…ild#20844)

When a flag specified with `common` in a `.bazelrc` file expanded to a
flag unsupported by the current command, it resulted in an error rather
than the flag being ignored.

Fixes
bazelbuild#18130 (comment)

Closes bazelbuild#20720.

Commit
bazelbuild@b303144

PiperOrigin-RevId: 597271727
Change-Id: Ieaba4dc00e13495a859e1eedf802759ad7dbf774

Co-authored-by: Fabian Meumertzheim <fabian@meumertzhe.im>
)

This seems superfluous because we already have one for their Starlark
implementation function, but sometimes significant amount of time is
spent in RepositoryFunction just to determine that the repository is in
fact up-to-date.

In this case, it's useful to see in which repository the time is spent.

RELNOTES: None.
PiperOrigin-RevId: 597165396
Change-Id: I2c327450459a41dc7eec6ec2ac89c186cb43b34f
As of JDK 14, `OperatingSystemMXBean` provides information about system
memory that is container-aware. Outside containers, it uses the same
mechanisms as Bazel to determine available RAM (`/proc/meminfo` on
Linux, `hw.memsize` on macOS) and can thus be used as a drop-in
replacement for the custom implementation.

A small caveat is that Bazel's macOS RAM estimate was based on
converting bytes to "MB" via a divisor of `1000^2` instead of `1024^2`,
resulting in a consistent overestimate compared to an identical Linux
machine that is now corrected.

This opportunity was missed in
bazelbuild#16512 since
`OperatingSystemMXBean` is based on a complete Java implementation of
cgroups handling and doesn't go through the `os::total_memory` or
`os::physical_memory` Hotspot functions.

RELNOTES[INC]:
* On Linux, Bazel's RAM estimate for the host machine is now aware of
container resource limits.
* On macOS, Bazel no longer consistently overestimates the total RAM by
~5% (`1024^2/1000^2`).
* On Windows, Bazel's RAM estimate is now generally more accurate as it
is no longer influenced by JVM heuristics.

Fixes bazelbuild#3886

Closes bazelbuild#20435.

Commit
bazelbuild@2f3cdc5

PiperOrigin-RevId: 588718034
Change-Id: I2daafa0567740a1b149ca8756ec27f102129283c

Co-authored-by: Fabian Meumertzheim <fabian@meumertzhe.im>
…ested (bazelbuild#20762)

This adds bazel support for fixing
bazelbuild/intellij#5845. Once released, the
necessary changes will need to be made to the IntelliJ plugin.

PiperOrigin-RevId: 592136548
Change-Id: I6158f379e76b61e75ca51f34888aeecaf0303cc6

Co-authored-by: Googler <hvd@google.com>
…tall_base` (bazelbuild#20648)

Currently if the `--install_base` path passed is not writable by the
user invoking Bazel, the Bazel client crashes:
```console
❯ bazel --install_base=/some/read/only/path version
FATAL: failed to set timestamp on '/some/read/only/path': (error: 30): Read-only file system
```

This happens because the Bazel client (unconditionally) attempts to
update the `mtime` of this path:

https://github.com/bazelbuild/bazel/blob/a3c677dfea2de636a719d50345a5a97af96fae60/src/main/cpp/blaze.cc#L1010-L1021

This commit updates the client to ignore such errors. See bazelbuild#20373 for
context.

Closes bazelbuild#20442.

Commit
bazelbuild@7f782e3

PiperOrigin-RevId: 591266054
Change-Id: If53e7cad48cb62406f7883f72b413e4b5a0bb8e2

Co-authored-by: Rahul Butani <rrbutani@users.noreply.github.com>
Co-authored-by: Ian (Hee) Cha <heec@google.com>
…azelbuild#20645)

Without this there can be large gaps in the profile when using remote
execution that are hard to reason about.

Closes bazelbuild#20474.

Commit
bazelbuild@75fffbf

PiperOrigin-RevId: 590475989
Change-Id: Ic6e042c36f85e8098a468c73c62bd45cc367423e

Co-authored-by: Brentley Jones <github@brentleyjones.com>
Co-authored-by: Yun Peng <pcloudy@google.com>
…azelbuild#20650)

9% of samples in a profile of one of our builds were inside the
`fillInStackTrace()` method. Collecting the valid names into a hashset
avoids needing to construct errors every time an invalid digest function
name is passed into this function.

Tested with Bazel 6.4.0. Our codebase is not yet compatible with Bazel
7.

I have not investigated why this function was receiving so many invalid
names.

Before:

![Screenshot 2023-12-15 at 2 39 01
pm](https://github.com/bazelbuild/bazel/assets/18002432/be4bd311-ca73-46ec-a06d-93bb0ca9c6ba)

After:

![Screenshot 2023-12-15 at 2 43 10
pm](https://github.com/bazelbuild/bazel/assets/18002432/64b15739-538f-4752-aafd-6b2c94886595)

My understanding is that this will not speed up builds directly, but it
will allow BEP events to be processed more quickly.

Closes bazelbuild#20574.

Commit
bazelbuild@cb08d62

PiperOrigin-RevId: 592094151
Change-Id: Ie23241c9ec40e59ba2aac1fc83e4830340260f45

Co-authored-by: Christian Scott <christian.s@canva.com>
Co-authored-by: Ian (Hee) Cha <heec@google.com>
Co-authored-by: Yun Peng <pcloudy@google.com>
…azelbuild#20642)

`DeflaterInputStream`, `GZIPInputStream`, `GZIPOutputStream`, and
`InflaterInputStream`, all use an internal byte buffer of 512 bytes by
default.

Whenever the wrapped stream exceeds this size, a full copy to a new
buffer will occur, which will increase at increments of the same size.
For example, a stream of length 2K will be copied four times. Increasing
the size of the buffer we use can result in significant reductions in
CPU usage (read: copies).

Examples in the repository
--------------------------

There are already two places where we increase the default size of these
buffers:

-
`//src/main/java/com/google/devtools/build/lib/bazel/repository/TarGzFunction.java`
-
`//src/main/java/com/google/devtools/build/lib/bazel/repository/downloader/HttpStream.java`

Prior art
---------

There is an open enhancement issue in the OpenJDK tracker on this which
contains a benchmark for `InflaterOutputStream`:

> Increase the default, internal buffer size of the Streams in
`java.util.zip`
> https://bugs.openjdk.org/browse/JDK-8242864

A similar change was merged in for JDK15+ in 2020:

> Improve performance of `InflaterOutputStream.write()`
> https://bugs.openjdk.org/browse/JDK-8242848

Providing a simple benchmark
----------------------------

I'm inlining a simple `jmh` benchmark and the results underneath it for
one `GzipInputStream` case.

The benchmark:

```
@fork(1)
@threads(1)
@WarmUp(iterations = 2)
@State(Scope.Benchmark)
@OutputTimeUnit(TimeUnit.NANOSECONDS)
public class GZIPInputStreamBenchmark {
    @param({"1024", "3072", "9216"})
    long inputLength;
    @param({"512", "1024", "4096", "8192"})
    int bufferSize;
    private byte[] content;

    @setup(Level.Iteration)
    public void setup() throws IOException {
        var baos = new ByteArrayOutputStream();
        // No need to set the buffer size on this as it's a one-time cost for setup and not counted in the result.
        var gzip = new GZIPOutputStream(baos);

        var inputBytes = generateRandomByteArrayOfLength(inputLength);
        gzip.write(inputBytes);
        gzip.finish();

        this.content = baos.toByteArray();
    }

    @benchmark
    @BenchmarkMode(Mode.AverageTime)
    public void getGzipInputStream(Blackhole bh) throws IOException {
        try (var is = new ByteArrayInputStream(this.content);
             var gzip = new GZIPInputStream(is, bufferSize)) {
            bh.consume(gzip.readAllBytes());
        }
    }

    byte[] generateRandomByteArrayOfLength(long length) {
        var random = new Random();
        var intStream = random.ints(0, 5000).limit(length).boxed();

        return intStream.collect(
                ByteArrayOutputStream::new,
                (baos, i) -> baos.write(i.intValue()),
                (baos1, baos2) -> baos1.write(baos2.toByteArray(), 0, baos2.size())
        ).toByteArray();
    }
}
```

The results:

```
Benchmark                                    (bufferSize)  (inputLength)  Mode  Cnt      Score    Error  Units
GZIPInputStreamBenchmark.getGzipInputStream           512           1024  avgt    5   3207.217 ± 24.919  ns/op
GZIPInputStreamBenchmark.getGzipInputStream           512           3072  avgt    5   5874.191 ±  5.827  ns/op
GZIPInputStreamBenchmark.getGzipInputStream           512           9216  avgt    5  15567.345 ± 93.281  ns/op
GZIPInputStreamBenchmark.getGzipInputStream          1024           1024  avgt    5   2580.566 ± 14.566  ns/op
GZIPInputStreamBenchmark.getGzipInputStream          1024           3072  avgt    5   4154.582 ± 16.016  ns/op
GZIPInputStreamBenchmark.getGzipInputStream          1024           9216  avgt    5   9942.521 ± 61.215  ns/op
GZIPInputStreamBenchmark.getGzipInputStream          4096           1024  avgt    5   2150.255 ± 52.770  ns/op
GZIPInputStreamBenchmark.getGzipInputStream          4096           3072  avgt    5   2289.185 ± 71.396  ns/op
GZIPInputStreamBenchmark.getGzipInputStream          4096           9216  avgt    5   5656.891 ± 28.499  ns/op
GZIPInputStreamBenchmark.getGzipInputStream          8192           1024  avgt    5   2177.427 ± 30.896  ns/op
GZIPInputStreamBenchmark.getGzipInputStream          8192           3072  avgt    5   2517.390 ± 21.296  ns/op
GZIPInputStreamBenchmark.getGzipInputStream          8192           9216  avgt    5   5227.932 ± 55.525  ns/op
```

Co-authored-by: Kushal Pisavadia <kushal.p@apple.com>

Closes bazelbuild#20316.

Commit
bazelbuild@75a6693

PiperOrigin-RevId: 588444920
Change-Id: I1fb47f0b08dcb8d72f3e2c43534c33d60efb87f2

Co-authored-by: Ed Schouten <eschouten@apple.com>
…d#20646)

Bazel 7 upgrades its runtime JVM to 21. However, the implementation of
`ForkJoinPool` in JDK 21 is changed so that the number of concurrent
actions may be larger than `--jobs`.

Flag `--experimental_use_semaphore_for_jobs` was introduced to fix this
problem. This CL flips it to make `--jobs` work again by default.

Fixes bazelbuild#20521.

Commit
bazelbuild@3c298fd

PiperOrigin-RevId: 590834204
Change-Id: I0ddf1e1a088eb4d917036350533df94c17ac48e6

Co-authored-by: Googler <chiwang@google.com>
Co-authored-by: Yun Peng <pcloudy@google.com>
…azelbuild#20647)

After an action was executed remotely, RemoteSpawnRunner would use
the timestamps in the execution metadata to record appropriate timing
phases into the JSON profile.

However, there are durations in-between the existing phases that are
unaccounted for. Depending on the RBE server implemenation, these
phases could mean different things:
- Sandbox preparation
- Cleaning up sandbox environments post-execution
- Others

Missing these durations inside the timing profile would cause confusion
to end users as it would be interpreted as nothing happened in between
the existing phases.

Add these durations into the profile as "pre-X" phases so that user is
aware of activities could still be happening during that time. RBE
server implementation should be able to alter these label
programmatically if necessary.

Closes bazelbuild#20387.

Commit
bazelbuild@fe9e9e0

PiperOrigin-RevId: 590816782
Change-Id: I2bee36be928db24a14fab18bc519c3893723b7d6

Co-authored-by: Son Luong Ngoc <sluongng@gmail.com>
Co-authored-by: Ian (Hee) Cha <heec@google.com>
Co-authored-by: Yun Peng <pcloudy@google.com>
Closes bazelbuild#20394.

PiperOrigin-RevId: 586723311
Change-Id: I4a12af0d3a728686b6db3852f2e4740c888fd2d2

Co-authored-by: Justin Horvitz <jhorvitz@google.com>
Wyverald and others added 20 commits January 26, 2024 19:25
bzlmod: support git repos in source.json

Closes bazelbuild#20912.

RELNOTES: Added `init_submodules` attribute to `git_override`.
Registries now support the `git_repository` type in `source.json`.

Change-Id: Ia267131e6d081572a642888ba34e285805bbbe9f
PiperOrigin-RevId: 601268823

---------

Co-authored-by: Nachum Goldstein <nachum@google.com>
Co-authored-by: Keith Smiley <keithbsmiley@gmail.com>
`bazel mod dump_repo_mapping` with no arguments is explicitly made an
error so that a new mode that dumps all repository mappings with a
single Bazel invocation can be added later if needed, e.g. to support
IntelliJ's "sync" workflow.

RELNOTES: `bazel mod dump_repo_mapping <canonical repo name>...` returns
the repository mappings of the given repositories in NDJSON. This
information can be used by IDEs and Starlark language servers to resolve
labels with `--enable_bzlmod`.

Work towards bazelbuild#20631

Closes bazelbuild#20686.

Commit
bazelbuild@59ac9ce

PiperOrigin-RevId: 601332180
Change-Id: I828d7c88637bea175e11eccc52c6202f6da4c32c

Co-authored-by: Fabian Meumertzheim <fabian@meumertzhe.im>
…d#21082)

RELNOTES: The flag `--experimental_worker_for_repo_fetching` now
defaults to `auto`, which uses virtual threads from JDK 21 if it's
available. This eliminates restarts during repo fetching.
…21085)

These cause failures when relying on `depGraph` containing `ModuleKey`s
for all modules, which is the case in production.

Work towards bazelbuild#20997

Closes bazelbuild#21037.

Commit
bazelbuild@27419e3

PiperOrigin-RevId: 601822357
Change-Id: Ifcad9d7b73835491c1f1fca975e05834057a6825

Co-authored-by: Fabian Meumertzheim <fabian@meumertzhe.im>
… status. (bazelbuild#21084)

Previously, they were both displayed as `remote-cache`; there's now a
separate `disk-cache` form. If a combined cache is used, one or both
forms will appear, depending on which caches were looked up.

As a result, the progress status reporting is moved to the individual
cache implementations. While this is kind of unfortunate from an
architectural standpoint, it's likely the best we can do until we recast
cache lookups as spawn strategies (see bazelbuild#19904).

Closes bazelbuild#20935.

PiperOrigin-RevId: 601748051
Change-Id: I03710219973c95d4fca999d931b3513f6d240d94
…mote_scrubbing_config configuration format. (bazelbuild#21089)

PiperOrigin-RevId: 597764071
Change-Id: I9ea7000509e29c327034db5fd0087b35c0fcdfb9
…ld#21086)

This requires fixing an actual bug that causes the Windows Java launcher
to not find the Java runtime with `--nolegacy_external_runfiles`.

Work towards bazelbuild#12821

Closes bazelbuild#20680.

Commit
bazelbuild@3d4491c

PiperOrigin-RevId: 601826685
Change-Id: Ib45e60901d47efc06bf875c334edafcaeabc5f1e

Co-authored-by: Fabian Meumertzheim <fabian@meumertzhe.im>
…ryEntries(). (bazelbuild#21088)

Although the semantics weren't explicitly specified by the FileSystem
interface, the Unix implementation calls readdir(3), which always
dereferences a symlink for the directory itself.

Also deduplicate the common logic.

PiperOrigin-RevId: 598543913
Change-Id: I9bcab70c57ef4e8c4da58f4871397c6f2362f43c
…error. (bazelbuild#21090)

Unlike on Unix, we can't fall back to an alternative implementation.
Failure to emit an early error would cause later native method calls to
produce a less informative error message.

See bazelbuild#20677 for a recent
example where this would have been helpful.

Also note some minor changes:
* Since we attempt to load the JNI in two different ways, make sure to
propagate both exceptions (one suppressed by the other).
* The default switch case is unnecessary, as errorprone already enforces
exhaustiveness.

PiperOrigin-RevId: 595206761
Change-Id: I4b6258d59d9a4fcc9f83418375acc51a32d7f4a4
…o for getOutputStream. (bazelbuild#21087)

We're likely reporting permission errors as FileNotFoundException in
some cases, which makes debugging more difficult and might have
correctness implications.

Better yet would be to migrate to the standard
java.nio.file.FileSystemException types, but that's a big and scary
change, and it will have to wait for another day.

PiperOrigin-RevId: 601092545
Change-Id: I1b8777850c946c97e7faf2a47a60255f4e591a3a
…e bytes. (bazelbuild#20988)

When building without the bytes, outputs are only materialized when
either (1) an action prefetches them, or (2) they've been explicitly
requested. When both --remote_download_minimal and
--noexperimental_check_output_files are set, this presents a problem for
a build command followed by a run command for the same target: the build
command won't download the outputs and the run command won't check for
their presence, proceeding without them. A non-incremental run command
works, because the outputs are considered top-level even in minimal
mode.

This is only a problem for a run command because it exists outside of
action execution, and doesn't get a chance to prefetch inputs; it would
have been better to implement the run command by injecting a specially
crafted action into the build graph, but that will have to wait for
another day. The present fix might theoretically slow things down if
output checking is truly unnecessary, but I doubt that would cause a
significant regression in practice.

Fixes bazelbuild#20843.

Closes bazelbuild#20853.

Commit
bazelbuild@2f899ef

PiperOrigin-RevId: 597909909
Change-Id: I66aedef4994fbda41fe5378c80940fa8ba637bfd

Co-authored-by: Tiago Quelhas <tjgq@google.com>
Apple has since changed visionOS to only support arm macs

Closes bazelbuild#20335.

Commit
bazelbuild@ce9fa8e

PiperOrigin-RevId: 601022333
Change-Id: I6322c68102129eb230bbbb4546249cb22899fe94

Co-authored-by: Keith Smiley <keithbsmiley@gmail.com>
Unfortuantely, there isn't an easy way to add integration test for this
specific edge case on Windows.

Fixes bazelbuild#20741.

Closes bazelbuild#20752.

Commit
bazelbuild@958e0c4

PiperOrigin-RevId: 596547046
Change-Id: I4f517b161c03793329d5a4e21ec8ac4a5b53abb0

Co-authored-by: Chi Wawng <coeuvre@gmail.com>
…azelbuild#20990)

Bazel forces use of `lld` and `ld.gold` if found, but performed feature
detection on the default linker instead.

Fixes bazelbuild#20834

Closes bazelbuild#20833.

Commit
bazelbuild@2bfe045

PiperOrigin-RevId: 598565598
Change-Id: I4890f278c5cc33d4e6a6ebb10d796fb1c22f9ba6

---------

Co-authored-by: Fabian Meumertzheim <fabian@meumertzhe.im>
…ompression. (bazelbuild#21124)

As discussed in bazelbuild#18997, compression causes performance degradation for
small blobs. The flag defaults to zero (always compress) for backwards
compatibility, but the discussion suggests that 100 might be an
appropriate value to set it to.

Fixes bazelbuild#18997.

Commit
bazelbuild@69eb0ec

PiperOrigin-RevId: 602353295
Change-Id: Ie9ef8c10e221e9190d3c5cb5f267fcf35a63fd3b

Co-authored-by: Googler <tjgq@google.com>
Commit
bazelbuild@275000c

PiperOrigin-RevId: 602440856
Change-Id: I5c1a3c4ba3fa70d1ea928ad3d23f472da7af65d7
…ding delimited protos. (bazelbuild#21143)

It's not a reliable method to check for EOF: it returns how many bytes
are guaranteed to be read without blocking, and the base implementation
always returns 0.

Instead, leverage the fact that parseDelimitedFrom() returns null if the
stream is at EOF.

PiperOrigin-RevId: 602257598
Change-Id: I61e51774611196fb44745dc0aa2dc836b41fcd68
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

9 participants