{"payload":{"feedbackUrl":"https://github.com/orgs/community/discussions/53140","repo":{"id":143197952,"defaultBranch":"master","name":"bazel","ownerLogin":"EdSchouten","currentUserCanPush":false,"isFork":true,"isEmpty":false,"createdAt":"2018-08-01T19:14:01.000Z","ownerAvatar":"https://avatars.githubusercontent.com/u/736085?v=4","public":true,"private":false,"isOrgOwned":false},"refInfo":{"name":"","listCacheKey":"v0:1715689897.0","currentOid":""},"activityList":{"items":[{"before":"b8cf82979476b5958ecccb8f33784d2362778aa7","after":"d4e12365baebdbcbf132cf3ebe67fc8d3da17c4e","ref":"refs/heads/eschouten/20240513-supports-remote-execution","pushedAt":"2024-05-14T18:27:13.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"EdSchouten","name":"Ed Schouten","path":"/EdSchouten","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/736085?s=80&v=4"},"commit":{"message":"Add a test for platform(exec_properties={\"no-remote-exec\": \"true\"})\n\nApparently platform(exec_properties) propagates to\nctx.actions.run(execution_requirements). This means that it is possible\nto mark entire platforms as \"no-remote-exec\".\n\nLet's add some testing coverage for this, to make sure it continues to\nwork going forward.","shortMessageHtmlLink":"Add a test for platform(exec_properties={\"no-remote-exec\": \"true\"})"}},{"before":"4f1602a4742c179b333b67c76d30782d44d8bd81","after":"b8cf82979476b5958ecccb8f33784d2362778aa7","ref":"refs/heads/eschouten/20240513-supports-remote-execution","pushedAt":"2024-05-14T14:19:13.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"EdSchouten","name":"Ed Schouten","path":"/EdSchouten","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/736085?s=80&v=4"},"commit":{"message":"Add platform(supports_remote_execution=False)\n\nBazel currently assumes that every platform() that is declared can be\nused to perform remote execution. This may not be necessarily true. It\nmay only be possible to use this platform through local execution. In\nthat case the provided platform properties may only be used for remote\ncaching (if there is write access to the Action Cache).\n\nThis change extends platform() to permit annotating it to indicate that\nno workers are present. This is different from the solution that is\nproposed here, where additional command line flags are being proposed:\n\nhttps://github.com/bazelbuild/bazel/discussions/20497\n\nThat said, the goal of this change is to act as a workaround until the\nupstream proposal is implemented.","shortMessageHtmlLink":"Add platform(supports_remote_execution=False)"}},{"before":"6d6eaf9dc3b80cc7ac5e0ca6a51b1e5bac403ccf","after":"4f1602a4742c179b333b67c76d30782d44d8bd81","ref":"refs/heads/eschouten/20240513-supports-remote-execution","pushedAt":"2024-05-14T13:09:19.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"EdSchouten","name":"Ed Schouten","path":"/EdSchouten","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/736085?s=80&v=4"},"commit":{"message":"Add platform(supports_remote_execution=False)\n\nBazel currently assumes that every platform() that is declared can be\nused to perform remote execution. This may not be necessarily true. It\nmay only be possible to use this platform through local execution. In\nthat case the provided platform properties may only be used for remote\ncaching (if there is write access to the Action Cache).\n\nThis change extends platform() to permit annotating it to indicate that\nno workers are present. This is different from the solution that is\nproposed here, where additional command line flags are being proposed:\n\nhttps://github.com/bazelbuild/bazel/discussions/20497\n\nThat said, the goal of this change is to act as a workaround until the\nupstream proposal is implemented.","shortMessageHtmlLink":"Add platform(supports_remote_execution=False)"}},{"before":null,"after":"6d6eaf9dc3b80cc7ac5e0ca6a51b1e5bac403ccf","ref":"refs/heads/eschouten/20240513-supports-remote-execution","pushedAt":"2024-05-14T12:31:37.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"EdSchouten","name":"Ed Schouten","path":"/EdSchouten","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/736085?s=80&v=4"},"commit":{"message":"Add platform(supports_remote_execution=False)\n\nBazel currently assumes that every platform() that is declared can be\nused to perform remote execution. This may not be necessarily true. It\nmay only be possible to use this platform through local execution. In\nthat case the provided platform properties may only be used for remote\ncaching (if there is write access to the Action Cache).\n\nThis change extends platform() to permit annotating it to indicate that\nno workers are present. This is different from the solution that is\nproposed here, where additional command line flags are being proposed:\n\nhttps://github.com/bazelbuild/bazel/discussions/20497\n\nThat said, the goal of this change is to act as a workaround until the\nupstream proposal is implemented.","shortMessageHtmlLink":"Add platform(supports_remote_execution=False)"}},{"before":"2327bde9e95f8306c4bbd92c3e62618f4088217f","after":"258ca750789c7ccb71eb92e5d7365acd1e1ffc94","ref":"refs/heads/eschouten/20231127-zip","pushedAt":"2023-12-06T10:59:47.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"EdSchouten","name":"Ed Schouten","path":"/EdSchouten","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/736085?s=80&v=4"},"commit":{"message":"Use a larger buffer size for `java.util.zip.*Stream` classes\n\n`DeflaterInputStream`, `GZIPInputStream`, `GZIPOutputStream`,\nand `InflaterInputStream`, all use an internal byte buffer of 512\nbytes by default.\n\nWhenever the wrapped stream exceeds this size, a full copy to a new\nbuffer will occur, which will increase at increments of the same\nsize. For example, a stream of length 2K will be copied four\ntimes. Increasing the size of the buffer we use can result in\nsignificant reductions in CPU usage (read: copies).\n\nExamples in the repository\n--------------------------\n\nThere are already two places where we increase the default size of\nthese buffers:\n\n- `//src/main/java/com/google/devtools/build/lib/bazel/repository/TarGzFunction.java`\n- `//src/main/java/com/google/devtools/build/lib/bazel/repository/downloader/HttpStream.java`\n\nPrior art\n---------\n\nThere is an open enhancement issue in the OpenJDK tracker on this\nwhich contains a benchmark for `InflaterOutputStream`:\n\n> Increase the default, internal buffer size of the Streams in `java.util.zip`\n> https://bugs.openjdk.org/browse/JDK-8242864\n\nA similar change was merged in for JDK15+ in 2020:\n\n> Improve performance of `InflaterOutputStream.write()`\n> https://bugs.openjdk.org/browse/JDK-8242848\n\nProviding a simple benchmark\n----------------------------\n\nI'm inlining a simple `jmh` benchmark and the results underneath it\nfor one `GzipInputStream` case.\n\nThe benchmark:\n\n```\n@Fork(1)\n@Threads(1)\n@Warmup(iterations = 2)\n@State(Scope.Benchmark)\n@OutputTimeUnit(TimeUnit.NANOSECONDS)\npublic class GZIPInputStreamBenchmark {\n    @Param({\"1024\", \"3072\", \"9216\"})\n    long inputLength;\n    @Param({\"512\", \"1024\", \"4096\", \"8192\"})\n    int bufferSize;\n    private byte[] content;\n\n    @Setup(Level.Iteration)\n    public void setup() throws IOException {\n        var baos = new ByteArrayOutputStream();\n        // No need to set the buffer size on this as it's a one-time cost for setup and not counted in the result.\n        var gzip = new GZIPOutputStream(baos);\n\n        var inputBytes = generateRandomByteArrayOfLength(inputLength);\n        gzip.write(inputBytes);\n        gzip.finish();\n\n        this.content = baos.toByteArray();\n    }\n\n    @Benchmark\n    @BenchmarkMode(Mode.AverageTime)\n    public void getGzipInputStream(Blackhole bh) throws IOException {\n        try (var is = new ByteArrayInputStream(this.content);\n             var gzip = new GZIPInputStream(is, bufferSize)) {\n            bh.consume(gzip.readAllBytes());\n        }\n    }\n\n    byte[] generateRandomByteArrayOfLength(long length) {\n        var random = new Random();\n        var intStream = random.ints(0, 5000).limit(length).boxed();\n\n        return intStream.collect(\n                ByteArrayOutputStream::new,\n                (baos, i) -> baos.write(i.intValue()),\n                (baos1, baos2) -> baos1.write(baos2.toByteArray(), 0, baos2.size())\n        ).toByteArray();\n    }\n}\n```\n\nThe results:\n\n```\nBenchmark                                    (bufferSize)  (inputLength)  Mode  Cnt      Score    Error  Units\nGZIPInputStreamBenchmark.getGzipInputStream           512           1024  avgt    5   3207.217 ± 24.919  ns/op\nGZIPInputStreamBenchmark.getGzipInputStream           512           3072  avgt    5   5874.191 ±  5.827  ns/op\nGZIPInputStreamBenchmark.getGzipInputStream           512           9216  avgt    5  15567.345 ± 93.281  ns/op\nGZIPInputStreamBenchmark.getGzipInputStream          1024           1024  avgt    5   2580.566 ± 14.566  ns/op\nGZIPInputStreamBenchmark.getGzipInputStream          1024           3072  avgt    5   4154.582 ± 16.016  ns/op\nGZIPInputStreamBenchmark.getGzipInputStream          1024           9216  avgt    5   9942.521 ± 61.215  ns/op\nGZIPInputStreamBenchmark.getGzipInputStream          4096           1024  avgt    5   2150.255 ± 52.770  ns/op\nGZIPInputStreamBenchmark.getGzipInputStream          4096           3072  avgt    5   2289.185 ± 71.396  ns/op\nGZIPInputStreamBenchmark.getGzipInputStream          4096           9216  avgt    5   5656.891 ± 28.499  ns/op\nGZIPInputStreamBenchmark.getGzipInputStream          8192           1024  avgt    5   2177.427 ± 30.896  ns/op\nGZIPInputStreamBenchmark.getGzipInputStream          8192           3072  avgt    5   2517.390 ± 21.296  ns/op\nGZIPInputStreamBenchmark.getGzipInputStream          8192           9216  avgt    5   5227.932 ± 55.525  ns/op\n```\n\nCo-authored-by: Kushal Pisavadia <kushal.p@apple.com>","shortMessageHtmlLink":"Use a larger buffer size for <code>java.util.zip.*Stream</code> classes"}},{"before":"121935fbb69eafa805c24c8ac7e023de0a656d8c","after":"2327bde9e95f8306c4bbd92c3e62618f4088217f","ref":"refs/heads/eschouten/20231127-zip","pushedAt":"2023-11-27T10:10:42.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"EdSchouten","name":"Ed Schouten","path":"/EdSchouten","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/736085?s=80&v=4"},"commit":{"message":"Use a larger buffer size for `java.util.zip.*Stream` classes\n\n`DeflaterInputStream`, `GZIPInputStream`, `GZIPOutputStream`,\nand `InflaterInputStream`, all use an internal byte buffer of 512\nbytes by default.\n\nWhenever the wrapped stream exceeds this size, a full copy to a new\nbuffer will occur, which will increase at increments of the same\nsize. For example, a stream of length 2K will be copied four\ntimes. Increasing the size of the buffer we use can result in\nsignificant reductions in CPU usage (read: copies).\n\nExamples in the repository\n--------------------------\n\nThere are already two places where we increase the default size of\nthese buffers:\n\n- `//src/main/java/com/google/devtools/build/lib/bazel/repository/TarGzFunction.java`\n- `//src/main/java/com/google/devtools/build/lib/bazel/repository/downloader/HttpStream.java`\n\nPrior art\n---------\n\nThere is an open enhancement issue in the OpenJDK tracker on this\nwhich contains a benchmark for `InflaterOutputStream`:\n\n> Increase the default, internal buffer size of the Streams in `java.util.zip`\n> https://bugs.openjdk.org/browse/JDK-8242864\n\nA similar change was merged in for JDK15+ in 2020:\n\n> Improve performance of `InflaterOutputStream.write()`\n> https://bugs.openjdk.org/browse/JDK-8242848\n\nProviding a simple benchmark\n----------------------------\n\nI'm inlining a simple `jmh` benchmark and the results underneath it\nfor one `GzipInputStream` case.\n\nThe benchmark:\n\n```\n@Fork(1)\n@Threads(1)\n@Warmup(iterations = 2)\n@State(Scope.Benchmark)\n@OutputTimeUnit(TimeUnit.NANOSECONDS)\npublic class GZIPInputStreamBenchmark {\n    @Param({\"1024\", \"3072\", \"9216\"})\n    long inputLength;\n    @Param({\"512\", \"1024\", \"4096\", \"8192\"})\n    int bufferSize;\n    private byte[] content;\n\n    @Setup(Level.Iteration)\n    public void setup() throws IOException {\n        var baos = new ByteArrayOutputStream();\n        // No need to set the buffer size on this as it's a one-time cost for setup and not counted in the result.\n        var gzip = new GZIPOutputStream(baos);\n\n        var inputBytes = generateRandomByteArrayOfLength(inputLength);\n        gzip.write(inputBytes);\n        gzip.finish();\n\n        this.content = baos.toByteArray();\n    }\n\n    @Benchmark\n    @BenchmarkMode(Mode.AverageTime)\n    public void getGzipInputStream(Blackhole bh) throws IOException {\n        try (var is = new ByteArrayInputStream(this.content);\n             var gzip = new GZIPInputStream(is, bufferSize)) {\n            bh.consume(gzip.readAllBytes());\n        }\n    }\n\n    byte[] generateRandomByteArrayOfLength(long length) {\n        var random = new Random();\n        var intStream = random.ints(0, 5000).limit(length).boxed();\n\n        return intStream.collect(\n                ByteArrayOutputStream::new,\n                (baos, i) -> baos.write(i.intValue()),\n                (baos1, baos2) -> baos1.write(baos2.toByteArray(), 0, baos2.size())\n        ).toByteArray();\n    }\n}\n```\n\nThe results:\n\n```\nBenchmark                                    (bufferSize)  (inputLength)  Mode  Cnt      Score    Error  Units\nGZIPInputStreamBenchmark.getGzipInputStream           512           1024  avgt    5   3207.217 ± 24.919  ns/op\nGZIPInputStreamBenchmark.getGzipInputStream           512           3072  avgt    5   5874.191 ±  5.827  ns/op\nGZIPInputStreamBenchmark.getGzipInputStream           512           9216  avgt    5  15567.345 ± 93.281  ns/op\nGZIPInputStreamBenchmark.getGzipInputStream          1024           1024  avgt    5   2580.566 ± 14.566  ns/op\nGZIPInputStreamBenchmark.getGzipInputStream          1024           3072  avgt    5   4154.582 ± 16.016  ns/op\nGZIPInputStreamBenchmark.getGzipInputStream          1024           9216  avgt    5   9942.521 ± 61.215  ns/op\nGZIPInputStreamBenchmark.getGzipInputStream          4096           1024  avgt    5   2150.255 ± 52.770  ns/op\nGZIPInputStreamBenchmark.getGzipInputStream          4096           3072  avgt    5   2289.185 ± 71.396  ns/op\nGZIPInputStreamBenchmark.getGzipInputStream          4096           9216  avgt    5   5656.891 ± 28.499  ns/op\nGZIPInputStreamBenchmark.getGzipInputStream          8192           1024  avgt    5   2177.427 ± 30.896  ns/op\nGZIPInputStreamBenchmark.getGzipInputStream          8192           3072  avgt    5   2517.390 ± 21.296  ns/op\nGZIPInputStreamBenchmark.getGzipInputStream          8192           9216  avgt    5   5227.932 ± 55.525  ns/op\n```\n\nPatch by: Kushal Pisavadia <kushal.p@apple.com>","shortMessageHtmlLink":"Use a larger buffer size for <code>java.util.zip.*Stream</code> classes"}},{"before":null,"after":"121935fbb69eafa805c24c8ac7e023de0a656d8c","ref":"refs/heads/eschouten/20231127-zip","pushedAt":"2023-11-27T10:08:05.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"EdSchouten","name":"Ed Schouten","path":"/EdSchouten","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/736085?s=80&v=4"},"commit":{"message":"Use a larger buffer size for `java.util.zip.*Stream` classes\n\n`DeflaterInputStream`, `GZIPInputStream`, `GZIPOutputStream`,\nand `InflaterInputStream`, all use an internal byte buffer of 512\nbytes by default.\n\nWhenever the wrapped stream exceeds this size, a full copy to a new\nbuffer will occur, which will increase at increments of the same\nsize. For example, a stream of length 2K will be copied four\ntimes. Increasing the size of the buffer we use can result in\nsignificant reductions in CPU usage (read: copies).\n\nExamples in the repository\n--------------------------\n\nThere are already two places where we increase the default size of\nthese buffers:\n\n- `//src/main/java/com/google/devtools/build/lib/bazel/repository/TarGzFunction.java`\n- `//src/main/java/com/google/devtools/build/lib/bazel/repository/downloader/HttpStream.java`\n\nPrior art\n---------\n\nThere is an open enhancement issue in the OpenJDK tracker on this\nwhich contains a benchmark for `InflaterOutputStream`:\n\n> Increase the default, internal buffer size of the Streams in `java.util.zip`\n> https://bugs.openjdk.org/browse/JDK-8242864\n\nA similar change was merged in for JDK15+ in 2020:\n\n> Improve performance of `InflaterOutputStream.write()`\n> https://bugs.openjdk.org/browse/JDK-8242848\n\nProviding a simple benchmark\n----------------------------\n\nI'm inlining a simple `jmh` benchmark and the results underneath it\nfor one `GzipInputStream` case.\n\nThe benchmark:\n\n```\n@Fork(1)\n@Threads(1)\n@Warmup(iterations = 2)\n@State(Scope.Benchmark)\n@OutputTimeUnit(TimeUnit.NANOSECONDS)\npublic class GZIPInputStreamBenchmark {\n    @Param({\"1024\", \"3072\", \"9216\"})\n    long inputLength;\n    @Param({\"512\", \"1024\", \"4096\", \"8192\"})\n    int bufferSize;\n    private byte[] content;\n\n    @Setup(Level.Iteration)\n    public void setup() throws IOException {\n        var baos = new ByteArrayOutputStream();\n        // No need to set the buffer size on this as it's a one-time cost for setup and not counted in the result.\n        var gzip = new GZIPOutputStream(baos);\n\n        var inputBytes = generateRandomByteArrayOfLength(inputLength);\n        gzip.write(inputBytes);\n        gzip.finish();\n\n        this.content = baos.toByteArray();\n    }\n\n    @Benchmark\n    @BenchmarkMode(Mode.AverageTime)\n    public void getGzipInputStream(Blackhole bh) throws IOException {\n        try (var is = new ByteArrayInputStream(this.content);\n             var gzip = new GZIPInputStream(is, bufferSize)) {\n            bh.consume(gzip.readAllBytes());\n        }\n    }\n\n    byte[] generateRandomByteArrayOfLength(long length) {\n        var random = new Random();\n        var intStream = random.ints(0, 5000).limit(length).boxed();\n\n        return intStream.collect(\n                ByteArrayOutputStream::new,\n                (baos, i) -> baos.write(i.intValue()),\n                (baos1, baos2) -> baos1.write(baos2.toByteArray(), 0, baos2.size())\n        ).toByteArray();\n    }\n}\n```\n\nThe results:\n\n```\nBenchmark                                    (bufferSize)  (inputLength)  Mode  Cnt      Score    Error  Units\nGZIPInputStreamBenchmark.getGzipInputStream           512           1024  avgt    5   3207.217 ± 24.919  ns/op\nGZIPInputStreamBenchmark.getGzipInputStream           512           3072  avgt    5   5874.191 ±  5.827  ns/op\nGZIPInputStreamBenchmark.getGzipInputStream           512           9216  avgt    5  15567.345 ± 93.281  ns/op\nGZIPInputStreamBenchmark.getGzipInputStream          1024           1024  avgt    5   2580.566 ± 14.566  ns/op\nGZIPInputStreamBenchmark.getGzipInputStream          1024           3072  avgt    5   4154.582 ± 16.016  ns/op\nGZIPInputStreamBenchmark.getGzipInputStream          1024           9216  avgt    5   9942.521 ± 61.215  ns/op\nGZIPInputStreamBenchmark.getGzipInputStream          4096           1024  avgt    5   2150.255 ± 52.770  ns/op\nGZIPInputStreamBenchmark.getGzipInputStream          4096           3072  avgt    5   2289.185 ± 71.396  ns/op\nGZIPInputStreamBenchmark.getGzipInputStream          4096           9216  avgt    5   5656.891 ± 28.499  ns/op\nGZIPInputStreamBenchmark.getGzipInputStream          8192           1024  avgt    5   2177.427 ± 30.896  ns/op\nGZIPInputStreamBenchmark.getGzipInputStream          8192           3072  avgt    5   2517.390 ± 21.296  ns/op\nGZIPInputStreamBenchmark.getGzipInputStream          8192           9216  avgt    5   5227.932 ± 55.525  ns/op\n```\n\nCo-authored-by: Kushal Pisavadia <kushal.p@apple.com>","shortMessageHtmlLink":"Use a larger buffer size for <code>java.util.zip.*Stream</code> classes"}},{"before":"be44525396bdab5b7ed544180dfbbcd9b2a5da45","after":null,"ref":"refs/heads/eschouten/20230831-remote-minimal-symlink","pushedAt":"2023-09-04T11:58:42.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"EdSchouten","name":"Ed Schouten","path":"/EdSchouten","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/736085?s=80&v=4"}},{"before":"5e3f6ae4754cef8dcced1533b9ada131beff3e7f","after":"b9bfcf07152eb78e323c619b19a85a621bc9b494","ref":"refs/heads/eschouten/20210115-remote-output-service-integration","pushedAt":"2023-09-01T12:23:53.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"EdSchouten","name":"Ed Schouten","path":"/EdSchouten","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/736085?s=80&v=4"},"commit":{"message":"Remote Output Service: place bazel-out/ on a FUSE file system\n\nBuilds that yield large output files can generate lots of network\nbandwidth when remote execution is used. To combat this, we have flags\nsuch as --remote_download_minimal to disable downloading of output\nfiles. Unfortunately, this makes it hard to perform ad hoc exploration\nof build outputs.\n\nIn an attempt to solve this, this change adds an option to Bazel named\n--remote_output_service. When enabled, Bazel effectively gives up the\nresponsibility of managing a bazel-out/ directory. Instead, it calls\ninto a gRPC service to request a directory and creates a symlink that\npoints to it.\n\nSmart implementations of this gRPC service may use things like FUSE to\nlet this replacement bazel-out/ directory be lazy-loading, thereby\nreducing network bandwidth significantly. In order to create\nlazy-loading files and directories, Bazel can call into a BatchCreate()\nRPC that takes a list of Output{File,Directory,Symlink} messages,\nsimilar to REv2's ActionResult. This call is also used to create\nrunfiles directories by providing fictive instances of OutputSymlink.\n\nTo prevent Bazel from reading the contents of files stored in the FUSE\nfile system (which would cause network I/O), the protocol offers a\nBatchStat() call that can return information such as the REv2 digest.\nThough this is redundant with --unix_digest_hash_attribute_name, there\nare a couple of reasons why I added this feature:\n\n1. For non-Linux operating systems, it may make more sense to use NFSv4\n   instead of FUSE (i.e., running a virtual NFS daemon on localhost).\n   Even though RFC 8276 adds support for extended attributes to NFSv4,\n   not all operating systems implement it.\n\n2. It addresses the security/hermeticity concerns that people raised\n   when this feature was added. There is no way to add extended\n   attributes to files that can't be tampered with (as a non-root user),\n   while using gRPC solves that problem inherently.\n\n3. Callers of Bazel's BatchStat.batchStat() may generate many system\n   calls successively. This causes a large number of context switches\n   between Bazel and the FUSE daemon. Using gRPC seems to be cheaper.\n\nBy requiring that the output path returned by the gRPC service is\nwritable, no-remote actions can still run as before, both with\nsandboxing enabled and disabled. The only difference is that it will use\nspace on the gRPC service side, as opposed to the user's home directory\n(though the gRPC service may continue to store data in the user's home\ndirectory.\n\nI have a server implementation is written in Go on top of Buildbarn's\nstorage and file system layer. My plan is to release the code for this\nservice as soon as I've got a 'thumbs up' on the overall approach.","shortMessageHtmlLink":"Remote Output Service: place bazel-out/ on a FUSE file system"}},{"before":"3925745bde535b97e2c53fe03c876314907fe1f0","after":"5e3f6ae4754cef8dcced1533b9ada131beff3e7f","ref":"refs/heads/eschouten/20210115-remote-output-service-integration","pushedAt":"2023-09-01T12:14:28.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"EdSchouten","name":"Ed Schouten","path":"/EdSchouten","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/736085?s=80&v=4"},"commit":{"message":"Remote Output Service: place bazel-out/ on a FUSE file system\n\nBuilds that yield large output files can generate lots of network\nbandwidth when remote execution is used. To combat this, we have flags\nsuch as --remote_download_minimal to disable downloading of output\nfiles. Unfortunately, this makes it hard to perform ad hoc exploration\nof build outputs.\n\nIn an attempt to solve this, this change adds an option to Bazel named\n--remote_output_service. When enabled, Bazel effectively gives up the\nresponsibility of managing a bazel-out/ directory. Instead, it calls\ninto a gRPC service to request a directory and creates a symlink that\npoints to it.\n\nSmart implementations of this gRPC service may use things like FUSE to\nlet this replacement bazel-out/ directory be lazy-loading, thereby\nreducing network bandwidth significantly. In order to create\nlazy-loading files and directories, Bazel can call into a BatchCreate()\nRPC that takes a list of Output{File,Directory,Symlink} messages,\nsimilar to REv2's ActionResult. This call is also used to create\nrunfiles directories by providing fictive instances of OutputSymlink.\n\nTo prevent Bazel from reading the contents of files stored in the FUSE\nfile system (which would cause network I/O), the protocol offers a\nBatchStat() call that can return information such as the REv2 digest.\nThough this is redundant with --unix_digest_hash_attribute_name, there\nare a couple of reasons why I added this feature:\n\n1. For non-Linux operating systems, it may make more sense to use NFSv4\n   instead of FUSE (i.e., running a virtual NFS daemon on localhost).\n   Even though RFC 8276 adds support for extended attributes to NFSv4,\n   not all operating systems implement it.\n\n2. It addresses the security/hermeticity concerns that people raised\n   when this feature was added. There is no way to add extended\n   attributes to files that can't be tampered with (as a non-root user),\n   while using gRPC solves that problem inherently.\n\n3. Callers of Bazel's BatchStat.batchStat() may generate many system\n   calls successively. This causes a large number of context switches\n   between Bazel and the FUSE daemon. Using gRPC seems to be cheaper.\n\nBy requiring that the output path returned by the gRPC service is\nwritable, no-remote actions can still run as before, both with\nsandboxing enabled and disabled. The only difference is that it will use\nspace on the gRPC service side, as opposed to the user's home directory\n(though the gRPC service may continue to store data in the user's home\ndirectory.\n\nI have a server implementation is written in Go on top of Buildbarn's\nstorage and file system layer. My plan is to release the code for this\nservice as soon as I've got a 'thumbs up' on the overall approach.","shortMessageHtmlLink":"Remote Output Service: place bazel-out/ on a FUSE file system"}},{"before":"746eb9dd58d609482dc452ed4e0c2e8e6aa5bc94","after":"be44525396bdab5b7ed544180dfbbcd9b2a5da45","ref":"refs/heads/eschouten/20230831-remote-minimal-symlink","pushedAt":"2023-09-01T11:21:38.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"EdSchouten","name":"Ed Schouten","path":"/EdSchouten","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/736085?s=80&v=4"},"commit":{"message":"Let RemoteOutputService ensure bazel-out/ is not a symlink\n\nWhen doing a regular build without passing in --remote_download_*,\nExecutionTool will automatically clean up any traces of an OutputService\nthat replaces bazel-out/ with a symlink pointing to a remote file\nsystem.\n\nThe --remote_download_* option is implemented by installing an\nOutputService of its own, meaning that the regular logic in\nExecutionTool is skipped. The rationale being that an OutputService\nessentially takes ownership of bazel-out/.\n\nThis change extends RemoteOutputService to be a more complete\nimplementation of OutputService that ensures that bazel-out/ is set up\nproperly. This improves interoperability with changes such as the gRPC\nbased Remote Output Service protocol (see #12823).","shortMessageHtmlLink":"Let RemoteOutputService ensure bazel-out/ is not a symlink"}},{"before":"07fa54e3642b364b0fc9445f157c1ad8cd0abd9c","after":"746eb9dd58d609482dc452ed4e0c2e8e6aa5bc94","ref":"refs/heads/eschouten/20230831-remote-minimal-symlink","pushedAt":"2023-09-01T10:58:46.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"EdSchouten","name":"Ed Schouten","path":"/EdSchouten","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/736085?s=80&v=4"},"commit":{"message":"Let RemoteOutputService ensure bazel-out/ is not a symlink\n\nWhen doing a regular build without passing in --remote_download_*,\nExecutionTool will automatically clean up any traces of an OutputService\nthat replaces bazel-out/ with a symlink pointing to a remote file\nsystem.\n\nThe --remote_download_* option is implemented by installing an\nOutputService of its own, meaning that the regular logic in\nExecutionTool is skipped. The rationale being that an OutputService\nessentially takes ownership of bazel-out/.\n\nThis change extends RemoteOutputService to be a more complete\nimplementation of OutputService that ensures that bazel-out/ is set up\nproperly. This improves interoperability with changes such as the gRPC\nbased Remote Output Service protocol (see #12823).\n\nChange-Id: I5bedea2895785f3b8836fe8c5f8ae97ca2689233","shortMessageHtmlLink":"Let RemoteOutputService ensure bazel-out/ is not a symlink"}},{"before":"4574a4bcc69a4e0afad160e5cda5ea52255526d3","after":"07fa54e3642b364b0fc9445f157c1ad8cd0abd9c","ref":"refs/heads/eschouten/20230831-remote-minimal-symlink","pushedAt":"2023-08-31T16:10:53.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"EdSchouten","name":"Ed Schouten","path":"/EdSchouten","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/736085?s=80&v=4"},"commit":{"message":"Let RemoteOutputService ensure bazel-out/ is not a symlink\n\nWhen doing a regular build without passing in --remote_download_*,\nExecutionTool will automatically clean up any traces of an OutputService\nthat replaces bazel-out/ with a symlink pointing to a remote file\nsystem.\n\nThe --remote_download_* option is implemented by installing an\nOutputService of its own, meaning that the regular logic in\nExecutionTool is skipped. The rationale being that an OutputService\nessentially takes ownership of bazel-out/.\n\nThis change extends RemoteOutputService to be a more complete\nimplementation of OutputService that ensures that bazel-out/ is set up\nproperly. This improves interoperability with changes such as the gRPC\nbased Remote Output Service protocol (see #12823).","shortMessageHtmlLink":"Let RemoteOutputService ensure bazel-out/ is not a symlink"}},{"before":null,"after":"3bc2ef29e48b5a9f096d102ed7589dfd32f4e1c7","ref":"refs/heads/eschouten/20230831-create-or-set-writable","pushedAt":"2023-08-31T15:45:51.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"EdSchouten","name":"Ed Schouten","path":"/EdSchouten","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/736085?s=80&v=4"},"commit":{"message":"Ensure DirectoryContext.createOrSetWritable() always creates directories\n\nIn case DirectoryContext.createOrSetWritable() is called on a directory\ninside of a Tree, it already does a nice forward sweep along the path to\ncreate missing intermediate directories. When creating a bare output\nfile in a nested directory, it does not. It implicitly assumes that the\noutput directory was already created at a previous point in time.\n\nThis doesn't cause issues whenever we do a fully clean build, as some\nearlier phase (the analysis phase?) creates the missing parent\ndirectories for us. However, if we remove the nested directory structure\nfrom bazel-bin/ manually and rerun the test, this causes failures along\nthe lines:\n\n    java.io.FileNotFoundException: /REDACTED/execroot/main/bazel-out/darwin_x86_64-fastbuild/bin/some/very/very/very/deeply/nested (No such file or directory)\n\nSolve this issue by properly catching FileNotFoundExceptions that are\ngenerated when we call Path.isWritable() on the parent directory of the\noutput file. In that case we resort to calling\nPath.createDirectoryAndParents().\n\nI think this is likely the most efficient strategy, as it means we don't\nneed to do a full forward scan of the path in the common case where the\nparent directory does exist.\n\nThis issue can be noticed in practice when switching from\n--remote_output_service (PR #12823) to --remote_download_minimal.","shortMessageHtmlLink":"Ensure DirectoryContext.createOrSetWritable() always creates directories"}},{"before":"cdadd2ca04d4479452d911fbfa27b102b5f157df","after":"4574a4bcc69a4e0afad160e5cda5ea52255526d3","ref":"refs/heads/eschouten/20230831-remote-minimal-symlink","pushedAt":"2023-08-31T12:36:23.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"EdSchouten","name":"Ed Schouten","path":"/EdSchouten","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/736085?s=80&v=4"},"commit":{"message":"Let RemoteOutputService ensure bazel-out/ is not a symlink\n\nWhen doing a regular build without passing in --remote_download_*,\nExecutionTool will automatically clean up any traces of an OutputService\nthat replaces bazel-out/ with a symlink pointing to a remote file\nsystem.\n\nThe --remote_download_* option is implemented by installing an\nOutputService of its own, meaning that the regular logic in\nExecutionTool is skipped. The rationale being that an OutputService\nessentially takes ownership of bazel-out/.\n\nThis change extends RemoteOutputService to be a more complete\nimplementation of OutputService that ensures that bazel-out/ is set up\nproperly. This improves interoperability with changes such as the gRPC\nbased Remote Output Service protocol (see #12823).","shortMessageHtmlLink":"Let RemoteOutputService ensure bazel-out/ is not a symlink"}},{"before":null,"after":"cdadd2ca04d4479452d911fbfa27b102b5f157df","ref":"refs/heads/eschouten/20230831-remote-minimal-symlink","pushedAt":"2023-08-31T12:27:08.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"EdSchouten","name":"Ed Schouten","path":"/EdSchouten","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/736085?s=80&v=4"},"commit":{"message":"Let RemoteOutputService ensure bazel-out/ is not a symlink\n\nWhen doing a regular build without passing in --remote_download_*,\nExecutionTool will automatically clean up any traces of an OutputService\nthat replaces bazel-out/ with a symlink pointing to a remote file\nsystem.\n\nThe --remote_download_* option is implemented by installing an\nOutputService of its own, meaning that the regular logic in\nExecutionTool is skipped. The rationale being that an OutputService\nessentially takes ownership of bazel-out/.\n\nThis change extends RemoteOutputService to be a more complete\nimplementation of OutputService that ensures that bazel-out/ is set up\nproperly. This improves interoperability with changes such as the gRPC\nbased Remote Output Service protocol (see #12823).","shortMessageHtmlLink":"Let RemoteOutputService ensure bazel-out/ is not a symlink"}},{"before":"ae6a90a143b1ef25ff5fd620c662ab5f81d0043d","after":"cdadd2ca04d4479452d911fbfa27b102b5f157df","ref":"refs/heads/master","pushedAt":"2023-08-31T12:26:39.000Z","pushType":"push","commitsCount":3214,"pusher":{"login":"EdSchouten","name":"Ed Schouten","path":"/EdSchouten","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/736085?s=80&v=4"},"commit":{"message":"Let RemoteOutputService ensure bazel-out/ is not a symlink\n\nWhen doing a regular build without passing in --remote_download_*,\nExecutionTool will automatically clean up any traces of an OutputService\nthat replaces bazel-out/ with a symlink pointing to a remote file\nsystem.\n\nThe --remote_download_* option is implemented by installing an\nOutputService of its own, meaning that the regular logic in\nExecutionTool is skipped. The rationale being that an OutputService\nessentially takes ownership of bazel-out/.\n\nThis change extends RemoteOutputService to be a more complete\nimplementation of OutputService that ensures that bazel-out/ is set up\nproperly. This improves interoperability with changes such as the gRPC\nbased Remote Output Service protocol (see #12823).","shortMessageHtmlLink":"Let RemoteOutputService ensure bazel-out/ is not a symlink"}},{"before":"d41261378ad1e86bb7d409e1f511d429c41935a7","after":"ef88553906c67c8e2a87c2e153a9971470a4d18d","ref":"refs/heads/eschouten/20230814-sandbox-tempfile-2","pushedAt":"2023-08-31T07:32:19.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"EdSchouten","name":"Ed Schouten","path":"/EdSchouten","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/736085?s=80&v=4"},"commit":{"message":"atomicallyWriteTo: remove temporary file prior to creating it\n\nThis change is the same in spirit to #19241. When using Bazel in\ncombination with bb_clientd, bb_clientd may restore a copy of bazel-out/\nfrom a snapshot. Files restored from the snapshot are not mutable, as\nthey may be backed by other snapshots, a remote CAS, etc. etc. etc..\n\nThis change extends atomicallyWriteTo() to always write contents into a\nnew file that is guaranteed to be writable. The logic that's added here\nis merely copy-pasted from what's already present at the bottom of the\nsame function.","shortMessageHtmlLink":"atomicallyWriteTo: remove temporary file prior to creating it"}},{"before":"b71883072152f43a3df85b1ff03d1537332b729d","after":"90d56d59039360fe077d9b5e7742a5f8f982e62d","ref":"refs/heads/eschouten/20230814-runfiles-tempfile","pushedAt":"2023-08-31T07:27:14.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"EdSchouten","name":"Ed Schouten","path":"/EdSchouten","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/736085?s=80&v=4"},"commit":{"message":"build-runfiles: remove temporary file prior to creating it\n\nWhen building with Remote Output Service, bb_clientd has the ability to\nrestore snapshots of previous bazel-out/ directories. Though the file\ncontents of these snapshots are identical to what's created in the past,\nthe files will be read-only. This is because the files may be shared by\nmultiple snapshots.\n\nWe have noticed that most of Bazel is fine with that. Most of the times\nBazel is a good citizen, where it removes any files before recreating\nthem. We did notice a very rare case where build-runfiles tries to make\nin-place modifications to a temporary file that it maintains. This\nchange ensures that build-runfiles stops doing this.","shortMessageHtmlLink":"build-runfiles: remove temporary file prior to creating it"}},{"before":null,"after":"d41261378ad1e86bb7d409e1f511d429c41935a7","ref":"refs/heads/eschouten/20230814-sandbox-tempfile-2","pushedAt":"2023-08-14T15:34:55.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"EdSchouten","name":"Ed Schouten","path":"/EdSchouten","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/736085?s=80&v=4"},"commit":{"message":"atomicallyWriteTo: remove temporary file prior to creating it\n\nThis change is the same in spirit to #19241. When using Bazel in\ncombination with bb_clientd, bb_clientd may restore a copy of bazel-out/\nfrom a snapshot. Files restored from the snapshot are not mutable, as\nthey may be backed by other snapshots, a remote CAS, etc. etc. etc..\n\nThis change extends atomicallyWriteTo() to always write contents into a\nnew file that is guaranteed to be writable. The logic that's added here\nis merely copy-pasted from what's already present at the bottom of the\nsame function.","shortMessageHtmlLink":"atomicallyWriteTo: remove temporary file prior to creating it"}},{"before":"4990fca1e1afc69544b13153edce66dc6d65e57f","after":"b71883072152f43a3df85b1ff03d1537332b729d","ref":"refs/heads/eschouten/20230814-runfiles-tempfile","pushedAt":"2023-08-14T14:53:08.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"EdSchouten","name":"Ed Schouten","path":"/EdSchouten","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/736085?s=80&v=4"},"commit":{"message":"build-runfiles: remove temporary file prior to creating it\n\nWhen building with Remote Output Service, bb_clientd has the ability to\nrestore snapshots of previous bazel-out/ directories. Though the file\ncontents of these snapshots are identical to what's created in the past,\nthe files will be read-only. This is because the files may be shared by\nmultiple snapshots.\n\nWe have noticed that most of Bazel is fine with that. Most of the times\nBazel is a good citizen, where it removes any files before recreating\nthem. We did notice a very rare case where build-runfiles tries to make\nin-place modifications to a temporary file that it maintains. This\nchange ensures that build-runfiles stops doing this.","shortMessageHtmlLink":"build-runfiles: remove temporary file prior to creating it"}},{"before":null,"after":"4990fca1e1afc69544b13153edce66dc6d65e57f","ref":"refs/heads/eschouten/20230814-runfiles-tempfile","pushedAt":"2023-08-14T08:50:21.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"EdSchouten","name":"Ed Schouten","path":"/EdSchouten","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/736085?s=80&v=4"},"commit":{"message":"build-runfiles: remove temporary file prior to creating it\n\nWhen building with Remote Output Service, bb_clientd has the ability to\nrestore snapshots of previous bazel-out/ directories. Though the file\ncontents of these snapshots are identical to what's created in the past,\nthe files will be read-only. This is because the files may be shared by\nmultiple snapshots.\n\nWe have noticed that most of Bazel is fine with that. Most of the times\nBazel is a good citizen, where it removes any files before recreating\nthem. We did notice a very rare case where build-runfiles tries to make\nin-place modifications to a temporary file that it maintains. This\nchange ensures that build-runfiles stops doing this.","shortMessageHtmlLink":"build-runfiles: remove temporary file prior to creating it"}},{"before":"4af16e4235cae8697b61fe5837335dc1f700e443","after":"7f0d6c02c699c3dc57193ceea27636503207c006","ref":"refs/heads/eschouten/20230701-signal","pushedAt":"2023-07-03T17:11:13.000Z","pushType":"push","commitsCount":6,"pusher":{"login":"EdSchouten","name":"Ed Schouten","path":"/EdSchouten","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/736085?s=80&v=4"},"commit":{"message":"Merge branch 'master' into eschouten/20230701-signal","shortMessageHtmlLink":"Merge branch 'master' into eschouten/20230701-signal"}},{"before":null,"after":"4af16e4235cae8697b61fe5837335dc1f700e443","ref":"refs/heads/eschouten/20230701-signal","pushedAt":"2023-07-01T11:57:22.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"EdSchouten","name":"Ed Schouten","path":"/EdSchouten","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/736085?s=80&v=4"},"commit":{"message":"test-setup.sh: Attempt to raise the original signal once more\n\nOn POSIX-like systems, processes may either terminate normally with an\ninteger exit code. The exit code may span the full range of int, even\nthough all but the waitid() function retur the bottom eight bits. In\naddition to that, processes may terminate abnormally due to a signal\n(SIGABRT, SIGSEGV, etc.)\n\nPOSIX shells (sh, bash, etc.) are more restrictive, in that they can\nonly return exit codes between 0 and 126. 127 is used to denote that the\nexecutable cannot be found. Exit codes above 128 indicate that the\nprocess terminated due to a signal.\n\nRight now we let test-setup.sh terminate using the exit code obtained\nusing $?. This means that if a program terminates due to SIGABRT,\ntest-setup.sh terminates with exit code 128+6=134. This causes us to\nlose some information, as the (remote) execution environment now only\nsees plain exit codes.\n\nThis change extends test-setup.sh to check for exit codes above 128. In\nthat case it will send a signal to itself, so that the original signal\ncondition is raised once again.\n\nSee also: https://github.com/bazelbuild/remote-apis/issues/240","shortMessageHtmlLink":"test-setup.sh: Attempt to raise the original signal once more"}},{"before":"1a9623003911905e57b41d197b89187a624aa24b","after":null,"ref":"refs/heads/eschouten/20230404-create-output-directories","pushedAt":"2023-04-13T09:16:35.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"EdSchouten","name":"Ed Schouten","path":"/EdSchouten","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/736085?s=80&v=4"}},{"before":"1d5c4c4ec9ad9c1d98ba88358fc5f98c96921482","after":"1a9623003911905e57b41d197b89187a624aa24b","ref":"refs/heads/eschouten/20230404-create-output-directories","pushedAt":"2023-04-04T11:43:47.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"EdSchouten","name":"Ed Schouten","path":"/EdSchouten","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/736085?s=80&v=4"},"commit":{"message":"Prevent failures creating output directories\n\nI would receive reports of build failures along these lines:\n\n> ERROR: some/path/BUILD:156:32: Executing genrule //some/path:some_target failed: failed to create output directory '.../some/path/foo'\n\nIn order to root cause this, I made a minor change to Bazel to extend\nits logging. See https://github.com/bazelbuild/bazel/pull/17951. With\nthat change applied, the error became:\n\n> ERROR: some/path/BUILD:156:32: Executing genrule //some/path:some_target failed: failed to create output directory '.../some/path/foo': .../some/path/foo (Not a directory)\n\nI was eventually able to reproduce this by manually creating an output\nfile at bazel-bin/some/path, followed by attempting to build the target.\nIt looks like Bazel simply attempts to create output directories without\ntaking into consideration whether a file already exists at the target\nlocation. This is problematic when someone alters an existing build\ntarget to emit a directory instead of a regular file.\n\nNote that this only an issue when the remote action file system is used.\nPlain builds (ones that don't use Builds without the Bytes or\nbb_clientd) are unaffected. This change addresses this issue by reusing\nthe same fallback path creation strategy that plain builds already use.","shortMessageHtmlLink":"Prevent failures creating output directories"}},{"before":null,"after":"1d5c4c4ec9ad9c1d98ba88358fc5f98c96921482","ref":"refs/heads/eschouten/20230404-create-output-directories","pushedAt":"2023-04-04T07:30:47.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"EdSchouten","name":"Ed Schouten","path":"/EdSchouten","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/736085?s=80&v=4"},"commit":{"message":"Prevent failures creating output directories when using bb_clientd\n\nI would receive reports of build failures along these lines:\n\n> ERROR: some/path/BUILD:156:32: Executing genrule //some/path:some_target failed: failed to create output directory '.../some/path/foo'\n\nIn order to root cause this, I made a minor change to Bazel to extend\nits logging. See https://github.com/bazelbuild/bazel/pull/17951. With\nthat change applied, the error became:\n\n> ERROR: some/path/BUILD:156:32: Executing genrule //some/path:some_target failed: failed to create output directory '.../some/path/foo': .../some/path/foo (Not a directory)\n\nI was eventually able to reproduce this by manually creating an output\nfile at bazel-bin/some/path/foo, followed by attempting to build the\ntarget. It looks like Bazel simply attempts to create output directories\nwithout taking into consideration whether a file already exists at the\ntarget location. This is problematic when someone alters an existing\nbuild target to emit a directory instead of a regular file.\n\nbb_clientd reports itself as being an\nActionFileSystemType.REMOTE_FILE_SYSTEM. Based on the documentation for\nthis enum value, I think this is correct. bb_clientd is capable of\npersisting information on previous builds and reloading it. This means\nthat if a previous build yielded a regular file, a successive build\nwould need to explicitly remove it before replacing it with an output\ndirectory.\n\nreateActionFsOutputDirectories() assumes that it doesn't need to do any\ncleanups. I therefore think it's only safe to use in case of\nActionFileSystemType.IN_MEMORY_ONLY_FILE_SYSTEM. We should therefore\nonly call it if !ActionFileSystemType.supportsLocalActions().","shortMessageHtmlLink":"Prevent failures creating output directories when using bb_clientd"}},{"before":"cdb04dc58fe98c28f1504d8dc1404ba1468685b1","after":"67c6ec60cbd1acab5d8806044a635040b7f197ad","ref":"refs/heads/eschouten/20230402-error-message","pushedAt":"2023-04-04T07:12:59.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"EdSchouten","name":"Ed Schouten","path":"/EdSchouten","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/736085?s=80&v=4"},"commit":{"message":"Add exception message to 'failed to create output directory'\n\nIt looks like all of the similar code paths in this file properly attach\nthe exception message to the error message, while this one does not.\nOmitting the message makes it hard to figure out the root cause.\n\nWhile there, make a minor error message string formatting improvement.","shortMessageHtmlLink":"Add exception message to 'failed to create output directory'"}},{"before":"34d80b87e57a772801d5169756e6e6a6d8d4155c","after":"cdb04dc58fe98c28f1504d8dc1404ba1468685b1","ref":"refs/heads/eschouten/20230402-error-message","pushedAt":"2023-04-03T17:14:53.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"EdSchouten","name":"Ed Schouten","path":"/EdSchouten","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/736085?s=80&v=4"},"commit":{"message":"Add exception message to 'failed to create output directory'\n\nIt looks like all of the similar code paths in this file properly attach\nthe exception message to the error message, while this one does not.\nOmitting the message makes it hard to figure out the root cause.\n\nWhile there, make a minor error message string formatting improvement.","shortMessageHtmlLink":"Add exception message to 'failed to create output directory'"}},{"before":null,"after":"34d80b87e57a772801d5169756e6e6a6d8d4155c","ref":"refs/heads/eschouten/20230402-error-message","pushedAt":"2023-04-02T09:27:29.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"EdSchouten","name":"Ed Schouten","path":"/EdSchouten","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/736085?s=80&v=4"},"commit":{"message":"Add exception message to 'failed to create output directory'\n\nIt looks like all of the similar code paths in this file properly attach\nthe exception message to the error message, while this one does not.\nOmitting the message makes it hard to figure out the root cause.\n\nWhile there, make a minor error message string formatting improvement.","shortMessageHtmlLink":"Add exception message to 'failed to create output directory'"}}],"hasNextPage":true,"hasPreviousPage":false,"activityType":"all","actor":null,"timePeriod":"all","sort":"DESC","perPage":30,"cursor":"Y3Vyc29yOnYyOpK7MjAyNC0wNS0xNFQxODoyNzoxMy4wMDAwMDBazwAAAARJyeaC","startCursor":"Y3Vyc29yOnYyOpK7MjAyNC0wNS0xNFQxODoyNzoxMy4wMDAwMDBazwAAAARJyeaC","endCursor":"Y3Vyc29yOnYyOpK7MjAyMy0wNC0wMlQwOToyNzoyOS4wMDAwMDBazwAAAAMQWKfP"}},"title":"Activity · EdSchouten/bazel"}