Release 3.0 - March 2020 #10742
Comments
|
We continue our policy of major release every 3 month. We expect a very limited set of breaking changes in release 3.0 |
|
I hope the Bazel team can aim to review @jmillikin-stripe's work on the remote repository cache (#10818, #10622) on time for it to be included in 3.0. Just yesterday we ran into GoogleContainerTools/distroless#474, where accidentally an external dependency was removed from a gcp bucket leading to a few build failures for us. Although its not impossible to guard against that by using a resolved workspace file or http proxy, today it is quite hard to setup Bazel to use internal mirrors for all downloadable dependencies (especially the ones transitively pulled in are tricky to change). @jmillikin-stripe's work should make this significantly easier and has been in progress for a while, just want to throw another voice out there that it would solve a pain point and I would love to start using it. |
|
I will attempt to cut a release on 2020/03/04, in the morning EST. |
|
@robbertvanginkel Looks like #10818 is merged, but #10622 is still in the middle of the review. Unfortunately the team members who were most familiar with our remote execution client have changed teams and are no longer available for Bazel code reviews. I hope that we can pick this up in the next months, but in the meantime, merging PRs in that area will take more time than usual, especially of that size. 😔 @ulfjack Would you be interested and available to "own" this part of the code? I'd be happy to merge PRs that you approve! |
|
I am extremely motivated to get that change in. A delay of "months" would be overwhelmingly bad. Can I do a pair-programming review session or something over Hangouts to go over it? |
|
Hi folks, My name is Andrew, and I'm the PTL for Security here at Stripe. This feature is very important for us; we're counting on it to be able to ensure the integrity and provenance of dependencies in our builds. The fact that it has taken almost 45 days to get a review on an accepted proposal, which was itself submitted nearly a year ago, and especially with someone that has been so willing to take feedback and iterate quickly, is starting to erode my confidence in Bazel's ability to accept contributions from outside Google. Can you please prioritize reviewing and merging @jmillikin-stripe's work? Best, |
|
First I want to apologize to @jmillikin-stripe for the long review on that PR. unfortunately, we've discovered that we've lost all of our in-house domain experts on that code (either they've left the team or the company entirely). This means that not only is it difficult to accept external contributions to this code, it's difficult to accept internal contributions! We're working to find and train people to handle this, and we're sorry that it hasn't been ideal. For PR #10622 specifically, @michajlo has volunteered to take a look (and is actually in the same timezone as the author, so that should help with turnaround time). I am going to go ahead and cut the RC for 3.0 today, but assuming #10622 is committed in the next few days, I will cherrypick it into a RC before the final release to ensure that it is released with 3.0. My understanding is that the code changes are behind an experimental flag so this seems safe, but I will make sure to run additional tests to verify the RC. Again, I want to apologize both personally and for the entire team that this has been so difficult to get organized. |
Status of Bazel 3.0
To report a release-blocking bug, please file a bug using the Task list:
|
|
I'm definitely interested in reviewing changes to the remote execution parts of the codebase. As a caveat, I am starting my own company and that's where most of my time is going, so I can't promise an SLA on code reviews. Feel free to ping me directly if the progress is too slow. |
|
RC1 created: Currently uploading and building. |
|
RC is created: https://releases.bazel.build/3.0.0/rc1/index.html |
|
Downstream tests have passed: https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/1412 |
|
I'd like to propose these two commits for cherry-picking: 4ee52db and 9ada0fe. The rationale is that these commits will allow Bazel 3.0.0 to work alongside sandboxfs 0.2.0 (about to be released), which provides major performance improvements over the previous version. But without these commits, Bazel cannot understand the protocol spoken by 0.2.0 so things would not work. The risk of these cherry-picks is minimal because they only affect code that is gated behind the |
|
I propose not to cherry-pick feature work. This is against our policy, and 3.1 will be out in a month. |
|
I've noticed that |
|
Sounds like it should be migration-3.0: @katre could you fix that? Thanks! |
|
I've renamed the label. |
|
Did I misunderstand the release playbook? At https://github.com/bazelbuild/continuous-integration/blob/master/docs/release-playbook.md#review-breaking-changes I thought I was supposed to add a "migration-4.0" (that being the next breaking changes release) to anything not ready to migrate yet. |
|
You need add a label migration-3.0 and breaking-change-4.0. With Bazel 3.0, users should migrate for changes that will break in Bazel 4.0. (feel free to update the playbook if it's confusing) |
|
@dslomov I don't know exactly what "I propose not to cherry-pick feature work." was in reference to, but I suspect it's about my sandboxfs commits. Those changes I referenced are fixes, not "feature work", because if we release Bazel 3.0 as it is today, its support for sandboxfs 0.2.0 will be broken. I could do other contortions to avoid user visible breakage and then wait for 3.1, but I'd rather not have to do those. |
|
@jmmv support for sandboxfs 0.2.0 is a feature, not a bug: as far as I understand sandboxfs 0.2.0 is not even released yet, right? |
|
Because Bazel 3.0 already has support for sandboxfs 0.2.0, but it's broken. |
|
Ahh, ok now I understand - thanks for bearing with me. In that case, I leave the final decision to the release manager, @katre. |
|
Thanks. It's not a big deal really and I only raised this because I had heard from @katre that he might have to cut a new RC anyway due to other cherrypicks. So: if we have to cut another RC, I'd appreciate if these two commits can make it into it; if not, let's leave the release as is. |
|
Correction: I am not cherrypicking f32b0fd, because it is before the baseline and so is already included. |
|
Commands: Had to manually clean up applying 289efe5 due to conflicts in remote/BUILD. |
Extracted from #10622 Per discussion on that PR, there's still some unanswered questions about how exactly we plumb the new `Downloader` type into `RemoteModule`. And per #10742 (comment), it is unlikely that even heroic effort from me will get the full end-to-end functionality into v3.0. Given this, to simplify the review, I'm taking some of the bits the reviewer is happy with and moving them to a separate PR. After merger, `GrpcRemoteDownloader` and its tests will exist in the source tree, but will not yet be available as CLI options. R: @michajlo CC: @adunham-stripe @dslomov @EricBurnett @philwo @sstriker Closes #10914. PiperOrigin-RevId: 299908615
|
RC2 is available at https://releases.bazel.build/3.0.0/rc2/index.html |
|
@jmillikin-stripe Please test rc2 for the download functionality, let me know if there are any remaining issues. |
|
Downstream tests pass: https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/1419 |
|
My local tests of the remote downloader are working with the rc2 build. |
|
The sandboxfs fixes look good: https://travis-ci.org/github/jmmv/sandboxfs/jobs/661101748 😬 Thanks! |
|
The release looks good based on the lack of any comments, but we're waiting on some internal processes before finalizing. Hopefully it will be this week. |
|
My apologies for the continued delay on releasing 3.0.0. We are still dealing with slowdowns and internal processes due to the disruption in business, and hopefully will be able to finalize the release soon. Unfortunately I can't even guess at a schedule at this point. |
|
@meteorcloudy noticed I tested the wrong branch for the RC2 build, the correct tests are here: https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/1442. The only failures (NodeJS on Windows) have the same error at HEAD (https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/1441#72508030-fd92-47c8-85da-c44e0c8d9f18) and seems to be infrastructure-related. |
|
rules_nodejs is on RC2 itself https://github.com/bazelbuild/rules_nodejs/blob/master/.bazelversion so yeah you can ignore that CI failure. We are fighting some flakes lately. |
|
We have all internal approvals, so I will finalize the release today. |
|
I have pushed the release: https://github.com/bazelbuild/bazel/releases/tag/3.0.0 @vbatts @petemounce @excitoon, can you update? |
|
The release is finished (except for updating Homebrew (bazelbuild/homebrew-tap#90), and other downstream release channels. |
|
fedora/centos builds of "bazel3" package are in progress
https://copr.fedorainfracloud.org/coprs/build/1330079
|
All succeeded except epel7-aarch64 and fedora30-aarch64. Other aarch64 did build successfully, so I am investigating these to see if their build chroot had issue. |
|
@vbatts any conclusions on the aarch64 build failure investigation mentioned above? |
|
@vielmetti Apologies. This fell off my radar. The logs feel like it was an issue with the chroot. I have kicked off a build of just these two chroots https://copr.fedorainfracloud.org/coprs/vbatts/bazel/build/1336263/ |
|
epel7-aarch64 failed again pretty quickly. the log doesn't show any smoking gun still Start: rpmbuild bazel3-3.0.0-1.el7.src.rpm
Building target platforms: aarch64
Building for target aarch64
Executing(%prep): /bin/sh -e /var/tmp/rpm-tmp.gJwBJt
+ umask 022
+ cd /builddir/build/BUILD
+ cd /builddir/build/BUILD
+ rm -rf bazel-3.0.0
+ /usr/bin/mkdir -p bazel-3.0.0
+ cd bazel-3.0.0
+ /usr/bin/unzip -qq /builddir/build/SOURCES/bazel-3.0.0-dist.zip
+ STATUS=0
+ '[' 0 -ne 0 ']'
+ /usr/bin/chmod -Rf a+rX,u+w,g-w,o-w .
+ echo 'Patch #1 (bazel-1.0.0-log-warning.patch):'
Patch #1 (bazel-1.0.0-log-warning.patch):
+ /usr/bin/cat /builddir/build/SOURCES/bazel-1.0.0-log-warning.patch
+ /usr/bin/patch -p0 --fuzz=0
patching file src/java_tools/buildjar/java/com/google/devtools/build/buildjar/javac/plugins/dependency/StrictJavaDepsPlugin.java
Hunk #1 succeeded at 178 (offset -1 lines).
+ exit 0
Executing(%build): /bin/sh -e /var/tmp/rpm-tmp.PUgT5N
+ umask 022
+ cd /builddir/build/BUILD
+ cd bazel-3.0.0
+ export 'EXTRA_BAZEL_ARGS= --host_force_python=PY2'
+ EXTRA_BAZEL_ARGS=' --host_force_python=PY2'
+ export 'EXTRA_BAZEL_ARGS= --host_force_python=PY2 --nokeep_state_after_build --notrack_incremental_state --nokeep_state_after_build'
+ EXTRA_BAZEL_ARGS=' --host_force_python=PY2 --nokeep_state_after_build --notrack_incremental_state --nokeep_state_after_build'
+++ head -1 CHANGELOG.md
+++ /usr/bin/grep -Eo '\b[[:digit:]]{4}-[[:digit:]]{2}-[[:digit:]]{2}\b'
++ date -d 2020-04-06 +%s
+ export SOURCE_DATE_EPOCH=1586131200
+ SOURCE_DATE_EPOCH=1586131200
+ export EMBED_LABEL=3.0.0
+ EMBED_LABEL=3.0.0
+ which g++
/usr/bin/g++
+ g++ --version
g++ (GCC) 4.8.5 20150623 (Red Hat 4.8.5-39)
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+ export TMPDIR=/var/tmp
+ TMPDIR=/var/tmp
+ export CC=gcc
+ CC=gcc
+ export CXX=g++
+ CXX=g++
+ export 'EXTRA_BAZEL_ARGS= --host_force_python=PY2 --nokeep_state_after_build --notrack_incremental_state --nokeep_state_after_build --sandbox_debug --host_javabase=@local_jdk//:jdk --verbose_failures'
+ EXTRA_BAZEL_ARGS=' --host_force_python=PY2 --nokeep_state_after_build --notrack_incremental_state --nokeep_state_after_build --sandbox_debug --host_javabase=@local_jdk//:jdk --verbose_failures'
+ env ./compile.sh
Building Bazel from scratcherror: Bad exit status from /var/tmp/rpm-tmp.PUgT5N (%build)
RPM build errors:
Bad exit status from /var/tmp/rpm-tmp.PUgT5N (%build)
Finish: rpmbuild bazel3-3.0.0-1.el7.src.rpm |
|
(and the fedora-30-aarch64 succeeded) |
@vbatts Hi, should I specify any particular arguments while building bazel for my aarch64 board? Or, just the following command should do? $ bazel version
Extracting Bazel installation...
Starting local Bazel server and connecting to it...
Build label: 3.0.0- (@non-git)
Build target: bazel-out/aarch64-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Mon Apr 20 22:21:14 2020 (1587421274)
Build timestamp: 1587421274
Build timestamp as int: 1587421274I tried to build tensorflow afterwards, but got the following ERRORS: ERROR: ~/.cache/bazel/_bazel_khadas/0de19da26a472240f23447517e34d888/external/local_config_sycl/crosstool/BUILD:12:1: @local_config_sycl//crosstool:cc-compiler-local: missing value for mandatory attribute 'toolchain_config' in 'cc_toolchain' rule
ERROR: ~/.cache/bazel/_bazel_khadas/0de19da26a472240f23447517e34d888/external/local_config_sycl/crosstool/BUILD:12:1: Target '@local_config_sycl//crosstool:empty' contains an error and its package is in error and referenced by '@local_config_sycl//crosstool:cc-compiler-local'
ERROR: ~/.cache/bazel/_bazel_khadas/0de19da26a472240f23447517e34d888/external/local_config_sycl/crosstool/BUILD:5:1: Target '@local_config_sycl//crosstool:cc-compiler-local' contains an error and its package is in error and referenced by '@local_config_sycl//crosstool:toolchain'
ERROR: ~/.cache/bazel/_bazel_khadas/0de19da26a472240f23447517e34d888/external/bazel_tools/src/tools/launcher/BUILD:9:1: every rule of type cc_binary implicitly depends upon the target '@local_config_sycl//crosstool:toolchain', but this target could not be found because of: Target '@local_config_sycl//crosstool:toolchain' contains an error and its package is in error
ERROR: Analysis of target '//tensorflow/tools/pip_package:build_pip_package' failed; build aborted: Analysis failed
INFO: Elapsed time: 0.724s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded, 3 targets configured) |
|
@katre - Thank you for your help with getting this release out. |
Cut date 2020-03-02
The text was updated successfully, but these errors were encountered: