Skip to content

infra: remove dependency on org_tensorflow #5528

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
yatbear merged 15 commits into from
Feb 3, 2022
Merged

infra: remove dependency on org_tensorflow #5528

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
d81f789
remove dependency on `org_tensorflow`
yatbear Jan 24, 2022
27c102e
remove wrong link, revert bazel version check change
yatbear Jan 24, 2022
5137877
upgrade bazel to 5.0.0
yatbear Jan 24, 2022
aca636e
update protos
yatbear Jan 25, 2022
dc4f32a
use es2022
yatbear Jan 25, 2022
184ace0
update grpc dep to v1.42.0
yatbear Jan 31, 2022
21578f8
add mirror link for rules_rust
yatbear Jan 31, 2022
3ba8e77
fix bazel_skylib mirror link and uploaded missing links (go/tensorboa…
yatbear Jan 31, 2022
46a027e
rm `third_party/tensorflow.patch`
yatbear Feb 1, 2022
268ea4f
add mirror link for grpc
yatbear Feb 1, 2022
25bf072
update rules_rust dep to HEAD
yatbear Feb 1, 2022
41ee045
update bazel_skylib sha256 && add strip_prefix
yatbear Feb 1, 2022
772d658
upgrade to Rust 1.58.1
yatbear Feb 1, 2022
5d80e74
set mininum bazel version to 4.0.0
yatbear Feb 1, 2022
8392de4
fix bazel checksum
yatbear Feb 1, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,8 +27,8 @@ on:
env:
# Keep this Bazel version in sync with the `versions.check` directive
# in our WORKSPACE file.
BAZEL_VERSION: '3.7.0'
BAZEL_SHA256SUM: 'b7583eec83cc38302997098a40b8c870c37e0ab971a83cb3364c754a178b74ec'
BAZEL_VERSION: '4.0.0'
BAZEL_SHA256SUM: '7bee349a626281fc8b8d04a7a0b0358492712377400ab12533aeb39c2eb2b901'
BUILDTOOLS_VERSION: '3.0.0'
BUILDIFIER_SHA256SUM: 'e92a6793c7134c5431c58fbc34700664f101e5c9b1c1fcd93b97978e8b7f88db'
BUILDOZER_SHA256SUM: '3d58a0b6972e4535718cdd6c12778170ea7382de7c75bc3728f5719437ffb84d'
Expand Down Expand Up @@ -121,11 +121,11 @@ jobs:
matrix:
mode: ['native']
platform: ['ubuntu-18.04', 'macos-10.15']
rust_version: ['1.52.0']
rust_version: ['1.58.1']
include:
- mode: 'universal'
platform: 'ubuntu-18.04'
rust_version: '1.52.0'
rust_version: '1.58.1'
steps:
- uses: actions/checkout@50fbc622fc4ef5163becd7fab6573eac35f8462e
- uses: actions/setup-python@152ba7c4dd6521b8e9c93f72d362ce03bf6c4f20
Expand Down Expand Up @@ -244,7 +244,7 @@ jobs:
runs-on: ubuntu-18.04
strategy:
matrix:
rust_version: ['1.52.0']
rust_version: ['1.58.1']
cargo_raze_version: ['0.13.0']
steps:
- uses: actions/checkout@50fbc622fc4ef5163becd7fab6573eac35f8462e
Expand Down
70 changes: 34 additions & 36 deletions WORKSPACE
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,22 @@ workspace(name = "org_tensorflow_tensorboard")

load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")

http_archive(
name = "bazel_skylib",
sha256 = "07b4117379dde7ab382345c3b0f5edfc6b7cff6c93756eac63da121e0bbcc5de",
strip_prefix = "bazel-skylib-1.1.1",
urls = [
"http://mirror.tensorflow.org/github.com/bazelbuild/bazel-skylib/archive/1.1.1.tar.gz",
"https://github.com/bazelbuild/bazel-skylib/releases/download/1.1.1/bazel-skylib-1.1.1.tar.gz", # 2021-09-27
],
)

load("@bazel_skylib//lib:versions.bzl", "versions")

# Keep this version in sync with the BAZEL environment variable defined
# in our .github/workflows/ci.yml config.
versions.check(minimum_bazel_version = "4.0.0")

http_archive(
name = "io_bazel_rules_webtesting",
sha256 = "9bb461d5ef08e850025480bab185fd269242d4e533bca75bfb748001ceb343c3",
Expand Down Expand Up @@ -73,35 +89,6 @@ http_archive(
],
)

http_archive(
name = "org_tensorflow",
patches = [
# Patch TF's python_configure.bzl to ensure it reconfigures its python
# toolchain when environment variables like `PATH` and `PYTHONPATH`
# change, to avoid the stale genrule py_binary issue described in:
# https://github.com/tensorflow/tensorboard/issues/4862
"//third_party:tensorflow.patch",
],
# NOTE: when updating this, MAKE SURE to also update the protobuf_js runtime version
# in third_party/workspace.bzl to >= the protobuf/protoc version provided by TF.
sha256 = "2595a5c401521f20a2734c4e5d54120996f8391f00bb62a57267d930bce95350",
strip_prefix = "tensorflow-2.3.0",
urls = [
"http://mirror.tensorflow.org/github.com/tensorflow/tensorflow/archive/v2.3.0.tar.gz", # 2020-07-23
"https://github.com/tensorflow/tensorflow/archive/v2.3.0.tar.gz",
],
)

load("@org_tensorflow//tensorflow:workspace.bzl", "tf_workspace")

tf_workspace()

load("@bazel_skylib//lib:versions.bzl", "versions")

# Keep this version in sync with the BAZEL environment variable defined
# in our .github/workflows/ci.yml config.
versions.check(minimum_bazel_version = "3.7.0")

load("@io_bazel_rules_sass//:package.bzl", "rules_sass_dependencies")

rules_sass_dependencies()
Expand All @@ -110,22 +97,33 @@ load("@io_bazel_rules_sass//:defs.bzl", "sass_repositories")

sass_repositories()

# gRPC.
http_archive(
name = "com_github_grpc_grpc",
sha256 = "b2f2620c762427bfeeef96a68c1924319f384e877bc0e084487601e4cc6e434c",
strip_prefix = "grpc-1.42.0",
urls = [
"http://mirror.tensorflow.org/github.com/grpc/grpc/archive/v1.42.0.tar.gz",
"https://github.com/grpc/grpc/archive/v1.42.0.tar.gz", # 2021-11-17
],
)

load("@com_github_grpc_grpc//bazel:grpc_deps.bzl", "grpc_deps")

grpc_deps()
Copy link
Contributor

@nfelt nfelt Jan 29, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So without TF, it looks like grpc is requiring that we have python headers installed in order to build - when I checked out this PR and try to build it, I initially got this:

ERROR: /usr/local/google/home/nickfelt/tb/WORKSPACE:129:10: //external:python_headers depends on @local_config_python//:python_headers in repository @local_config_python which failed to fetch. no such package '@local_config_python//': Python Configuration Error: Unable to find Python headers for /usr/bin/python2
Traceback (most recent call last):
  File "<string>", line 1, in <module>
AssertionError: /usr/include/python2.7/Python.h does not exist.
Are Python headers installed? Try installing python-dev or python3-dev on Debian-based systems. Try python-devel or python3-devel on Redhat-based systems.

On my workstation I already have python3-dev installed, and I was able to sudo apt-get install python2-dev, which resolved the issue. But that's still a bit of a regression from before when we didn't need python headers installed at all. As far as I could tell, it boils down to our python toolchain configuration - grpc uses a slightly different python_configure() rule from TensorFlow and I guess it's somehow stricter.

(As for the root issue, I looked around and found grpc/grpc#24665, which seems related. It sounds like there's some ambiguity in general about how the grpc header dependency should work.)

Anyway - I guess this is fine, but let's add a mention about the need to have python-dev or python2-dev and python3-dev installed to DEVELOPMENT.md?

Copy link
Member Author

@yatbear yatbear Jan 31, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think I have python{1,2,3}-dev or python2 on my workstation, and my python version is 3.9.9. Will investigate the issue further.

Copy link
Contributor

@nfelt nfelt Feb 1, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, that's a bit odd. My python is also 3.9.9. I guess there must be some configuration difference?

Maybe we should check w/ one other person just in case, but if nobody else is seeing this error, then I'm fine just proceeding and not mentioning it in DEVELOPMENT.md since the error is at least pretty self-explanatory about what to do to fix it.

Copy link
Member Author

@yatbear yatbear Feb 2, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was able to reproduce the same error after installing Python2, related issue: grpc/grpc#21963.


load("@upb//bazel:repository_defs.bzl", "bazel_version_repository")
load("@com_github_grpc_grpc//bazel:grpc_extra_deps.bzl", "grpc_extra_deps")

bazel_version_repository(name = "bazel_version")
grpc_extra_deps()

http_archive(
name = "rules_rust",
sha256 = "9c7d7fd4378d75232858423d574cff677b16ac9a49cd1a11df545a6e72a315ca",
strip_prefix = "rules_rust-42a674093251fb6a603a2652980c9be28a6fea50",
sha256 = "08109dccfa5bbf674ff4dba82b15d40d85b07436b02e62ab27e0b894f45bb4a3",
strip_prefix = "rules_rust-d5ab4143245af8b33d1947813d411a6cae838409",
urls = [
# Master branch as of 2021-04-14
"http://mirror.tensorflow.org/github.com/bazelbuild/rules_rust/archive/42a674093251fb6a603a2652980c9be28a6fea50.tar.gz",
"https://github.com/bazelbuild/rules_rust/archive/42a674093251fb6a603a2652980c9be28a6fea50.tar.gz",
# Master branch as of 2022-01-31
"http://mirror.tensorflow.org/github.com/bazelbuild/rules_rust/archive/d5ab4143245af8b33d1947813d411a6cae838409.tar.gz",
"https://github.com/bazelbuild/rules_rust/archive/d5ab4143245af8b33d1947813d411a6cae838409.tar.gz",
],
)

Expand Down
6 changes: 3 additions & 3 deletions tensorboard/data/server/BUILD
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
load("@rules_rust//rust:rust.bzl", "rust_binary", "rust_doc", "rust_doc_test", "rust_library", "rust_test")
load("@rules_rust//rust:defs.bzl", "rust_binary", "rust_doc", "rust_doc_test", "rust_library", "rust_test")

package(default_visibility = ["//tensorboard:internal"])

Expand Down Expand Up @@ -117,13 +117,13 @@ rust_binary(

rust_doc_test(
name = "rustboard_core_doc_test",
dep = ":rustboard_core",
crate = ":rustboard_core",
tags = ["manual"], # https://github.com/bazelbuild/rules_rust/issues/689
)

rust_doc(
name = "rustboard_core_doc",
dep = ":rustboard_core",
crate = ":rustboard_core",
tags = ["manual"], # https://github.com/bazelbuild/rules_rust/issues/689
)

Expand Down
Binary file modified tensorboard/data/server/descriptor.bin
View file
Open in desktop
Binary file not shown.
2 changes: 1 addition & 1 deletion tensorboard/data/server/tensorboard.data.pb.rs
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion tensorboard/data/server/tensorboard.pb.rs
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 2 additions & 2 deletions tensorboard/data/server/update_protos.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Copyright 2020 The TensorFlow Authors. All Rights Reserved.
# Copyright 2022 The TensorFlow Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
Expand Down Expand Up @@ -28,7 +28,7 @@
_FILE_DESCRIPTOR_SET = "descriptor.bin"

_RUST_LICENSE = """\
/* Copyright 2020 The TensorFlow Authors. All Rights Reserved.
/* Copyright 2022 The TensorFlow Authors. All Rights Reserved.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
Expand Down
2 changes: 1 addition & 1 deletion third_party/rust.bzl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,5 +21,5 @@ load("//third_party/rust:crates.bzl", "raze_fetch_remote_crates")

def tensorboard_rust_workspace():
"""TensorBoard Rust dependencies."""
rust_repositories(version = "1.52.0")
rust_repositories(version = "1.58.1")
raze_fetch_remote_crates()
12 changes: 0 additions & 12 deletions third_party/tensorflow.patch
View file
Open in desktop

This file was deleted.

6 changes: 6 additions & 0 deletions third_party/workspace.bzl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,12 @@ def tensorboard_workspace(name = ""):
actual = "@org_pythonhosted_six",
)

# Needed by Protobuf.
native.bind(
name = "grpc_python_plugin",
actual = "@com_github_grpc_grpc//src/compiler:grpc_python_plugin",
)

platform_http_file(
name = "org_chromium_chromium", # pinned to Chromium 84.0.4147.0
licenses = ["notice"], # BSD 3-clause (maybe more?)
Expand Down