tracking issue for upgrading protobuf to 3.19.0+ (and gRPC as well) #5708
Description
Googlers, see also b/219030239.
This is a tracking issue for updating our protobuf dependency to 3.19.0 or greater. That would simplify our lives in several respects: it would solve #5703, and it would allow us to upgrade gRPC past 1.30.0 (blocked by grpc/grpc#23311).
That update would involve both updating the protoc compiler we use to generate Python bindings, as well as updating the Python protobuf runtime version we depend on (see below for what that means).
This is currently blocked on TensorFlow's own upgrade (which is blocked by tensorflow/tensorflow#53234; Googlers, see b/182876485). We're blocked on TF because a previous attempt to update protobuf to 3.18.1 resulted in build breakages for other TF-ecosystem projects that rely on us, since our generated protos were incompatible with the older protobuf runtimes required by those projects.
Explainer: the way we depend on protobuf is a bit subtle. We build our .proto files into python libraries (py_pb2 targets) using this tb_proto_library build rule: https://github.com/tensorflow/tensorboard/blob/master/tensorboard/defs/protos.bzl
(Example of tb_proto_library usage here)
The build rule depends on 2 components that are provided by the protobuf dependency:
-
protoc- the protocol buffer compiler:tensorboard/tensorboard/defs/protos.bzl
Line 35 in 454cfca
-
the "protobuf Python runtime":
tensorboard/tensorboard/defs/protos.bzl
Line 28 in 454cfca
The protoc compiler is the tool that at build time actually generates _pb2.py files from the .proto files. Note however that we check the generated pb2.py files into our pip package, so that users don't need to run protoc themselves to execute TensorBoard locally, but when building TensorBoard with Bazel we do run protoc for this reason (which has its own issues).
The protobuf Python runtime is a common library that provides much of the implementation of the generated _pb2.py files, and as such it is required to be present at runtime in order to actually use those proto messages. Also, the protobuf compatibility policy is that the runtime version must always be at least as high as the protoc version used to generated the _pb2.py files. The protobuf Python runtime provided as a Bazel build-time dependency when building with Bazel, which basically guarantees that we use the same version for both runtime and protoc. However, running the TensorBoard pip package, we don't have the Bazel-built dependencies available and instead get our dependencies from installed pip packages, where the protobuf runtime is provided by the pip package protobuf: https://pypi.org/project/protobuf/ That opens up the possibility of version skew, which is why we set a lower bound in our requirements.txt. This lower bound must always be at least as high as the protoc version used to compile the protos in that release of TensorBoard.