Excessive new warning in Python protobuf runtime

[chrismiller]

What version of protobuf and what language are you using?
Version: 5.28.0
Language: Python

What operating system (Linux, Windows, ...) and version?
Linux, but not relevant

What runtime / compiler are you using (e.g., python version or gcc version)
Compiler 5.27.2, runtime 5.28.0

What did you do?
Compile a .proto using the libprotoc 27.2 compiler, then run my code using the v.5.28.0 runtime.

What did you expect to see
I expect this to work without fuss, since it is a supported configuration according to https://protobuf.dev/support/cross-version-runtime-guarantee/#minor

What did you see instead?
A Python warning is emitted, saying:
google/protobuf/runtime_version.py:112: UserWarning: Protobuf gencode version 5.27.2 is older than the runtime version 5.28.0 at path/to/example.proto. Please avoid checked-in Protobuf gencode that can be obsolete.

This is a new warning that was intentionally added in v5.28.0 as a result of #17241. It seems excessive however as it can't easily be avoided in many situations, and can be awkward to disable. For example, I need the v5.28.0 runtime (due to a specific bugfix that's not in 5.27.x), however I also have some GRPC services, plus I use the mypy-protobuf plugin, so my build scripts can't just call the protoc 5.28.0 compiler binary directly. Instead, I am using a build venv containing grpcio-tools (which only has the 27.2 compiler). This means I can't reasonably avoid the runtime warning, and so have to jump through hoops to suppress it. See also GRPC issue grpc/grpc#37609 for further discussion on the impact of this. Given grpcio-tools always tends to lag the latest protoc version, I imagine others will encounter similar situations with this warning on an ongoing basis too.

It seems to me that a solution should really lie with the protobuf project rather than grpc. I appreciate the reasoning behind the new warning, but having a way to prevent the protoc compiler generating it (e.g. via a --warning-level <n> flag on protoc?) would make life a lot easier for people in situations like mine. As it stands I'm having to add several __init__.py files to various repositories, as well as generate some more of them dynamically from both makefiles and bazel scripts, to work around this issue.

[googleberg]

Thank you for the feeback. We'll review at our next technical blockers meeting.

[ummels]

Same issue happening with protobuf-java 4.28.x.

[googleberg]

We'll fix this in the 29.x release.

[shaod2]

We have pending fixes that might be back-ported to 28.x

[andrea-cassioli-maersk]

Same here on python, actually getting an error

google.protobuf.runtime_version.VersionError: Detected incompatible Protobuf Gencode/Runtime versions when loading ndse_network.proto: gencode 5.28.1 runtime 5.27.0. Runtime version cannot be older than the linked gencode version. See Protobuf version guarantees at https://protobuf.dev/support/cross-version-runtime-guarantee.

[chrismiller]

Same here on python, actually getting an error

google.protobuf.runtime_version.VersionError: Detected incompatible Protobuf Gencode/Runtime versions when loading ndse_network.proto: gencode 5.28.1 runtime 5.27.0. Runtime version cannot be older than the linked gencode version. See Protobuf version guarantees at https://protobuf.dev/support/cross-version-runtime-guarantee.

This error is different, and more valid. As the error says, using a runtime version that's lower than the compiler version isn't guaranteed to work. You should upgrade your protobuf runtime to 5.28.1 (or recompile with an older compiler version).

[stormblessed88]

Python users

There's a workaround for this until a proper fix is in place. I've fixed this by performing the steps below. Please note that my version mismatch was between gencode version 5.27.2 and runtime version 5.28.2

1. Download protoc binary for my os (Ubuntu) separately:

curl -OL https://github.com/protocolbuffers/protobuf/releases/download/v28.2/protoc-28.2-linux-x86_64.zip
sudo unzip -o protoc-28.2-linux-x86_64.zipP -d /usr/local bin/protoc
sudo unzip -o protoc-28.2-linux-x86_64.zip -d /usr/local 'include/*'
rm -f protoc-28.2-linux-x86_64.zip

2. Generate new .pb2.py using the proto and protoc binary

protoc -I=. --python_out=. ./my_proto_file.proto

Note that you can keep the same _pb2_grpc.py that you have previously generated with Python grpc_tools.protoc compiler.

[vsoch]

This was really helpful! My full workflow (after doing the curl / unzip install above) is to run the grpc one in python first, then protoc, and then I still need to do some sed fu to get the imports right.

python -m grpc_tools.protoc -I./protos --python_out=./ensemble/protos --pyi_out=./ensemble/protos --grpc_python_out=./ensemble/protos ./protos/ensemble-service.proto
protoc -I=./protos --python_out=./ensemble/protos ./protos/ensemble-service.proto
sed -i 's/import ensemble_service_pb2 as ensemble__service__pb2/from . import ensemble_service_pb2 as ensemble__service__pb2/' ./ensemble/protos/ensemble_service_pb2_grpc.py

Thanks to all of the folks above for the help, I had that exact warning and now it's gone. :)

[shaod2]

We have the fix in main now 67662fe

@zhangskz we should backport it to the 28.x