Update to Protobuf 4.x

[neeme-praks-sympower]

Edit by @ejona86: Protobuf has resolved the incompatibility with protobuf-java 4.x. Use protobuf-java 4.27.4 or later. This is being left open a bit longer for gRPC itself to upgrade, but you don't need to wait for gRPC.

---

Is your feature request related to a problem?

I would like to use the latest released version of Protobuf (4.26.0) but I cannot as gRPC is still using the old version.

Describe the solution you'd like

Update Protobuf dependency and make it work with the new version.

Describe alternatives you've considered

I hoped that the update would be easy (just a version bump) so I tried doing it myself but it resulted in a compilation failure so I'll leave it to you to figure it out.

The failure:

> Task :grpc-protobuf:compileJava FAILED
grpc-java/protobuf/src/main/java/io/grpc/protobuf/StatusProto.java:38: error: cannot access GeneratedMessageV3
          ProtoLiteUtils.metadataMarshaller(com.google.rpc.Status.getDefaultInstance()));
                                                                 ^
  class file for com.google.protobuf.GeneratedMessageV3 not found
grpc-java/protobuf/src/main/java/io/grpc/protobuf/StatusProto.java:171: error: cannot access Builder
        .setCode(status.getCode().value());
        ^
  class file for com.google.protobuf.GeneratedMessageV3$Builder not found
2 errors

[ganadist]

https://protobuf.dev/news/2023-12-05

Also, I'm wondering how this protobuf changes will affect

• gRPC public API changes
• Compatibility with artifacts which were generated by old protobuf/gRPC compilers.

[ejona86]

The protobuf breakage is pretty serious and goes against wisdom like https://jlbp.dev/JLBP-7 . I don't see how the community can absorb it. googleapis libraries are hit even harder than we are, so I think we'd wait until they have an idea how to deal with this (otherwise they couldn't upgrade to newer grpc versions). I don't expect that to be too big of a problem for the community, as I think most protobuf-using projects will be in shock.

I'm in discussions with others, but I'd expect the process to figure this out to take months.

[augi]

After updating to Protobuf 4.26.0, we observe this issue:

java.lang.NoClassDefFoundError: com/google/protobuf/GeneratedMessageV3
	at java.base/java.lang.ClassLoader.defineClass1(Native Method)
	at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1017)
	at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:150)
	at java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:862)
	at java.base/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(BuiltinClassLoader.java:760)
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:681)
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:639)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:525)
	at io.grpc.health.v1.HealthGrpc.getCheckMethod(HealthGrpc.java:38)
	at io.grpc.health.v1.HealthGrpc.getServiceDescriptor(HealthGrpc.java:413)
	at io.grpc.health.v1.HealthGrpc.bindService(HealthGrpc.java:350)
	at io.grpc.health.v1.HealthGrpc$HealthImplBase.bindService(HealthGrpc.java:169)

The announcement of the Protobuf release contains this: [Java] The base class for generated messages will be GeneratedMessage, not GeneratedMessageV3.

So I guess this has to be adjusted.

[suztomo]

@neeme-praks-sympower Does any library or dependency require you to upgrade Protobuf runtime version to 4.x?

[be-hase]

It seems that proto-google-common-protos, which grpc-protobuf depends on, is not yet ver4 compliant.
https://github.com/googleapis/sdk-platform-java/tree/main/java-common-protos

[krickert]

Why couldn't grpc just bring back generated message v3? The classes appear almost identical, I suppose it can just extend it?

I tried that myself - it somewhat worked. I got past the generated message v3 error, but I was getting serializaiton errors everywhere. I think they may have changed the serializaiton spec too?

Anyway, I'm looking forward to this fix. I might try this again today.

[neeme-praks-sympower]

@neeme-praks-sympower Does any library or dependency require you to upgrade Protobuf runtime version to 4.x?

No. We just try to keep our dependencies up-to-date.

[suztomo]

@neeme-praks-sympower Thank you.

[ummels]

I've opened an issue at Protobuf for restoring GeneratedMessageV3 as an alias for GeneatedMessage.

[krickert]

I've opened an issue at Protobuf for restoring GeneratedMessageV3 as an alias for GeneatedMessage.

I didn't think you're going to win that though

It was a major upgrade with breaking changes. They deprecated V3 awhile ago, better to fix instead of bringing it back. It'll take some time, but GRPC isn't always caught up to the latest protocol buffers.

[ejona86]

Appears the further issues may be resolved soon. Testing with the 4.28.0-RC3 looked good to me. protocolbuffers/protobuf#17247 (comment)

[ejona86]

Protobuf 27.4 was released yesterday. We're targeting upgrading in 1.67.

[ascopes]

do we have a timescale for this, out of curiosity?

[ejona86]

@ascopes, "We're targeting upgrading in 1.67". The schedule for 1.67 is in the milestone (Sept 17th). But you don't have to wait on us. You can upgrade protobuf today in your own app.

[ascopes]

@ejona86 thanks, was asking as on the most recent version of protobuf and gRPC, the introspection service for gRPC-java was still failing due to the use of v3 APIs within protobuf.

[ejona86]

@ascopes, use protobuf-java 4.27.4 or later.

[ascopes]

thanks, shall give it a try.

[dan-lind]

@ascopes, "We're targeting upgrading in 1.67". The schedule for 1.67 is in the milestone (Sept 17th). But you don't have to wait on us. You can upgrade protobuf today in your own app.

Was wondering if there are any updates on this? From what I can tell there is still no update, and it's not targeted for 1.69 either?

[ejona86]

The current plan is "wait and see." We're staying on 3.x until it seems our downstreams are successfully upgrading to 4.x. Basically, we're waiting until it is clear the ecosystem is compatible compatible with 4.x. That is most safely done from the leaves of dependency trees up to the root.

Notably, Google Cloud Java has a separate BOM for protobuf java 3.x, which is identical to the regular BOM except which protobuf-java runtime version it uses. They're watching the uptake of the 4.x version and we're talking with a few larger companies to see if there's common libraries that need fixes before being compatible with 4.x.

Us staying on 3.x shouldn't impact anyone for now. But we do encourage using 4.x when able so that you're more prepared for protobuf-java 5.x. If one of your dependencies isn't compatible with 4.x, please file an issue with them (and if it is a commonly-used library/project, you can share a link to the issue here for our tracking).