[Bug] Code that is generated with the protobuf version 3.22.0+ doesn't work and logs a IllegalAccessError exception

[artsiomau]

Search before asking

• I searched in the issues and found nothing similar.

Version

3.0.0

Minimal reproduce step

add the pulsar-client-3.0.0.jar as a dependency, try to create an admin client

What did you expect to see?

no warnings or a warning that reports a reason for method emptyList() to be unavailable

What did you see instead?

java.lang.IllegalAccessError: class com.company.grpc.inventory.Command tried to access method 'com.google.protobuf.LazyStringArrayList com.google.protobuf.LazyStringArrayList.emptyList()' (com.company.grpc.inventory.Command and com.google.protobuf.LazyStringArrayList are in unnamed module of loader 'app’)

Anything else?

I also use dependencies:

1. JavaVersion = '17’
2. protobufJavaVersion = '3.22.3'
3. grpcJavaVersion = '1.54.2'

I think I have this problem because Pulsar uses protobuf version 3.19.6, which the emptyList() method is package private https://github.com/protocolbuffers/protobuf/blob/5cba162a5d93f8df786d828621019e03e50edb4f/java/core/src/main/java/com/google/protobuf/LazyStringArrayList.java#L70
They have exposed the method to the public since 3.22.0 protocolbuffers/protobuf@c658e27

When I use these dependencies, I don't have this problem:

1. protobufJavaVersion = '3.21.7'
2. grpcJavaVersion = '1.54.2'

Do you know about such a bug and in which release it will be fixed?

Are you willing to submit a PR?

• I'm willing to submit a PR!

[lhotari]

I don't think this is a bug. It's more lack of documentation. :)

It seems that protobuf and grpc codegen should be done using a codegen version of the oldest version of protobuf and grpc-java to support at runtime. Generated code is designed to be compatible with newer runtimes, but not necessarily compatible with older runtimes. This is explained in comment grpc/grpc-java#10202 (comment) .

Could you use 3.19.6 protoc / 1.55.3 protoc-gen-grpc-java to generate the Java classes?
Using a newer protobuf & grpc library at runtime should be fine.

[lhotari]

Actually protobuf's compatibility is more limited than what I mentioned in my previous comment.

The cross version runtime compatibility is documented here:
https://protobuf.dev/support/cross-version-runtime-guarantee/

Protobuf will not support mixing generated code and runtimes across major version boundaries. We will add “poison pills” where possible to detect these mismatches.

[lhotari]

@artsiomau The problem you have reported doesn't seem to be a problem with the Pulsar client. Do you agree?

I'd recommend using a compatible grpc & protobuf version. One way to find out is to look in poms at https://search.maven.org/artifact/io.grpc/grpc-protobuf for a specific version. For example in grpc 1.58.0 , the protobuf-java version is 3.24.0 . If you happen to have dependencies to an older protobuf version, it might be better to downgrade the grpc version to a compatible version. For example, grpc 1.57.2 uses protobuf-java 3.22.3

[davidtrobinson]

I'm having similar issues. We have a protobuf project which has been building with protoc 3.21.7 and also provides transitive dependencies for protobuf-java-util:3.21.7 and protobuf-java:3.21.7. There is a security vulnerability in protobuf-java-util:3.21.7 in its transitive dependencies.

I upgraded to 4.26.1 for all 3 libraries. The protobuf library we build worked fine in a project that doesn't use pulsar, but in the project that does use pulsar, I get: "tried to access method 'com.google.protobuf.LazyStringArrayList com.google.protobuf.LazyStringArrayList.emptyList()'" errors.

The problem is the pulsar-client-all jar is including that com.google.protobuf.LazyStringArrayList class.

So I cannot upgrade to the latest protobuf libraries and use pulsar client in the same project. I think this is a bug. I think pulsar should shade the classes from protobuf that it is including in its jar.

[lhotari]

The problem is the pulsar-client-all jar is including that com.google.protobuf.LazyStringArrayList class.

So I cannot upgrade to the latest protobuf libraries and use pulsar client in the same project. I think this is a bug. I think pulsar should shade the classes from protobuf that it is including in its jar.

@davidtrobinson You might be able to workaround the problem by depending on pulsar-client-original and pulsar-admin-original dependencies instead of pulsar-client-all, pulsar-client or pulsar-admin. pulsar-client-original and pulsar-admin-original are unshaded. Those work in most cases when used with a recent compatible Netty version.

[davidtrobinson]

Thank you for the tip. I am using spring-boot-starter-pulsar. It was including the pulsar-client-all.jar as a dependency. I tried excluding the pulsar-client-all jar and then adding the original jars like you said but I get a weird class cast exception:

Caused by: java.lang.LinkageError: ClassCastException: attempting to castjar:file:/Users/drobinson/.gradle/caches/modules-2/files-2.1/javax.ws.rs/javax.ws.rs-api/2.1/426a0862406536e690c7caa8bb6ed32191986fac/javax.ws.rs-api-2.1.jar!/javax/ws/rs/client/ClientBuilder.class to jar:file:/Users/drobinson/.gradle/caches/modules-2/files-2.1/javax.ws.rs/javax.ws.rs-api/2.1/426a0862406536e690c7caa8bb6ed32191986fac/javax.ws.rs-api-2.1.jar!/javax/ws/rs/client/ClientBuilder.class
at javax.ws.rs.client.ClientBuilder.newBuilder(ClientBuilder.java:105) ~[javax.ws.rs-api-2.1.jar:2.1]
at org.apache.pulsar.client.admin.internal.PulsarAdminImpl.(PulsarAdminImpl.java:136) ~[pulsar-client-admin-original-3.1.2.jar:3.1.2]
at org.apache.pulsar.client.admin.internal.PulsarAdminBuilderImpl.build(PulsarAdminBuilderImpl.java:44) ~[pulsar-client-admin-original-3.1.2.jar:3.1.2]
at org.springframework.pulsar.core.PulsarAdministration.createAdminClient(PulsarAdministration.java:112) ~[spring-pulsar-1.0.3.jar:1.0.3]

It's weird because the classes are identical. I'm giving up on this for now and I'll just use the older protobuf version. Hopefully a better solution comes along in the future.

[lhotari]

Similar reported issue is #22263, this is addressed by #23468. Now you can include the protobuf-java version that you want since Pulsar client no longer embeds protobuf-java classes.