"java.lang.VerifyError: Bad type on operand stack" error when using Stackdriver trace exporter

[sebright]

See #1184 (comment).

With opencensus-java 0.13.2 (or 0.14.0-SNAPSHOT) and google-cloud-trace 0.47.0-beta, the Stackdriver trace exporter gave this error:

May 21, 2018 6:24:06 AM io.opencensus.implcore.trace.export.SpanExporterImpl$Worker onBatchExport
WARNING: Exception thrown by the service export io.opencensus.exporter.trace.stackdriver.StackdriverTraceExporter
java.lang.VerifyError: Bad type on operand stack
Exception Details:
Location:
com/google/api/AnnotationsProto.registerAllExtensions(Lcom/google/protobuf/ExtensionRegistryLite;)V @4: invokevirtual
Reason:
Type 'com/google/protobuf/GeneratedMessage$GeneratedExtension' (current frame, stack[1]) is not assignable to 'com/google/protobuf/
Current Frame:
bci: @4
flags: { }
locals: { 'com/google/protobuf/ExtensionRegistryLite' }
stack: { 'com/google/protobuf/ExtensionRegistryLite', 'com/google/protobuf/GeneratedMessage$GeneratedExtension' }
Bytecode:
0x0000000: 2ab2 0003 b600 04b1

at com.google.devtools.cloudtrace.v2.TraceProto.<clinit>(TraceProto.java:198)
at com.google.devtools.cloudtrace.v2.Span$Attributes$AttributeMapDefaultEntryHolder.<clinit>(Span.java:443)
at com.google.devtools.cloudtrace.v2.Span$Attributes$Builder.internalGetMutableAttributeMap(Span.java:925)
at com.google.devtools.cloudtrace.v2.Span$Attributes$Builder.putAttributeMap(Span.java:1079)
at io.opencensus.exporter.trace.stackdriver.StackdriverV2ExporterHandler.toAttributesProto(StackdriverV2ExporterHandler.java:227)
at io.opencensus.exporter.trace.stackdriver.StackdriverV2ExporterHandler.generateSpan(StackdriverV2ExporterHandler.java:135)
at io.opencensus.exporter.trace.stackdriver.StackdriverV2ExporterHandler.export(StackdriverV2ExporterHandler.java:338)
at io.opencensus.implcore.trace.export.SpanExporterImpl$Worker.onBatchExport(SpanExporterImpl.java:138)
at io.opencensus.implcore.trace.export.SpanExporterImpl$Worker.run(SpanExporterImpl.java:187)
at java.lang.Thread.run(Thread.java:745)

/cc @johanhaleby @songy23 @bogdandrutu

[songy23]

Found a few issues on cloud client that look similar to this:
googleapis/google-cloud-java#1239
googleapis/google-cloud-java#1363
protocolbuffers/protobuf#2017

FWIW this could be related to incompatible protobuf versions. @johanhaleby do you have a dependency on protobuf-java? If so which version are you using?

EDIT: if you're using a protobuf-java version other than 3.5.1, please try update it to 3.5.1 first.

[johanhaleby]

From what I can see we are already using version 3.5.1. Running mvn dependency:tree gives me the following:

[INFO] +- com.google.cloud:google-cloud-trace:jar:0.47.0-beta:compile
[INFO] |  +- com.google.cloud:google-cloud-core:jar:1.29.0:compile
[INFO] |  |  +- com.google.http-client:google-http-client:jar:1.23.0:compile
[INFO] |  |  +- com.google.api:api-common:jar:1.5.0:compile
[INFO] |  |  +- com.google.api:gax:jar:1.25.0:compile
[INFO] |  |  |  \- org.threeten:threetenbp:jar:1.3.3:compile
[INFO] |  |  +- com.google.protobuf:protobuf-java-util:jar:3.5.1:compile
[INFO] |  |  |  \- com.google.code.gson:gson:jar:2.7:compile
[INFO] |  |  +- com.google.api.grpc:proto-google-common-protos:jar:1.11.0:compile
[INFO] |  |  \- com.google.api.grpc:proto-google-iam-v1:jar:0.12.0:compile
[INFO] |  +- com.google.cloud:google-cloud-core-grpc:jar:1.29.0:compile
[INFO] |  |  +- com.google.protobuf:protobuf-java:jar:3.5.1:compile
[INFO] |  |  +- io.grpc:grpc-protobuf:jar:1.10.1:compile
[INFO] |  |  |  \- io.grpc:grpc-protobuf-lite:jar:1.10.1:compile
[INFO] |  |  \- com.google.api:gax-grpc:jar:1.25.0:compile
[INFO] |  +- com.google.api.grpc:proto-google-cloud-trace-v1:jar:0.12.0:compile
[INFO] |  +- com.google.api.grpc:proto-google-cloud-trace-v2:jar:0.12.0:compile
[INFO] |  +- io.grpc:grpc-netty-shaded:jar:1.10.1:compile
[INFO] |  |  \- io.grpc:grpc-core:jar:1.10.1:compile (version selected from constraint [1.10.1,1.10.1])
[INFO] |  |     \- io.opencensus:opencensus-contrib-grpc-metrics:jar:0.11.0:compile
[INFO] |  +- io.grpc:grpc-stub:jar:1.10.1:compile
[INFO] |  \- io.grpc:grpc-auth:jar:1.10.1:compile

This is the only instance of protobuf-java I can find in our code base when running mvn dependency:tree. But we're also using Guava version 25.0 if that makes a difference?

[johanhaleby]

I found the issue, we were using an old version of libphonenumber that bundled a version of GRPC. Upgrading to the latest version solved the issue. Thanks a lot for all your support! Would not have managed to solve it without you.

[songy23]

That's great! Thanks for your cooperation and patience! This turned out to be very good experience to us too - we now have a better understanding in case we ran into similar issues in the future.