Investigate spring-cloud-gcp's sudden appearance of Linkage Errors

[suztomo]

https://gist.github.com/suztomo/d40a4adc75882c6b4615ce8f3caf14c0

GoogleCloudPlatform/spring-cloud-gcp#371 is the PR that bumped the enforcer rule to 1.5.7.

CC: @meltsufin

[suztomo]

ab54aa7#diff-8ec093dd0aeb6f8f90d1a302ed66b338523762974f0a8bf33afcd79bce6532a0 fixed the handling of the entry point JARs for BOM projects (DEPENDENCY_MANAGEMENT) . It's part of 1.5.7 release.

Spring-cloud-gcp applies the enforcer rule to a BOM project (DEPENDENCY_MANAGEMENT) and it uses reportOnlyReachable=true:

<LinkageCheckerRule
  implementation="com.google.cloud.tools.dependencies.enforcer.LinkageCheckerRule">
  <dependencySection>DEPENDENCY_MANAGEMENT</dependencySection>
  <reportOnlyReachable>true</reportOnlyReachable>
</LinkageCheckerRule>

This meant that Linkage Checker was not working before 1.5.7 and now it reports errors.

Next step: are they correct linkage errors?

[suztomo]

Optional Dependencies

Most of them are caused by optional dependencies. Example:

2021-03-16T09:41:23.6210512Z Class org.springframework.data.rest.webmvc.spi.BackendIdConverter is not found;
2021-03-16T09:41:23.6213863Z   referenced by 2 class files
2021-03-16T09:41:23.6220536Z     com.google.cloud.spring.autoconfigure.spanner.SpannerKeyIdConverter (com.google.cloud:spring-cloud-gcp-autoconfigure:2.0.2-SNAPSHOT)
2021-03-16T09:41:23.6224874Z     com.google.cloud.spring.autoconfigure.datastore.DatastoreKeyIdConverter (com.google.cloud:spring-cloud-gcp-autoconfigure:2.0.2-SNAPSHOT)
2021-03-16T09:41:23.6227194Z   Cause:
2021-03-16T09:41:23.6230057Z     The valid symbol is in org.springframework.data:spring-data-rest-webmvc:jar:3.4.5 at com.google.cloud:spring-cloud-gcp-autoconfigure:2.0.2-SNAPSHOT (compile) / org.springframework.data:spring-data-rest-webmvc:3.4.5 (compile, optional) but it was not selected because the path contains an optional dependency
2021-03-16T09:41:23.6233651Z Class org.springframework.integration.file.remote.handler.FileTransferringMessageHandler is not found;
2021-03-16T09:41:23.6235715Z   referenced by 1 class file
2021-03-16T09:41:23.6241680Z     com.google.cloud.spring.storage.integration.outbound.GcsMessageHandler (com.google.cloud:spring-cloud-gcp-storage:2.0.2-SNAPSHOT)
2021-03-16T09:41:23.6244820Z   Cause:
2021-03-16T09:41:23.6256127Z     The valid symbol is in org.springframework.integration:spring-integration-file:jar:5.4.4 at com.google.cloud:spring-cloud-gcp-storage:2.0.2-SNAPSHOT (compile) / org.springframework.integration:spring-integration-file:5.4.4 (compile, optional) but it was not selected because the path contains an optional dependency

netty

Netty optional dependencies will be suppressed by #1979

Example

2021-03-16T09:41:24.4931081Z Class io.netty.internal.tcnative.SSLContext is not found;
2021-03-16T09:41:24.4931897Z   referenced by 6 class files
2021-03-16T09:41:24.4933147Z     io.netty.handler.ssl.OpenSslSessionContext (io.netty:netty-handler:4.1.52.Final)
2021-03-16T09:41:24.4935077Z     io.netty.handler.ssl.ReferenceCountedOpenSslContext (io.netty:netty-handler:4.1.52.Final)
2021-03-16T09:41:24.4937378Z     io.netty.handler.ssl.ReferenceCountedOpenSslClientContext (io.netty:netty-handler:4.1.52.Final)
2021-03-16T09:41:24.4940414Z     io.netty.handler.ssl.OpenSslSessionStats (io.netty:netty-handler:4.1.52.Final)
2021-03-16T09:41:24.4942481Z     io.netty.handler.ssl.ReferenceCountedOpenSslServerContext (io.netty:netty-handler:4.1.52.Final)
2021-03-16T09:41:24.4945608Z     io.netty.handler.ssl.OpenSslServerSessionContext (io.netty:netty-handler:4.1.52.Final)
2021-03-16T09:41:24.4946697Z   Cause:
2021-03-16T09:41:24.4949945Z     The valid symbol is in io.netty:netty-tcnative:jar:2.0.34.Final at io.grpc:grpc-netty:1.36.0 (compile) / io.netty:netty-codec-http2:4.1.52.Final (compile) / io.netty:netty-handler:4.1.52.Final (compile) / io.netty:netty-tcnative:2.0.34.Final (compile, optional) but it was not selected because the path contains an optional dependency

Unknown

Kotlin

2021-03-16T09:41:24.3129450Z Class kotlin.Unit is not found;
2021-03-16T09:41:24.3130000Z   referenced by 2 class files
2021-03-16T09:41:24.3131322Z     org.springframework.core.MethodParameter (org.springframework:spring-core:5.3.4)
2021-03-16T09:41:24.3133112Z     org.springframework.core.CoroutinesUtils (org.springframework:spring-core:5.3.4)
2021-03-16T09:41:24.3134083Z   Cause:
2021-03-16T09:41:24.3134415Z     Unknown

Groovy

2021-03-16T09:41:24.3694551Z Class groovy.lang.GroovySystem is not found;
2021-03-16T09:41:24.3695194Z   referenced by 2 class files
2021-03-16T09:41:24.3697178Z     org.springframework.context.support.GenericGroovyApplicationContext (org.springframework:spring-context:5.3.4)
2021-03-16T09:41:24.3700371Z     org.springframework.beans.factory.groovy.GroovyBeanDefinitionReader (org.springframework:spring-beans:5.3.4)
2021-03-16T09:41:24.3702012Z   Cause:
2021-03-16T09:41:24.3702345Z     Unknown
2021-03-16T09:41:24.3702978Z Class groovy.lang.MetaClassRegistry is not found;
2021-03-16T09:41:24.3703804Z   referenced by 2 class files
2021-03-16T09:41:24.3705822Z     org.springframework.context.support.GenericGroovyApplicationContext (org.springframework:spring-context:5.3.4)
2021-03-16T09:41:24.3708990Z     org.springframework.beans.factory.groovy.GroovyBeanDefinitionReader (org.springframework:spring-beans:5.3.4)
2021-03-16T09:41:24.3710619Z   Cause:
2021-03-16T09:41:24.3710953Z     Unknown

netty-shaded

The netty-shaded classes have different package name than the netty rule above

2021-03-16T09:41:24.1556880Z Class com.jcraft.jzlib.JZlib is not found;
2021-03-16T09:41:24.1557600Z   referenced by 4 class files
2021-03-16T09:41:24.1560683Z     io.grpc.netty.shaded.io.netty.handler.codec.spdy.SpdyHeaderBlockJZlibEncoder (io.grpc:grpc-netty-shaded:1.36.0)
2021-03-16T09:41:24.1564311Z     io.grpc.netty.shaded.io.netty.handler.codec.compression.JZlibEncoder (io.grpc:grpc-netty-shaded:1.36.0)
2021-03-16T09:41:24.1567340Z     io.grpc.netty.shaded.io.netty.handler.codec.compression.ZlibUtil (io.grpc:grpc-netty-shaded:1.36.0)
2021-03-16T09:41:24.1570600Z     io.grpc.netty.shaded.io.netty.handler.codec.compression.JZlibDecoder (io.grpc:grpc-netty-shaded:1.36.0)
2021-03-16T09:41:24.1572312Z   Cause:
2021-03-16T09:41:24.1572787Z     Unknown

reactivex

Spring framework seems to have lots of invalid references without declaration of (even optional) dependencies.

2021-03-16T09:41:24.3300569Z Class io.reactivex.BackpressureStrategy is not found;
2021-03-16T09:41:24.3301337Z   referenced by 1 class file
2021-03-16T09:41:24.3302772Z     org.springframework.core.ReactiveAdapterRegistry (org.springframework:spring-core:5.3.4)
2021-03-16T09:41:24.3303910Z   Cause:
2021-03-16T09:41:24.3304244Z     Unknown

Conflicts

2021-03-16T09:41:24.5262948Z (ch.qos.logback:logback-core:1.2.3) Interface ch.qos.logback.core.status.StatusManager's method add(ch.qos.logback.core.status.StatusListener) is expected to return void but instead returns boolean;
2021-03-16T09:41:24.5264615Z   referenced by 2 class files
2021-03-16T09:41:24.5265872Z     ch.qos.logback.classic.jmx.JMXConfigurator (ch.qos.logback:logback-classic:1.1.3)
2021-03-16T09:41:24.5267880Z     ch.qos.logback.classic.util.StatusListenerConfigHelper (ch.qos.logback:logback-classic:1.1.3)
2021-03-16T09:41:24.5269576Z   Cause:
2021-03-16T09:41:24.5271717Z     Dependency conflict: ch.qos.logback:logback-core:1.2.3 does not define Interface ch.qos.logback.core.status.StatusManager's method add(ch.qos.logback.core.status.StatusListener) but ch.qos.logback:logback-core:1.1.3 defines it.
2021-03-16T09:41:24.5274687Z       selected: com.google.cloud:spring-cloud-gcp-logging:2.0.2-SNAPSHOT (compile) / com.google.cloud:google-cloud-logging-logback:0.120.2-alpha (compile) / ch.qos.logback:logback-core:1.2.3 (compile)
2021-03-16T09:41:24.5277748Z       unselected: com.google.cloud:spring-cloud-gcp-logging:2.0.2-SNAPSHOT (compile) / ch.qos.logback.contrib:logback-json-classic:0.1.5 (compile) / ch.qos.logback:logback-classic:1.1.3 (compile) / ch.qos.logback:logback-core:1.1.3 (compile)
2021-03-16T09:41:24.5282376Z     Dependency conflict: ch.qos.logback:logback-core:1.2.3 does not define Interface ch.qos.logback.core.status.StatusManager's method add(ch.qos.logback.core.status.StatusListener) but ch.qos.logback:logback-core:1.1.3 defines it.
2021-03-16T09:41:24.5285361Z       selected: com.google.cloud:spring-cloud-gcp-logging:2.0.2-SNAPSHOT (compile) / com.google.cloud:google-cloud-logging-logback:0.120.2-alpha (compile) / ch.qos.logback:logback-core:1.2.3 (compile)
2021-03-16T09:41:24.5288039Z       unselected: com.google.cloud:spring-cloud-gcp-logging:2.0.2-SNAPSHOT (compile) / ch.qos.logback.contrib:logback-json-classic:0.1.5 (compile) / ch.qos.logback:logback-classic:1.1.3 (compile) / ch.qos.logback:logback-core:1.1.3 (compile)

java9+ java compiler option issue

2021-03-16T09:41:24.5333268Z java.nio.ByteBuffer's method mark() is expected to return java.nio.ByteBuffer but instead returns java.nio.Buffer;
2021-03-16T09:41:24.5334191Z   referenced by 1 class file
2021-03-16T09:41:24.5334951Z     zipkin2.internal.ReadBuffer (io.zipkin.zipkin2:zipkin:2.23.2)
2021-03-16T09:41:24.5335618Z   Cause:
2021-03-16T09:41:24.5335966Z     Unknown
2021-03-16T09:41:24.5337046Z java.nio.ByteBuffer's method reset() is expected to return java.nio.ByteBuffer but instead returns java.nio.Buffer;
2021-03-16T09:41:24.5337971Z   referenced by 1 class file
2021-03-16T09:41:24.5338716Z     zipkin2.internal.ReadBuffer (io.zipkin.zipkin2:zipkin:2.23.2)
2021-03-16T09:41:24.5339392Z   Cause:
2021-03-16T09:41:24.5339726Z     Unknown

This seems similar to protocolbuffers/protobuf#7827

[meltsufin]

@suztomo Do you have any updates on this? Are there any specific true positives we should be fixing? We would like to get our linkage check green.

[suztomo]

I'm afraid that I haven't come up with a good solution to resolve this.

I've added some of the exclusions rule for netty classes (https://github.com/GoogleCloudPlatform/cloud-opensource-java/pull/1979/files; 2 weeks ago) but this is just few among ~260 (almost all false positive) errors.

I'll update you by early next week.

[elharo]

Let's remember we're trying to make this self-service. If something is preventing clients from diagnosing and fixing the problems themselves, then that's what we need to address. It's not feasible for us to investigate each linkage error individually.

[meltsufin]

Can we have an option to ignore issues causes by dependencies being declared as optional?

[suztomo]

Unfortunately Spring-framework's pom.xml files do not have optional dependency information. Spring's pom.xml does not declare optional dependencies any more, even when the dependencies are present in their build.gradle. In past, I asked Spring team to add the optional dependency information to pom.xml (spring-projects/spring-framework#23234) but didn't succeed to persuade.

[meltsufin]

It looks like many linkage errors in our case come from not taking optional dependencies into consideration, even though we are declaring them in our pom.xml. See this file in particular: https://github.com/GoogleCloudPlatform/spring-cloud-gcp/blob/main/spring-cloud-gcp-autoconfigure/pom.xml.

[suztomo]

Linkage Checker can take filter file: https://github.com/GoogleCloudPlatform/cloud-opensource-java/wiki/Linkage-Checker-Enforcer-Rule#configuration

Idea:

• Filter linkage errors by classes that exist in optional dependency. (Can this be expressed as the filter xml files? Or Linkage Checker's logic)
• Special handling of ConditionOnClass annotation

I'll create a PR to filter by package and classes.