Investigate Avro linkage errors in java-bigquery. If they are safe, add them to suppress list #2136

suztomo · 2021-07-20T18:49:47Z

Jul 20, 2021 6:34:47 PM com.google.cloud.tools.dependencies.linkagemonitor.LinkageMonitor run
SEVERE: Newly introduced problems:
Class org.joda.time.LocalDate is not found
  referenced from org.apache.avro.data.TimeConversions (org.apache.avro:avro:1.8.2)
  referenced from org.codehaus.jackson.map.ext.JodaDeserializers (org.codehaus.jackson:jackson-mapper-asl:1.9.13)
  referenced from org.codehaus.jackson.map.ext.JodaSerializers (org.codehaus.jackson:jackson-mapper-asl:1.9.13)
Class org.joda.time.Days is not found
  referenced from org.apache.avro.data.TimeConversions (org.apache.avro:avro:1.8.2)
Class org.joda.time.LocalTime$Property is not found
  referenced from org.apache.avro.data.TimeConversions (org.apache.avro:avro:1.8.2)
Class org.joda.time.LocalTime is not found
  referenced from org.apache.avro.data.TimeConversions (org.apache.avro:avro:1.8.2)
Class org.joda.time.DateTime is not found
  referenced from org.apache.avro.data.TimeConversions (org.apache.avro:avro:1.8.2)
  referenced from org.codehaus.jackson.map.ext.JodaDeserializers (org.codehaus.jackson:jackson-mapper-asl:1.9.13)
  referenced from org.codehaus.jackson.map.ext.JodaSerializers (org.codehaus.jackson:jackson-mapper-asl:1.9.13)
Class org.joda.time.DateTimeZone is not found
  referenced from org.apache.avro.data.TimeConversions (org.apache.avro:avro:1.8.2)
  referenced from org.codehaus.jackson.map.ext.JodaDeserializers (org.codehaus.jackson:jackson-mapper-asl:1.9.13)
Class org.joda.time.DateMidnight is not found
  referenced from org.codehaus.jackson.map.ext.JodaDeserializers (org.codehaus.jackson:jackson-mapper-asl:1.9.13)
  referenced from org.codehaus.jackson.map.ext.JodaSerializers (org.codehaus.jackson:jackson-mapper-asl:1.9.13)
Class org.joda.time.format.DateTimeFormatter is not found
  referenced from org.codehaus.jackson.map.ext.JodaDeserializers (org.codehaus.jackson:jackson-mapper-asl:1.9.13)
  referenced from org.codehaus.jackson.map.ext.JodaSerializers (org.codehaus.jackson:jackson-mapper-asl:1.9.13)
Class org.joda.time.format.ISODateTimeFormat is not found
  referenced from org.codehaus.jackson.map.ext.JodaDeserializers (org.codehaus.jackson:jackson-mapper-asl:1.9.13)
  referenced from org.codehaus.jackson.map.ext.JodaSerializers (org.codehaus.jackson:jackson-mapper-asl:1.9.13)
Class org.joda.time.LocalDateTime is not found
  referenced from org.codehaus.jackson.map.ext.JodaDeserializers (org.codehaus.jackson:jackson-mapper-asl:1.9.13)
  referenced from org.codehaus.jackson.map.ext.JodaSerializers (org.codehaus.jackson:jackson-mapper-asl:1.9.13)
Class org.joda.time.ReadablePeriod is not found
  referenced from org.codehaus.jackson.map.ext.JodaDeserializers (org.codehaus.jackson:jackson-mapper-asl:1.9.13)
Class org.joda.time.Period is not found
  referenced from org.codehaus.jackson.map.ext.JodaDeserializers (org.codehaus.jackson:jackson-mapper-asl:1.9.13)
  referenced from org.codehaus.jackson.map.ext.JodaSerializers (org.codehaus.jackson:jackson-mapper-asl:1.9.13)
Class org.joda.time.ReadableDateTime is not found
  referenced from org.codehaus.jackson.map.ext.JodaDeserializers (org.codehaus.jackson:jackson-mapper-asl:1.9.13)
Class org.joda.time.ReadableInstant is not found
  referenced from org.codehaus.jackson.map.ext.JodaDeserializers (org.codehaus.jackson:jackson-mapper-asl:1.9.13)
Class org.joda.time.DateMidnight$Property is not found
  referenced from org.codehaus.jackson.map.ext.JodaSerializers (org.codehaus.jackson:jackson-mapper-asl:1.9.13)
Class org.joda.time.LocalDate$Property is not found
  referenced from org.codehaus.jackson.map.ext.JodaSerializers (org.codehaus.jackson:jackson-mapper-asl:1.9.13)
Class org.joda.time.LocalDateTime$Property is not found
  referenced from org.codehaus.jackson.map.ext.JodaSerializers (org.codehaus.jackson:jackson-mapper-asl:1.9.13)

org.apache.avro:avro:1.8.2 is at:
  com.google.cloud:google-cloud-bigquery:1.137.1-SNAPSHOT (compile) / org.apache.avro:avro:1.8.2 (compile)
org.codehaus.jackson:jackson-mapper-asl:1.9.13 is at:
  com.google.cloud:google-cloud-bigquery:1.137.1-SNAPSHOT (compile) / org.codehaus.jackson:jackson-mapper-asl:1.9.13 (compile)

https://github.com/googleapis/java-bigquery/pull/1374/checks?check_run_id=3117079174

The text was updated successfully, but these errors were encountered:

suztomo · 2021-07-21T14:50:18Z

This gives the same errors.

  @Test
  public void testAvroDependency() throws Exception {
    Artifact avro =
        new DefaultArtifact("org.apache.avro:avro:1.8.2");
    ClassPathResult classPathResult =
        new ClassPathBuilder()
            .resolve(ImmutableList.of(avro), false, DependencyMediation.MAVEN);

    System.out.println(classPathResult);

    LinkageChecker linkageChecker = LinkageChecker.create(classPathResult.getClassPath());

    ImmutableSet<LinkageProblem> linkageProblems = linkageChecker.findLinkageProblems();
    Truth.assertThat(linkageProblems).isEmpty();
  }

Doesn't avro declare joda-time dependency? It's optional.

    <dependency>
      <groupId>joda-time</groupId>
      <artifactId>joda-time</artifactId>
      <version>2.7</version>
      <scope>compile</scope>
      <optional>true</optional>
    </dependency>

Linkage Monitor sets full=true to traverse optional dependencies. Why joda-time not included in the list?

    ClassPathResult classPathResult =
        (new ClassPathBuilder())
            .resolve(snapshotManagedDependencies, true, DependencyMediation.MAVEN);

In fact, flipping the flag from false to true in the unit test fixes the error.

Investigate why Linkage Monitor doesn't add joda-time there.

suztomo · 2021-07-21T16:45:38Z

@stephaniewang526 It turned out joda-time is explicitly excluded in the pom.xml (after the flatten plugin; see below). Is this expected?

suztomo-macbookpro44% vim ~/.m2/repository/com/google/cloud/google-cloud-bigquery/1.135.4-SNAPSHOT/google-cloud-bigquery-1.135.4-SNAPSHOT.pom
...
    <dependency>
      <groupId>org.apache.avro</groupId>
      <artifactId>avro</artifactId>
      <version>1.8.2</version>
      <scope>compile</scope>
      <exclusions>
        <exclusion>
          <artifactId>jackson-core-asl</artifactId>
          <groupId>org.codehaus.jackson</groupId>
        </exclusion>
        <exclusion>
          <artifactId>jackson-mapper-asl</artifactId>
          <groupId>org.codehaus.jackson</groupId>
        </exclusion>
        <exclusion>
          <artifactId>paranamer</artifactId>
          <groupId>com.thoughtworks.paranamer</groupId>
        </exclusion>
        <exclusion>
          <artifactId>snappy-java</artifactId>
          <groupId>org.xerial.snappy</groupId>
        </exclusion>
        <exclusion>
          <artifactId>commons-compress</artifactId>
          <groupId>org.apache.commons</groupId>
        </exclusion>
        <exclusion>
          <artifactId>xz</artifactId>
          <groupId>org.tukaani</groupId>
        </exclusion>
        <exclusion>
          <artifactId>joda-time</artifactId>
          <groupId>joda-time</groupId>
        </exclusion>
        <exclusion>

suztomo · 2021-07-21T16:52:16Z

Dependency tree of google-cloud-bigquery (commit 02cfa971b805a49a17c497e96d624283b8775ccc):

suztomo-macbookpro44% mvn dependency:tree
[INFO] Scanning for projects...
[INFO] Inspecting build with total of 1 modules...
[INFO] Installing Nexus Staging features:
[INFO]   ... total of 1 executions of maven-deploy-plugin replaced with nexus-staging-maven-plugin
[INFO] 
[INFO] ---------------< com.google.cloud:google-cloud-bigquery >---------------
[INFO] Building BigQuery 1.135.4-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] 
[INFO] --- maven-dependency-plugin:3.1.2:tree (default-cli) @ google-cloud-bigquery ---
[INFO] com.google.cloud:google-cloud-bigquery:jar:1.135.4-SNAPSHOT
[INFO] +- com.google.cloud:google-cloud-core:jar:1.95.4:compile
[INFO] |  +- com.google.protobuf:protobuf-java-util:jar:3.17.3:compile
[INFO] |  +- com.google.api.grpc:proto-google-common-protos:jar:2.3.2:compile
[INFO] |  +- com.google.api.grpc:proto-google-iam-v1:jar:1.0.14:compile
[INFO] |  +- com.google.auth:google-auth-library-credentials:jar:0.26.0:compile
[INFO] |  +- com.google.http-client:google-http-client-gson:jar:1.39.2:compile
[INFO] |  \- com.google.protobuf:protobuf-java:jar:3.17.3:compile
[INFO] +- com.google.cloud:google-cloud-core-http:jar:1.95.4:compile
[INFO] |  +- com.google.api-client:google-api-client:jar:1.32.1:compile
[INFO] |  +- com.google.http-client:google-http-client-appengine:jar:1.39.2:compile
[INFO] |  +- com.google.api:gax-httpjson:jar:0.83.0:compile
[INFO] |  +- io.opencensus:opencensus-api:jar:0.28.0:compile
[INFO] |  \- io.opencensus:opencensus-contrib-http-util:jar:0.28.0:compile
[INFO] +- com.google.http-client:google-http-client-jackson2:jar:1.39.2:compile
[INFO] |  \- com.fasterxml.jackson.core:jackson-core:jar:2.12.3:compile
[INFO] +- com.google.cloud:google-cloud-datacatalog:jar:1.4.2:test
[INFO] |  +- io.grpc:grpc-api:jar:1.39.0:test
[INFO] |  +- io.grpc:grpc-context:jar:1.39.0:compile
[INFO] |  +- io.grpc:grpc-stub:jar:1.39.0:test
[INFO] |  +- io.grpc:grpc-protobuf:jar:1.39.0:test
[INFO] |  +- io.grpc:grpc-protobuf-lite:jar:1.39.0:test
[INFO] |  +- javax.annotation:javax.annotation-api:jar:1.3.2:compile
[INFO] |  +- com.google.api.grpc:proto-google-cloud-datacatalog-v1beta1:jar:0.41.2:test
[INFO] |  +- com.google.guava:failureaccess:jar:1.0.1:compile
[INFO] |  +- com.google.guava:listenablefuture:jar:9999.0-empty-to-avoid-conflict-with-guava:compile
[INFO] |  +- com.google.j2objc:j2objc-annotations:jar:1.3:compile
[INFO] |  +- org.apache.httpcomponents:httpclient:jar:4.5.13:compile
[INFO] |  +- commons-logging:commons-logging:jar:1.2:test
[INFO] |  +- commons-codec:commons-codec:jar:1.15:test
[INFO] |  +- org.apache.httpcomponents:httpcore:jar:4.4.14:compile
[INFO] |  +- com.google.code.gson:gson:jar:2.8.7:compile
[INFO] |  +- com.google.api:gax-grpc:jar:1.66.0:test
[INFO] |  +- io.grpc:grpc-auth:jar:1.39.0:test
[INFO] |  +- io.grpc:grpc-netty-shaded:jar:1.39.0:test
[INFO] |  +- io.perfmark:perfmark-api:jar:0.23.0:test
[INFO] |  +- io.grpc:grpc-core:jar:1.39.0:test
[INFO] |  +- com.google.android:annotations:jar:4.1.1.4:test
[INFO] |  +- org.codehaus.mojo:animal-sniffer-annotations:jar:1.20:test
[INFO] |  +- io.grpc:grpc-alts:jar:1.39.0:test
[INFO] |  +- io.grpc:grpc-grpclb:jar:1.39.0:test
[INFO] |  \- org.conscrypt:conscrypt-openjdk-uber:jar:2.5.1:test
[INFO] +- com.google.api.grpc:proto-google-cloud-datacatalog-v1:jar:1.4.2:test
[INFO] +- com.google.cloud:google-cloud-storage:jar:1.117.1:test
[INFO] |  +- com.google.oauth-client:google-oauth-client:jar:1.31.5:compile
[INFO] |  +- com.google.http-client:google-http-client-apache-v2:jar:1.39.2:compile
[INFO] |  \- com.google.apis:google-api-services-storage:jar:v1-rev20210127-1.31.5:test
[INFO] +- com.google.auto.value:auto-value-annotations:jar:1.8.1:provided
[INFO] +- com.google.http-client:google-http-client:jar:1.39.2:compile
[INFO] +- org.checkerframework:checker-compat-qual:jar:2.5.5:compile
[INFO] +- com.google.auth:google-auth-library-oauth2-http:jar:0.26.0:compile
[INFO] +- com.google.code.findbugs:jsr305:jar:3.0.2:compile
[INFO] +- com.google.apis:google-api-services-bigquery:jar:v2-rev20210617-1.32.1:compile
[INFO] +- com.google.cloud:google-cloud-core:jar:tests:1.95.4:test
[INFO] +- com.google.api:api-common:jar:1.10.4:compile
[INFO] +- com.google.guava:guava:jar:30.1.1-android:compile
[INFO] +- com.google.api:gax:jar:1.66.0:compile
[INFO] +- org.threeten:threetenbp:jar:1.5.1:compile
[INFO] +- org.apache.avro:avro:jar:1.8.2:compile
[INFO] |  +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile
[INFO] |  +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile
[INFO] |  +- com.thoughtworks.paranamer:paranamer:jar:2.7:compile
[INFO] |  +- org.xerial.snappy:snappy-java:jar:1.1.1.3:compile
[INFO] |  +- org.apache.commons:commons-compress:jar:1.8.1:compile
[INFO] |  +- org.tukaani:xz:jar:1.5:compile
[INFO] |  \- org.slf4j:slf4j-api:jar:1.7.7:compile
[INFO] +- com.google.errorprone:error_prone_annotations:jar:2.7.1:compile
[INFO] +- junit:junit:jar:4.13.2:test
[INFO] |  \- org.hamcrest:hamcrest-core:jar:1.3:test
[INFO] +- com.google.truth:truth:jar:1.1.3:test
[INFO] |  +- org.checkerframework:checker-qual:jar:3.13.0:test
[INFO] |  \- org.ow2.asm:asm:jar:9.1:test
[INFO] +- org.mockito:mockito-core:jar:2.28.2:test
[INFO] |  +- net.bytebuddy:byte-buddy:jar:1.9.10:test
[INFO] |  +- net.bytebuddy:byte-buddy-agent:jar:1.9.10:test
[INFO] |  \- org.objenesis:objenesis:jar:2.6:test
[INFO] \- org.assertj:assertj-core:jar:2.9.1:test

mvn dependency:tree shows joda-time is not there.

[INFO] +- org.apache.avro:avro:jar:1.8.2:compile
[INFO] |  +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile
[INFO] |  +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile
[INFO] |  +- com.thoughtworks.paranamer:paranamer:jar:2.7:compile
[INFO] |  +- org.xerial.snappy:snappy-java:jar:1.1.1.3:compile
[INFO] |  +- org.apache.commons:commons-compress:jar:1.8.1:compile
[INFO] |  +- org.tukaani:xz:jar:1.5:compile
[INFO] |  \- org.slf4j:slf4j-api:jar:1.7.7:compile

https://search.maven.org/artifact/org.apache.avro/avro/1.8.2/bundle shows it's optional.

    <dependency>
      <groupId>joda-time</groupId>
      <artifactId>joda-time</artifactId>
      <version>2.7</version>
      <scope>compile</scope>
      <optional>true</optional>
    </dependency>

Maybe the flutten plugin somehow excluded joda-time because it's optional

suztomo · 2021-07-30T19:42:47Z

Stephanie was thinking whether Apache Avro or Apache Arrow is better choice. Not taking action for now.

stephaniewang526 · 2022-05-02T15:14:48Z

We are no longer using Avro. We are using Arrow instead. Hence, we can close this ticket.

suztomo · 2022-05-02T15:20:38Z

Thank you for update.

suztomo mentioned this issue Jul 20, 2021

feat: add Connection interface googleapis/java-bigquery#1374

Merged

suztomo changed the title ~~Investigate Avro linkage errors. If they are safe, add them to suppress list~~ Investigate Avro linkage errors in java-bigquery. If they are safe, add them to suppress list Jul 21, 2021

meltsufin added the p3 label Jul 21, 2021

suztomo closed this as completed May 2, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Investigate Avro linkage errors in java-bigquery. If they are safe, add them to suppress list #2136

Investigate Avro linkage errors in java-bigquery. If they are safe, add them to suppress list #2136

suztomo commented Jul 20, 2021 •

edited

Loading

suztomo commented Jul 21, 2021 •

edited

Loading

suztomo commented Jul 21, 2021

suztomo commented Jul 21, 2021 •

edited

Loading

suztomo commented Jul 30, 2021

stephaniewang526 commented May 2, 2022

suztomo commented May 2, 2022

Investigate Avro linkage errors in java-bigquery. If they are safe, add them to suppress list #2136

Investigate Avro linkage errors in java-bigquery. If they are safe, add them to suppress list #2136

Comments

suztomo commented Jul 20, 2021 • edited Loading

suztomo commented Jul 21, 2021 • edited Loading

suztomo commented Jul 21, 2021

suztomo commented Jul 21, 2021 • edited Loading

suztomo commented Jul 30, 2021

stephaniewang526 commented May 2, 2022

suztomo commented May 2, 2022

suztomo commented Jul 20, 2021 •

edited

Loading

suztomo commented Jul 21, 2021 •

edited

Loading

suztomo commented Jul 21, 2021 •

edited

Loading