Linkage Monitor

Linkage Monitor works as a presubmit check in a GitHub repository to prevent open source libraries in the Google Cloud Java orbit (GAX, google-http-java-client, gRPC, etc.) from releasing versions that would introduce new linkage errors.

A linkage error is a condition in which a "class has some dependency on another class; however, the latter class has incompatibly changed after the compilation of the former class." ^1 The inter-class dependency can be through a class literal, a field access, or a method invocation. Linkage errors encountered at runtime manifest as a subclass of LinkageError class, such as NoSuchMethodError, NoClassDefFoundError, and NoSuchFieldError.

For example, an argument could have been added to a public method between the version of a class file supplied at compile time and the version available in the runtime class path. A method could have been removed from a class, or a superclass might have been made final.

The linkage monitor presubmit check merges the code in the repository and the pull request with the latest release of the GCP Libraries BOM (com.google.cloud:libraries-bom). It then finds all linkage errors in the modified BOM with the change in the pull request. If any of these linkage errors are not present in the unmodified BOM, the presubmit check fails.

In more detail, the tool performs the following steps:

1. Run mvn clean install in the checked-out directory to install Maven artifacts of the project.
2. Read the latest version of com.google.cloud:libraries-bom from Maven Central.
3. Build a class path containing the dependency elements of the BOM's dependencyManagement section and their non-test-scoped transitive dependencies.
4. Check this class path for linkage errors and save the result.
5. Create a modified version of the pom.xml from com.google.cloud:libraries-bom in which each dependency element in the dependencyManagement section is replaced by the artifacts installed in Step 1. To get the list artifacts, it scans the current working directory for "pom.xml" files. Also for repositories that do not have pom.xml, then it reads linkage-monitor-artifacts.txt.
6. Find linkage errors in the modified BOM using the same procedure as in Step 3 and Step 4. Save the result.
7. Compare the linkage errors from Step 4 and Step 6. If the result of Step 6 contains any errors not found in Step 3, return a non-zero exit code. Otherwise return zero. Kokoro recognizes non-zero as a failure and zero as success.

Building a Class Path from a BOM

Given a BOM’s dependencyManagement section, the linkage checker generates a class path (a list of JAR file) through a dependency tree.

It builds a dependency tree where the root of this tree is the BOM itself. The children of the root are the dependency elements in the dependencyManagement section of the BOM's pom.xml file. It recursively provides each node in the tree with child nodes, which are the dependency elements in the dependencies section of the Maven artifact of the node.

Optional and provided dependencies are included. Test scoped dependencies are not included.

It then flattens this tree into a list of dependencies, starting at the root and moving through the tree in level order. If a dependency is encountered that has already been added to the class path, it is skipped. Thus if there are multiple versions of a dependency in the tree only the first version encountered is included in the class path.

Checking a Class Path for Linkage Errors

In brief, the linkage checker checks all references from one class to another in the class path to make sure they can be resolved. First, the linkage checker scans each .class file in each .jar file in the class path and makes a list of all symbols defined by the jars in the class path. If a symbol is defined more than once, for example because the same class appears in two different jars, only the first one encountered is used.

Then the linkage checker reads the class path a second time, this time looking for references to symbols. For each reference, it verifies that the referenced symbol exists in the class path and is accessible from the referring class.

Any symbol that is not found or is not accessible is reported as a linkage error.

Special Cases

The linkage checker does not determine whether the code along that path is actually invoked, and does not check whether the references are guarded by other checks. Several known linkage errors that do not occur in actual fact -- e.g.because they are surrounded by an if (classIsAvailable) check or included in a try-catch block that handles the corresponding NoClassDefFoundError -- are special cased and not reported.

linkage-monitor-artifacts.txt

Linkage Monitor by default searches for pom.xml files to get list of the artifacts that are generated by a repository. However certain repositories (such as gax-java) use Gradle and have no pom.xml files. Therefore Linkage Monitor has a functionality to let the repositories tell the list of artifacts. It reads linkage-monitor-artifacts.txt in the root of a repository.

Each line of the file consists of Maven coordinates with 3 elements (<groupId>:<artifactId>:<version>) or an empty line. For example, gax-java repository produces linkage-monitor-artifacts.txt with the following content:

suztomo-macbookpro44% cat linkage-monitor-artifacts.txt

com.google.api:gax:1.62.1-SNAPSHOT
com.google.api:gax-grpc:1.62.1-SNAPSHOT
com.google.api:gax-httpjson:0.79.1-SNAPSHOT

Here is the Gradle task that produces the file:

task createLinkageMonitorArtifactList {
  doLast {
    def httpJsonVersion = project(":gax-httpjson").version

    new File(projectDir, "linkage-monitor-artifacts.txt").text = """
com.google.api:gax:$version
com.google.api:gax-grpc:$version
com.google.api:gax-httpjson:$httpJsonVersion
"""
  }
}