JVM Build Recipe Repository

• Introduction
• Layout
  • scm.yaml
    • Tag Mapping Rules
  • build.yaml
    • Environment Variables
    • Special Variables
    • Java Installations
    • Maven Installations
    • Gradle Installations
    • Custom Repository Handling
  • tool.yaml
  • Disabled Plugins
    • maven.yaml
    • gradle.yaml

Introduction

This repository contains information used to rebuild Java dependencies, including SCM information and other build related info for the JVM Build Service

Layout

The five subdirectories are used as follows:

• repository-info describes extra third-party repositories (currently only Maven) that may be used to build an artifact.

• build-info contains specific build instructions to tell the system how to build an artifact.

• scm-info tells the system where to find the sources for a particular artifact.

• build-tool-info contains release information for various build tools.

• disabled-plugins contains the globally disabled plugins for each build tool.

scm.yaml

While the system will attempt to determine source location from the pom it will fall back to using the jvm-build-data repository. It will look up information via GAV e.g. for com.fasterxml.jackson.core:jackson-databind:2.13.4.2 it will be searched for in the following location, from most specific to least specific:

scm-info/com/fasterxml/jackson/core/_artifact/jackson-databind/_version/2.13.4.2/scm.yaml (1)
scm-info/com/fasterxml/jackson/core/_artifact/jackson-databind/scm.yaml (2)
scm-info/com/fasterxml/jackson/core/scm.yaml (3)

1. This approach specifies the group-id, the artifact-id and the version. Note that the version matches based on 'less than', so older versions (e.g. 2.1) would still match, while newer versions would not.

2. These match based on the group-id and artifact-id. This approach is good for when a specific group-id is used in lots of different repositories.

3. These match based on the group-id. This is the most common layout and is used when we know the majority of artifacts with the group id come from a single repository.

An example of a full file is shown below:

type: "git" (1)
uri: "https://github.com/apache/commons-logging.git" (2)
tagMapping: (3)
  - pattern: 1\.1\.1  (4)
    tag: commons-logging-1.1.1
  - pattern: (\d+)\.(\d+) (5)
    tag: LOGGING_$1_$2
  - pattern: (\d+)\.(\d+)\.(\d+)
    tag: LOGGING_$1_$2_$3
legacyRepos: (6)
  - type: "git"
    uri: "https://github.com/javaee/metro-stax-ex.git"
path: "stax-ex" (7)

1. The type is optional, Git is the only type supported at present

2. The SCM URI that contains the source. If the URI contains a hash fragment (e.g. https://github.com/lz4/lz4-java.git#pureJava) then the lookup will be done without the hash fragment as per normal but the fragment will be used within the build.yaml to lookup alternate build information.

3. Tag mappings let you map tags to versions if the system cannot do it automatically. Mappings are tried in order until one is successful. If none are successful the default automatic approach is used.

4. This is an explicit mapping between version 1.1.1 and the tag commons-logging-1.1.1. This is useful if you have a small number of tags that don’t follow the normal convention.

5. This maps versions in the form of x.y to LOGGING_x_y. $n can be used to substitute the regex capture groups, with $0 being the full match.

6. Additional repositories to search. This can be useful if a project has moved home over time.

7. Some projects are not in the root of the repo. The path tells us the directory they are in.

legacyRepos may also be used if the original source has no build system. Instead, the source is forked and stored at https://github.com/jvm-build-service-code together an appropriate build file. An example of this is at https://github.com/jvm-build-service-code/cs-au-dk-dk.brics.automaton. The 1.11-8 release had no build file, so the project was forked and one added. This was then added to the SCM information at https://github.com/redhat-appstudio/jvm-build-data/blob/main/scm-info/dk/brics/_artifact/automaton/scm.yaml#L6.

Tag Mapping Rules

In most cases the tagMapping element is not needed. The system will attempt to figure out the tag mapping using the following heuristic. The tag mapping is only required if this heuristic fails to match:

1. Iterate over any provided tag mappings in order. If the version matches the pattern, and the resulting tag exists in the repo, then this tag is used.

2. Map exact matches to the tag (i.e. version 1.2.3.Final will match tag 1.2.3.Final).

3. If only one tag exists that contains the version then this tag is used (e.g. version 1.2.3 would match tag 1.2.3-RELEASE, as long as there is no other tag that contains the version. If a 1.2.3-PRERELEASE was present as well, then this would not match).

4. If multiple tags contain the version, but only one ends with the version, we map the tar that ends with the version (e.g. for version 1.2.3 if we have tags acme-1.2.3 and acme-1.2.3-rc we will map acme-1.2.3).

5. Finally, if there is exactly one tag that completely matches the numeric part of the version, then we will map this tag (e.g. the version 1.2.3.Final will map to the tag 1.2.3, it will not map to 1.2 or 2.3, as it must match the full numeric part)

build.yaml

It is also possible to tweak build parameters to get them to pass. This is done by adding a build.yaml file to the build data repository. For our databind example the file would go in one of the below locations. The structure is based upon the repository name i.e. the above would match https://github.com/FasterXML/jackson-databind. It will be searched from most specific to least specific:

build-info/github.com/FasterXML/jackson-databind/_version/2.13.4.2/build.yaml (1)
build-info/github.com/FasterXML/jackson-databind/build.yaml (2)

1. This would apply the build configuration to a specific version. Note that the version matches based on 'less than', so older versions (e.g. 2.1) would still match, while newer versions would not.

2. This would apply it to all of jackson-databind.

An example of a complete (although nonsensical) file is shown below:

additionalArgs: (1)
  - "-DskipDocs"
additionalDownloads: (2)
  - uri: https://github.com/mikefarah/yq/releases/download/v4.30.4/yq_linux_amd64 (3)
    sha256: 30459aa144a26125a1b22c62760f9b3872123233a5658934f7bd9fe714d7864d (4)
    type: executable (5)
    fileName: yq (6)
    binaryPath: only_for_tar/bin (7)
  - type: rpm (8)
    packageName: glibc-devel
additionalMemory: 4096 (9)
additionalBuilds: (10)
  pureJava:
    preBuildScript: |
        ./autogen.sh
        /bin/sh -c "$(rpm --eval %configure); $(rpm --eval %__make) $(rpm --eval %_smp_mflags)"
    additionalArgs:
      - "-Dlz4-pure-java=true"
allowedDifferences: (11)
  - \Q-:jbossws-common-4.0.0.Final.jar:class:org/jboss/ws/common/CalendarTest\E
alternativeArgs: (12)
  - "'set Global / baseVersionSuffix:=\"\"'"
  - "enableOptimizer"
enforceVersion: true (13)
javaVersion: 11 (14)
preBuildScript: | (15)
    ./autogen.sh
    /bin/sh -c "$(rpm --eval %configure); $(rpm --eval %__make) $(rpm --eval %_smp_mflags)"
postBuildScript: | (16)
    mvn -DnewVersion=$(params.PROJECT_VERSION) -DgenerateBackupPoms=false -DprocessAllModules=true -f libraries/pom.xml versions:set
runTests: false (17)
repositories: (18)
  - "all"
  - "jetbrains-kotlinx"
toolVersion: 3.8.8 (19)

1. Additional parameters to add to the build command line.

2. Additional downloads required for the build.

3. The URI to download from

4. The expected SHA.

5. The type, can be either executable, or tar.

6. The final file name, this will be added to $PATH. This is only for executable files.

7. The path to the directory inside the tar file that contains executables, this will be added to $PATH.

8. Additional RPMs to be installed prior to running the build.

9. If the system should allocate additional memory for the build.

10. This is used for additional builds that are alternate to the default. Each name (e.g. pureJava) should match to a hash fragment in scm.yaml. Complete recipe information for the build must be included as it is not combined with the parent information.

11. Deploy verification exclusion support for known differences to upstream if e.g. upstream jars were corrupted in some known way. This should be a regular expression.

12. A complete replacement for the build command line, this should not be used with 'additionalArgs' as it will replace them. This is mostly used in SBT builds.

13. If the tag contains build files that do not match the version include this to override the version.

14. This is used to enforce a particular Java version. See below for available Java versions.

15. A script to run before the build. This can do things like build native components that are required.

16. A script to run after the build. Useful if a build requires a secondary tool to run to complete the deployment.

17. Whether to run tests (default: false)

18. Additional repositories to include in the build named as per those listed here.

19. This is used to override and denote a particular tool version to use.

Environment Variables

Note that within the build pod (for instance, accessible by the preBuildScript) several variables are defined:

• PROJECT_VERSION which is set to the version of the artifact that is being built.

• MAVEN_HOME where the current version of Maven is installed to.

• GRADLE_HOME where the current version of Gradle is installed to. [optional]

• JAVA_HOME where the current version of Java is installed to.

• ENFORCE_VERSION if enforceVersion (see above) has been defined.

• CACHE_URL is defined to point to the JBS cache which handles all dependency requests and tracking.

• TOOL_VERSION is the version of the build tool being used.

Note that to use these within additionalArgs, preBuildScript or postBuildScript it is recommended to use the following pattern:

additionalArgs:
  - "-Dmaven.repository.mirror=$(params.CACHE_URL)"
postBuildScript: |
  mvn -DnewVersion=$(params.PROJECT_VERSION) -DgenerateBackupPoms=false -DprocessAllModules=true -f libraries/pom.xml versions:set

Special Variables

A few special variables are available (which could be used in e.g. preBuildScript, postBuildScript, or additionalArgs):

• $(workspaces.build-settings.path) which is the location of the tool configuration files.

• $(workspaces.source.path)/artifacts which is the location of where the artifacts are deployed to.

• $(workspaces.source.path)/source which is the location of the project source to build.

Java Installations

Currently, the following Java installations are available (in case a build requires multiple versions of a JDK):

• /usr/lib/java-1.7.0 (only available within a UBI7 builder image)

• /usr/lib/java-1.8.0

• /usr/lib/java-11

• /usr/lib/java-17

• /usr/lib/java-21

Maven Installations

Currently, the following Maven installations are available:

• /opt/maven/3.8.8

• /opt/maven/3.9.5

These are defined by the jvm-build-service-builder-images

Gradle Installations

Currently, the following Gradle installations are available:

• /opt/gradle/8.6

• /opt/gradle/8.4

• /opt/gradle/8.3

• /opt/gradle/8.0.2

• /opt/gradle/7.6.3

• /opt/gradle/7.5.1

• /opt/gradle/7.4.2

• /opt/gradle/7.3.3

• /opt/gradle/6.9.4

• /opt/gradle/6.4.1

• /opt/gradle/6.2.2

• /opt/gradle/5.6.4

• /opt/gradle/5.3.1

• /opt/gradle/4.10.3

These are defined by the jvm-build-service-builder-images

Custom Repository Handling

By default, for Gradle builds, the system will remove Maven repositories and inject a reference to the cache. It is possible to modify this by setting:

export JBS_DISABLE_CACHE=true
export JBS_DISABLE_GRADLE_REPOSITORIES=false

The former completely disables the repository plugin (which routes requests to the cache) while the latter only disables the repository removal. This is useful in cases where the project’s configuration clashes with our plugin and/or it provides its own method to inject a replacement Maven repository (of which Kotlin does):

additionalArgs:
- "-Pkotlin.build.isObsoleteJdkOverrideEnabled=true"
- "-Pmaven.repository.mirror=$(params.CACHE_URL)"

tool.yaml

This file contains release information about a build tool.

- version: "8.4" (1)
  maxJdkVersion: 20 (2)
  minJdkVersion: 8 (3)
  releaseDate: "2023-10-04" (4)

1. The version string for a particular release.

2. The maximum JDK version number supported by the release.

3. The minimum JDK version number supported by the release.

4. The date of the release as specified in java.text.SimpleDateFormat.

Disabled Plugins

The disabled-plugins directory contains .yaml files with the basename of the file being that of the build tool, e.g., maven.yaml for Maven and gradle.yaml for Gradle.

maven.yaml

This file contains the globally disabled plugins (for Maven) which can be overridden on a per-build basis via the disabledPlugins key in build.yaml.

disabledPlugins: (1)
  - "org.glassfish.copyright:glassfish-copyright-maven-plugin" (2)

1. The disabledPlugins key (and only this key) must be present followed by the plugin list.

2. The list of disabled plugins are strings in tool-specific format, e.g., "<groupdId>:<artifactId>" for Maven.

gradle.yaml

This file contains the globally disabled plugins (for Gradle) which can be overridden on a per-build basis via the disabledPlugins key in build.yaml.

disabledPlugins: (1)
  - "kotlin.gradle.targets.js" (2)

1. The disabledPlugins key (and only this key) must be present followed by the plugin list.

2. The list of disabled plugins are strings in tool-specific format, e.g., `"<packageNameOfTask>" for Gradle.