Add Gradle Toolchain Support

[jamesward]

I'm using:

id("org.graalvm.buildtools.native") version "0.9.16"

And have a Gradle toolchain configured:

java {
	toolchain {
		languageVersion.set(JavaLanguageVersion.of(17))
		vendor.set(JvmVendorSpec.GRAAL_VM)
	}
}

And I get:

./gradlew nativeRun

FAILURE: Build failed with an exception.

* What went wrong:
Could not determine the dependencies of task ':nativeCompile'.
> Failed to calculate the value of task ':compileJava' property 'javaCompiler'.
   > No compatible toolchains found for request filter: {languageVersion=17, vendor=GRAAL_VM, implementation=vendor-specific} (auto-detect true, auto-download true)

It'd be awesome if the Gradle Native plugin worked together with Gradle Toolchains.

[graemerocher]

We would like to, but we need the Gradle team's help. See gradle/gradle#18028

[sschuberth]

We would like to, but we need the Gradle team's help. See gradle/gradle#18028

Note that according to @melix's comment "The PR, in itself, is irrelevant though" now that Gradle 7.6 added support for Java Toolchain downloads from arbitrary repositories, also see this comment of mine.

[sschuberth]

In conjunction with https://github.com/gradle/foojay-toolchains this now basically works as outlined e.g. at oss-review-toolkit/ort@main...graalvm-upgrade.

[jamesward]

Thanks @sschuberth but I couldn't get that working. Test project:
https://github.com/jamesward/hello-graalvm

For me:

$ ./gradlew nativeRun

Invalid Java installation found at '/home/jw/.gradle/jdks/graalvm-ce-java8-linux-amd64-21/graalvm-ce-java8-21.3.1' (Auto-provisioned by Gradle). It will be re-checked in the next build. This might have performance impact if it keeps failing. Run the 'javaToolchains' task for more details.
Invalid Java installation found at '/home/jw/.gradle/jdks/graalvm-ce-java17-linux-amd64-22/graalvm-ce-java17-22.3.0' (Auto-provisioned by Gradle). It will be re-checked in the next build. This might have performance impact if it keeps failing. Run the 'javaToolchains' task for more details.
Invalid Java installation found at '/home/jw/.gradle/jdks/graalvm-ce-java19-linux-amd64-22/graalvm-ce-java19-22.3.0' (Auto-provisioned by Gradle). It will be re-checked in the next build. This might have performance impact if it keeps failing. Run the 'javaToolchains' task for more details.

FAILURE: Build failed with an exception.

* What went wrong:
Could not determine the dependencies of task ':nativeCompile'.
> Failed to calculate the value of task ':compileJava' property 'javaCompiler'.
   > Unable to download toolchain matching the requirements ({languageVersion=17, vendor=GRAAL_VM, implementation=vendor-specific}) from 'https://api.foojay.io/disco/v3.0/ids/e663ebbd4ea908acb9135b3b41fdcade/redirect'.
      > Provisioned toolchain '/home/jw/.gradle/jdks/graalvm-ce-java17-linux-amd64-22/graalvm-ce-java17-22.3.0' could not be probed.

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 2s

Probably I'm using it wrong. Any help would be great!

[sschuberth]

@jamesward which version of Gradle are you using? I see you're using Gradle 7.6. I'd recommend to try with 8.0 RC 1 to be on the safer side, but even there is still a bug if the distribution archive contains duplicate files, which results in provisioning to fail.

Edit: Also, just delete all of /home/jw/.gradle/jdks between tries, as Gradle is currently bad at recovering from previously failed provisionings.

[jamesward]

Tried again with 8.0-rc-1 and cleared the jdks and got:

Execution failed for task ':compileJava'.
> Error while evaluating property 'javaCompiler' of task ':compileJava'.
   > Failed to calculate the value of task ':compileJava' property 'javaCompiler'.
      > Unable to download toolchain matching the requirements ({languageVersion=17, vendor=GRAAL_VM, implementation=vendor-specific}) from 'https://api.foojay.io/disco/v3.0/ids/e663ebbd4ea908acb9135b3b41fdcade/redirect'.
         > Entry graalvm-ce-java17-22.3.0/lib/libtrufflenfi.so is a duplicate but no duplicate handling strategy has been set. Please refer to https://docs.gradle.org/8.0-rc-1/dsl/org.gradle.api.tasks.Copy.html#org.gradle.api.tasks.Copy:duplicatesStrategy for details.

[sschuberth]

Yes, for that issue you need my patch to Gradle. But even that is not enough yet: As Gradle's Copy-task cannot handle symlinks, some build file logic is required to fix that up. I've updated my example branch accordingly.

[jamesward]

Oh cool. Thanks for all your work on this!

[sschuberth]

FYI, with my fix in Gradle 8.1, I'm able to use this code to bootstrap GraalVM via the foojay-resolver plugin:

tasks.named<BuildNativeImageTask>("nativeCompile").configure {
    // Gradle's "Copy" task cannot handle symbolic links, see https://github.com/gradle/gradle/issues/3982. That is why
    // links contained in the GraalVM distribution archive get broken during provisioning and are replaced by empty
    // files. Address this by recreating the links in the toolchain directory.
    val toolchainDir = options.get().javaLauncher.get().executablePath.asFile.parentFile.run {
        if (name == "bin") parentFile else this
    }

    val toolchainFiles = toolchainDir.walkTopDown().filter { it.isFile }
    val emptyFiles = toolchainFiles.filter { it.length() == 0L }

    // Find empty toolchain files that are named like other toolchain files and assume these should have been links.
    val links = toolchainFiles.mapNotNull { file ->
        emptyFiles.singleOrNull { it != file && it.name == file.name }?.let {
            file to it
        }
    }

    // Fix up symbolic links.
    links.forEach { (target, link) ->
        logger.quiet("Fixing up '$link' to link to '$target'.")

        if (link.delete()) {
            Files.createSymbolicLink(link.toPath(), target.toPath())
        } else {
            logger.warn("Unable to delete '$link'.")
        }
    }
}

[XhstormR]

According to this PR oracle/graal#5995 , Native Image is now shipped as part of the GraalVM JDK and thus no longer needs to be installed via gu install native-image.

[fniephaus]

@melix wjat is the current status of the toolchain support in Gradle? Are there any plans to support "JDK features" such as JavaFX or Native Image that would help automatically pick the right JDK for the user?

GraalVM has become easier to set up and use in the last couple of months, not sure this issue is still relevant?

[melix]

The issue is still relevant to me. We still have to disable toolchain support by default because of the limitations of the Gradle APIs. There is no progress made on that, as per a conversation I had with @ljacomet at Devoxx.

[jamesward]

Given the inclusion of native-image in the GraalVM JDK, what is needed to make this work now?

[morki]

The only missing piece is gradle/gradle#25264
I think after merge of this one, this plugin can enable toolchains by default.

[melix]

I don't think that's the only thing missing. As far as I can tell there's no way for Gradle to make the difference between a regular Oracle JDK and an Oracle GraalVM JDK, (nor can it make the difference between another vendor's GraalVM), because it can only "look" a the jvm.vendor property which is "Oracle Corporation" in both cases.

[sschuberth]

because it can only "look" a the jvm.vendor property which is "Oracle Corporation" in both cases.

Exactly. Also see my gradle/gradle#25521 in this context.