fix #239 - add property javaVersion to support GraalVM 19.3+

[muhlba91]

Before this PR

Issue #239: GraalVM 19.3+ supports Java 11 and has a different download URL and layout causing the plugin to fail when using Java 11 or GraalVM 19.3+.

After this PR

Support for GraalVM 19.3+ and Java 11 including backwards compatibility.

Possible downsides?

Still havent' got the chance to test it on a project.

[palantirtech]

Thanks for your interest in palantir/gradle-graal, @muhlba91! Before we can accept your pull request, you need to sign our contributor license agreement - just visit https://cla.palantir.com/ and follow the instructions. Once you sign, I'll automatically update this pull request.

[muhlba91]

Some test on Windows still fail like before:

• GradleGraalPluginIntegrationSpec fail because of the path separators (/ and \) - I don't know what about the Linux enviornment failures for CircleCI

@cosminpolifronie maybe you can help here?

[cosminpolifronie]

I will look on it tonight.

[cosminpolifronie]

On my machine a couple more tests are failing:

• com.palantir.gradle.graal.GradleGraalEndToEndSpec > test default version nativeImage FAILED

The system cannot find the path specified.
Error: Unknown argument: com.palantir.test.Main

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':nativeImage'.
> Process 'command 'cmd.exe'' finished with non-zero exit value 1

This means that native-image cannot find the class that's defining main(). The definition of the class in the test seems to be all right, so I don't know what is causing it. I remember I actually read about this problem in an issue open in the graalvm repository, but I cannot find it and I have no idea how to solve it.

I've tried replacing the separators to fix the tests in GradleGraalPluginIntegrationSpec but it makes no difference, the tests are still failing.

About the other errors: I've been trying to understand but I don't even know where to start solving them.

Most of them fail this way:

> Task :nativeImage FAILED
Caching disabled for task ':nativeImage' because:
  Build cache is disabled
Task ':nativeImage' is not up-to-date because:
  No history is available.
Starting process 'command 'cmd.exe''. Working directory: D:\Works\Repos\github\cosminpolifronie\gradle-graal\build\nebulatest\com.palantir.gradle.graal.GradleGraalEndToEndSpec\allows-specifying-additional-properties Command: cmd.exe /E:ON /V:ON /c "D:\Works\Repos\github\cosminpolifronie\gradle-graal\build\nebulatest\com.palantir.gradle.graal.GradleGraalEndToEndSpec\allows-specifying-additional-properties\build\tmp\com.palantir.graal\native-image.cmd"
Successfully started process 'command 'cmd.exe''
Warning: the '=' character in program arguments is not fully supported.
Make sure that command line arguments using it are wrapped in double quotes.
Example:
"--vm.Dfoo=bar"

:nativeImage (Thread[Execution worker for ':' Thread 2,5,main]) completed. Took 0.882 secs.
5 actionable tasks: 5 executed

Gradle Standard Error:
The system cannot find the path specified.
Error: Unknown arguments: hello-world, com.palantir.test.Main

FAILURE: Build failed with an exception.

All the tests that are failing on my machine:

• com.palantir.gradle.graal.GradleGraalEndToEndSpec > test default version nativeImage FAILED
• com.palantir.gradle.graal.GradleGraalEndToEndSpec > allows specifying additional properties FAILED
• com.palantir.gradle.graal.GradleGraalEndToEndSpec > can build shared libraries on default version FAILED
• com.palantir.gradle.graal.GradleGraalPluginIntegrationSpec > allows specifying different GA graal version (windows) FAILED
• com.palantir.gradle.graal.GradleGraalPluginIntegrationSpec > allows specifying GA graal version Java 8 19.3+ (windows) FAILED
• com.palantir.gradle.graal.GradleGraalPluginIntegrationSpec > allows specifying GA graal version Java 11 19.3+ (windows) FAILED

This is way above my level of understanding of this platform. I don't even know how to properly debug these, as these tests automatically jump to the last instruction in them (most of them fail with an exception well before that), I cannot use breakpoints properly.

[frozenice]

@muhlba91 I made you a PR that fixes the tests on Windows for me.

I am not sure about the failing tests from @cosminpolifronie, as all the tests pass on my machine. @cosminpolifronie are you able to use native-image manually? Do the tests from the palantir/gradle-graal develop pass for you?
It sounds kinda like your Graal can't compile to a native-image. Try running gradle with --info to see the native-image.cmd output.

[cosminpolifronie]

I've tried building @frozenice PR on another Windows machine, without success. This is the output of the build:

.\gradlew.bat --profile --parallel --stacktrace --continue check
Starting a Gradle Daemon (subsequent builds will be faster)

> Task :compileJava UP-TO-DATE
> Task :compileGroovy NO-SOURCE
> Task :compileJavaCircleFinalizer UP-TO-DATE
> Task :pluginDescriptors UP-TO-DATE
> Task :processResources UP-TO-DATE
> Task :classes UP-TO-DATE
> Task :checkstyleMain UP-TO-DATE
> Task :checkstyleMainCircleFinalizer UP-TO-DATE
> Task :compileTestJava NO-SOURCE
> Task :compileTestGroovy UP-TO-DATE
> Task :compileTestJavaCircleFinalizer UP-TO-DATE
> Task :processTestResources NO-SOURCE
> Task :testClasses UP-TO-DATE
> Task :checkstyleTest NO-SOURCE
> Task :checkstyleTestCircleFinalizer UP-TO-DATE
> Task :spotlessJava UP-TO-DATE
> Task :spotlessJavaCheck UP-TO-DATE
> Task :spotlessCheck UP-TO-DATE
> Task :checkJUnitDependencies
> Task :pluginUnderTestMetadata UP-TO-DATE

> Task :test
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.codehaus.groovy.vmplugin.v7.Java7$1 (file:/C:/Users/U2546/.gradle/wrapper/dists/gradle-5.6.4-bin/bxirm19lnfz6nurbatndyydux/gradle-5.6.4/lib/groovy-all-1.3-2.5.4.jar) to constructor java.lang.invoke.MethodHandles$Lookup(java.lang.Class,int)
WARNING: Please consider reporting this to the maintainers of org.codehaus.groovy.vmplugin.v7.Java7$1
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release

com.palantir.gradle.graal.GradleGraalEndToEndSpec > test default version nativeImage FAILED
    org.gradle.api.GradleException at GradleGraalEndToEndSpec.groovy:52
        Caused by: org.gradle.internal.exceptions.LocationAwareException
            Caused by: org.gradle.api.tasks.TaskExecutionException
                Caused by: org.gradle.process.internal.ExecException

com.palantir.gradle.graal.GradleGraalEndToEndSpec > allows specifying additional properties FAILED
    org.spockframework.runtime.ConditionNotSatisfiedError at GradleGraalEndToEndSpec.groovy:197

com.palantir.gradle.graal.GradleGraalEndToEndSpec > can build shared libraries on default version FAILED
    org.gradle.api.GradleException at GradleGraalEndToEndSpec.groovy:216
        Caused by: org.gradle.internal.exceptions.LocationAwareException
            Caused by: org.gradle.api.tasks.TaskExecutionException
                Caused by: org.gradle.process.internal.ExecException

29 tests completed, 3 failed, 6 skipped

> Task :test FAILED
> Task :validateTaskProperties UP-TO-DATE
> Task :verifyLocks UP-TO-DATE

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':test'.
> There were failing tests. See the report at: file:///C:/Users/U2546/Desktop/gradle-graal/build/reports/tests/test/index.html

* Try:
Run with --info or --debug option to get more log output. Run with --scan to get full insights.

For the com.palantir.gradle.graal.GradleGraalEndToEndSpec > allows specifying additional properties test, I have the following error on task native-image:

> Task :nativeImage
    Caching disabled for task ':nativeImage' because:
      Build cache is disabled
    Task ':nativeImage' is not up-to-date because:
      No history is available.
    Starting process 'command 'cmd.exe''. Working directory: C:\Users\U2546\Desktop\gradle-graal\build\nebulatest\com.palantir.gradle.graal.GradleGraalEndToEndSpec\allows-specifying-additional-properties Command: cmd.exe /E:ON /V:ON /c "C:\Users\U2546\Desktop\gradle-graal\build\nebulatest\com.palantir.gradle.graal.GradleGraalEndToEndSpec\allows-specifying-additional-properties\build\tmp\com.palantir.graal\native-image.cmd"
    Successfully started process 'command 'cmd.exe''
    [hello-world:14864]    classlist:   3,410.64 ms

    > Task :nativeImage FAILED
    :nativeImage (Thread[Connection worker,5,main]) completed. Took 5.259 secs.
    3 actionable tasks: 3 executed

    Gradle Standard Error:
    The system cannot find the path specified.
    Fatal error: java.lang.UnsupportedClassVersionError: com/palantir/test/Main has been compiled by a more recent version of the Java Runtime (class file version 55.0), this version of the Java Runtime only recognizes class file versions up to 52.0

For the com.palantir.gradle.graal.GradleGraalEndToEndSpec > can build shared libraries on default version and com.palantir.gradle.graal.GradleGraalEndToEndSpec > test default version nativeImage I have the following error (it's probably the same as above):

org.gradle.api.GradleException: Build aborted because of an internal error.
        at nebula.test.functional.internal.DefaultExecutionResult.rethrowFailure(DefaultExecutionResult.groovy:97)
        at nebula.test.IntegrationSpec.runTasksSuccessfully(IntegrationSpec.groovy:149)
        at com.palantir.gradle.graal.GradleGraalEndToEndSpec.can build shared libraries on default version(GradleGraalEndToEndSpec.groovy:216)

        Caused by:
        org.gradle.internal.exceptions.LocationAwareException: Execution failed for task ':sharedLibrary'.

            Caused by:
            org.gradle.api.tasks.TaskExecutionException: Execution failed for task ':sharedLibrary'.

                Caused by:
                org.gradle.process.internal.ExecException: Process 'command 'cmd.exe'' finished with non-zero exit value 1

So my file has been compiled by Java 11 (version 55.0) but it is trying to run it using Java 8 (version 52.0). I have no idea why. Both my JAVA_HOME and GRAALVM_HOME point to the graalvm folder, java from Path variable points to the bin directory of the graalvm folder.

I cannot actually build a native-image using a simple hello-world example:

public class MainClass {
    public static void main(String[] args) {
        System.out.println("Hello World!");
    }
}

In standard PowerShell Core it says this:

native-image.cmd MainClass
[mainclass:11100]    classlist:   2,056.84 ms
[mainclass:11100]        (cap):     688.29 ms
[mainclass:11100]        setup:     994.63 ms
Error: Unable to compile C-ABI query code. Make sure GCC toolchain is installed on your system.
Error: Use -H:+ReportExceptionStackTraces to print stacktrace of underlying exception
Error: Image build request failed with exit status 1

Running native-image.cmd in Developer Command Prompt for VS 2017 (version 15.9.18) it says this:

native-image.cmd MainClass -H:+ReportExceptionStackTraces
[mainclass:12660]    classlist:   2,231.25 ms
[mainclass:12660]        (cap):   1,308.56 ms
[mainclass:12660]        setup:   1,686.70 ms
Error: Error compiling query code (in C:\Users\U2546\AppData\Local\Temp\SVM-10542413364113237373\JNIHeaderDirectives.cpp). Compiler command  CL -IC:\graalvm\graalvm-ce-java11-19.3.0\include\win32 C:\Users\U2546\AppData\Local\Temp\SVM-10542413364113237373\JNIHeaderDirectives.cpp /FeC:\Users\U2546\AppData\Local\Temp\SVM-10542413364113237373\JNIHeaderDirectives.exe output included error: [Microsoft (R) C/C++ Optimizing Compiler Version 19.16.27034 for x86, Copyright (C) Microsoft Corporation.  All rights reserved., ]
com.oracle.svm.core.util.UserError$UserException: Error compiling query code (in C:\Users\U2546\AppData\Local\Temp\SVM-10542413364113237373\JNIHeaderDirectives.cpp). Compiler command  CL -IC:\graalvm\graalvm-ce-java11-19.3.0\include\win32 C:\Users\U2546\AppData\Local\Temp\SVM-10542413364113237373\JNIHeaderDirectives.cpp /FeC:\Users\U2546\AppData\Local\Temp\SVM-10542413364113237373\JNIHeaderDirectives.exe output included error: [Microsoft (R) C/C++ Optimizing Compiler Version 19.16.27034 for x86, Copyright (C) Microsoft Corporation.  All rights reserved., ]
        at com.oracle.svm.core.util.UserError.abort(UserError.java:114)
        at com.oracle.svm.hosted.c.NativeLibraries.reportErrors(NativeLibraries.java:214)
        at com.oracle.svm.hosted.NativeImageGenerator.processNativeLibraryImports(NativeImageGenerator.java:1518)
        at com.oracle.svm.hosted.NativeImageGenerator.setupNativeLibraries(NativeImageGenerator.java:1006)
        at com.oracle.svm.hosted.NativeImageGenerator.setupNativeImage(NativeImageGenerator.java:835)
        at com.oracle.svm.hosted.NativeImageGenerator.doRun(NativeImageGenerator.java:528)
        at com.oracle.svm.hosted.NativeImageGenerator.lambda$run$0(NativeImageGenerator.java:445)
        at java.base/java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(ForkJoinTask.java:1407)
        at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
        at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
        at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
        at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
        at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:177)
Error: Image build request failed with exit status 1

[muhlba91]

@frozenice thank you. I have merged your change.

@cosminpolifronie finally, I got some tests for Java 11 working and figured out some errors and problems.

First, some tests will always fail (obviously) - if you use Java 8 for running the tests, all Java 11 tests will fail and vice-versa... How can we deal with this properly?

Secondly, on Windows you need VS 2017+ for GraalVM 19.3+ (see oracle/graal#1852). However, this creates one problem: depending on the VS version and, to make it even worse, also the VS version to be used (Community, Enterpise, ...) change the path for the vcvars64.bat.
Are there any good ideas how to deal with those? I don't wanna add more and more parameters as it becomes quite tidious to setup the plugin. I am thinking more of making the entire vcvars64.bat file path configurable and leave it - by default - for 2019 Community (it's the only one you can download from Microsoft directly). What do you think?

Additionally, I figured that all command line arguments must be enclosed within "" - is this also working with GraalVM < 19.3 and Java 8?
As an example: "C:\Users\Q395001\.gradle\caches\com.palantir.graal\19.3.0\11\graalvm-ce-java11-19.3.0\bin\native-image.cmd" "-cp" "C:\dev\devops\projects\gradle-graal\build\nebulatest\com.palantir.gradle.graal.GradleGraalEndToEndSpec\test-version-19-3-0-nativeImage-Java-11\build\libs\test-version-19-3-0-nativeImage-Java-11.jar" "-H:Path=C:\dev\devops\projects\gradle-graal\build\nebulatest\com.palantir.gradle.graal.GradleGraalEndToEndSpec\test-version-19-3-0-nativeImage-Java-11\build\graal" "-H:Name=hello-world" "com.palantir.test.Main"

[cosminpolifronie]

That's the thing, I only have Java 11 installed, why is it running under Java 8 mode?

I have managed to get native-image working standalone using your suggestion to look on oracle/graal#1852 (I was missing the C++ package for ARM and ARM64, why is that needed though?). Thank you!

Tests still fail for me, even with calling vcvars64.bat before building.

[muhlba91]

The tests which are testing the default version (19.2.0) and default Java version (8) do not set the property javaVersion to 11, meaning as default it‘s still Java 8 (backwards compatibility).
Hence, the tests fail if you use Java 11 and only those stating „Java 11“ should pass.
This also seems to be a problem for the CI I assume?

As for the tests, did you check out my latest commit where I set the vcvars64 path to the VS 2019 Community one? I am no specialist in Windows Shells but maybe it matters whether the Gradle‘s executor sets the environment or you do it beforehand?
As for my tests, the shared library test for Graal 19.3.0 and Java 11 passes now, the native image one somehow doesn‘t whereas I will try having a look at it the next days (before my vacation starts) with a clean checkout and Gradle state.

[cosminpolifronie]

Oh, correct, you are right, the default version uses Java 8.

As for the tests, did you check out my latest commit where I set the vcvars64 path to the VS 2019 Community one?

Yes. The call operation most probably fails as I am trying to build on a work machine that has VS 2017/2019 Professional installed. Should I do a pull request with the other possible variants of the path (Professional and Enterprise instead of Community) or will you add those variants?

[muhlba91]

Yes, this would be really awesome if you could add it, permitting you have time for it. Also it would add a test on a different VS version than I have (VS 2019 Community). ;)

[stale]

This PR has been automatically marked as stale because it has not been touched in the last 14 days. If you'd like to keep it open, please leave a comment or add the 'long-lived' label, otherwise it'll be closed in 7 days.

[carterkozak]

I don't think this is stale, happy holidays everyone :-)

[cosminpolifronie]

It's not, I will get to it soon after the holidays. Happy holidays! :)

[stale]

This PR has been automatically marked as stale because it has not been touched in the last 14 days. If you'd like to keep it open, please leave a comment or add the 'long-lived' label, otherwise it'll be closed in 7 days.

[carterkozak]

I've applied the label to keep this open.

[muhlba91]

@cosminpolifronie - any updates from your side? :)

[cosminpolifronie]

Sadly I am quite overworked these days, as I need to finish my BSc final project until February. Maybe I will be able to push this during the weekend.

[muhlba91]

Thank you for the update. 👍

Please let me know how you proceed, otherwise I‘ll have some time next week - most probably. :)

[cosminpolifronie]

I didn't even think about other OS'es, good catch. Well, I guess this is done, right?

[muhlba91]

Yes, if it looks good from your side now too, I guess we can proceed and merge. :)

[cosminpolifronie]

I think some variables and methods in GraalExtension.java should be renamed to reflect that they work on Windows only.

[cosminpolifronie]

Other than that, I think we are ready for merge :).

[muhlba91]

I renamed all VS/Windows only parameters and method to contain the wording windows.

[cosminpolifronie]

All good.

@iamdanfox could you review this, please?

[machaval]

Any update on this?

[muhlba91]

@cosminpolifronie any way to get this sorted and merged, finally?

[carterkozak]

Hi friends, I'm really sorry for the delay and slow reviews on this project. I can try to make time for a full review this weekend. Thanks for the contributions!

[limewxr]

Any update? Thanks! 😄

[yogesh-dhamija]

Is there any movement on this?

To avoid such a thing in the future, could maybe provide configuration for the full download URL rather than just the base?

[muhlba91]

@cosminpolifronie @iamdanfox @carterkozak any updates on this? it would be really awesome to get this merged. otherwise, it renders the plugin slowly useless with more and more people updating.

[carterkozak]

What timing, I'd just started digging into this :-)

[carterkozak]

I've released https://github.com/palantir/gradle-graal/releases/tag/0.6.1 including this change

[saboya]

@carterkozak I believe something is wrong with the CircleCI step, preventing the new release from being published.

[schlosna]

Looks like publish build failed with Error applying workspace layer for job ca397999-0f68-4f74-9516-2b4d043988d4: Concurrent upstream jobs persisted the same file(s)

[dansanduleac]

@carterkozak this is causing the duplicate workspace files, we should just delete this.

[dansanduleac]

I'd also be in favor of either not doing check-11 (we don't do that anywhere else!), or merging it into with compile-11, such that this branch of jobs doesn't persist extra stuff to the workspace, nor require compiling with java 11 twice (if we don't persist, both compile-11 and check-11 would have to compile from scratch)

[dansanduleac]

also we don't need to be saving this cache twice (we're already doing it in check).
Not to mention I don't believe this never changes, so we should probably checksum some files into this cache key...

[carterkozak]

0.7.0 has published successfully.