native-image: fails to create image when --language:js is included.

[renatoathaydes]

I have a small Java project that tries to run a JS script.
I coded it against the old ScriptManager API and it works when running with HotSpot/Nashorn.

It also seems to work fine when I compile and run using GraalVM like a normal JVM.

./gradlew clean jar
java -Dpolyglot.js.nashorn-compat=true -jar build/libs/rawhttp-scraper-1.0-SNAPSHOT.jar samples/requests/get-speakers samples/js/parseJson.js

Now, when I try to create a native image including the JS environment, it does not work:

native-image --language:js -jar build/libs/rawhttp-scraper-1.0-SNAPSHOT.jar

Error:

Build on Server(pid: 58707, port: 52955)
   classlist:     842.15 ms
       (cap):   1,184.43 ms
       setup:   1,778.32 ms
    analysis:   1,104.00 ms
fatal error: java.lang.NoClassDefFoundError: Could not initialize class rawhttp.core.RawHttpOptions
	at sun.misc.Unsafe.ensureClassInitialized(Native Method)
	at jdk.vm.ci.hotspot.HotSpotResolvedObjectTypeImpl.initialize(HotSpotResolvedObjectTypeImpl.java:382)
	at com.oracle.graal.pointsto.meta.AnalysisUniverse.createType(AnalysisUniverse.java:213)
	at com.oracle.graal.pointsto.meta.AnalysisUniverse.lookupAllowUnresolved(AnalysisUniverse.java:193)
	at com.oracle.graal.pointsto.meta.AnalysisUniverse.lookup(AnalysisUniverse.java:170)
	at com.oracle.graal.pointsto.meta.AnalysisField.<init>(AnalysisField.java:104)
	at com.oracle.graal.pointsto.meta.AnalysisUniverse.createField(AnalysisUniverse.java:357)
	at com.oracle.graal.pointsto.meta.AnalysisUniverse.lookupAllowUnresolved(AnalysisUniverse.java:345)
	at com.oracle.graal.pointsto.meta.AnalysisUniverse.lookup(AnalysisUniverse.java:312)
	at com.oracle.graal.pointsto.meta.AnalysisType.convertFields(AnalysisType.java:857)
	at com.oracle.graal.pointsto.meta.AnalysisType.convertInstanceFields(AnalysisType.java:850)
	at com.oracle.graal.pointsto.meta.AnalysisType.getInstanceFields(AnalysisType.java:841)
	at com.oracle.graal.pointsto.flow.context.object.AnalysisObject.getInstanceFieldTypeStore(AnalysisObject.java:209)
	at com.oracle.graal.pointsto.flow.context.object.AnalysisObject.getInstanceFieldFlow(AnalysisObject.java:199)
	at com.oracle.graal.pointsto.flow.LoadFieldTypeFlow$LoadInstanceFieldTypeFlow.onObservedUpdate(LoadFieldTypeFlow.java:159)
	at com.oracle.graal.pointsto.flow.TypeFlow.notifyObservers(TypeFlow.java:347)
	at com.oracle.graal.pointsto.flow.TypeFlow.update(TypeFlow.java:389)
	at com.oracle.graal.pointsto.BigBang$2.run(BigBang.java:500)
	at com.oracle.graal.pointsto.util.CompletionExecutor.lambda$execute$0(CompletionExecutor.java:174)
	at java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1402)
	at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
	at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
	at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
	at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
Error: Processing image build request failed

If I do not include the --language:js argument, it creates the image, but then it does not find the JS script engine.

Is there something I can do to work around the problem for now, or I will have to wait for a bug fix?

[cstancu]

Can you try adding the -report-unsupported-elements-at-runtime to the native-image command?

[renatoathaydes]

When I use --report-unsupported-elements-at-runtime with --language:js I get the exact same output as shown above.
Without using --language:js the result is also the same: the image is created without warnings, but the JS script engine is not found at runtime.

[cstancu]

Thanks! The trace is basically the same as in #494. That's a somewhat complicated corner case of the logic that deals with incomplete class paths. The issue appears when you load/store a field with a known type, but the field is declared by a class that has also some fields with unknown types, so our eager class resolution chockes. This error should be pushed to runtime when using -report-unsupported-elements-at-runtime.

[renatoathaydes]

The error says Could not initialize class rawhttp.core.RawHttpOptions which can be seen here.
This class is needed by the code I am using because I call new RawHttp(), which gets the default RawHttpOptions instance, which is a singleton (see this line).

This is a pretty simple case, I think. The jar I am using is a fat jar including the code in my simple project, plus the rawhttp-core.jar, which has no external dependencies.

I hope this helps you resolve the issue. Thanks!

[cstancu]

The error that you reported should be fixed. See #494 (comment) for reference.

You were probably hitting that issue due to some classpath configuration error since I am not able to reproduce it with your repo. I cloned your project and I was able to build the image even with GraalVM 1.0.0 rc5, i.e., without my fix, without hitting that issue and without using -report-unsupported-elements-at-runtime. However, running the image that I created I get:

$ ./rawhttp-scraper-1.0-SNAPSHOT samples/requests/google.com samples/js/parseJson.js
Exception in thread "main" java.lang.NullPointerException
	at com.athaydes.rawhttp.scraper.RawHttpScraper.run(RawHttpScraper.java:38)
	at com.athaydes.rawhttp.scraper.RawHttpScraper.runRequestFromFile(RawHttpScraper.java:74)
	at com.athaydes.rawhttp.scraper.RawHttpScraper.main(RawHttpScraper.java:53)
	at com.oracle.svm.core.JavaMainWrapper.run(JavaMainWrapper.java:177)

There might be some issue with how the nashorn compatibility mode is set.

[cstancu]

Your code tries to get the js engine. I think you should use graal.js, i.e., new ScriptEngineManager().getEngineByName("graal.js") according to this article. However, I still get the NPE, even with this change.

[olpaw]

ScriptEngineManager().getEngineByName("graal.js")

There should be no need to use "graal.js". "js" should be fine. I suspect this is caused by our analysis not being able to see through ScriptEngineManager().getEngineByName("js") here. This will need a fix on our side.

[renatoathaydes]

You were probably hitting that issue due to some classpath configuration error since I am not able to reproduce it with your repo. I cloned your project and I was able to build the image even with GraalVM 1.0.0 rc5

I don't think that's possible. As I mentioned before, I am using a fat jar so there's nothing else in the classpath, only a single jar.

Did you take jars separately to create the image?

In my project, ./gradlew jar creates a fat jar. The jar must be correct as it runs on any JVM, including GraalVM.

I also get a NPE if I don't use the --language:js option for native-image as, naturally, the JS engine is not found at runtime.

[cstancu]

Here is what I get with a fresh clone of your repo:

~/projects/gh-589/rawhttp-scraper$ native-image --version
GraalVM Version 1.0.0-rc5
~/projects/gh-589/rawhttp-scraper$ ./gradlew clean jar

BUILD SUCCESSFUL in 0s
3 actionable tasks: 3 executed
~/projects/gh-589/rawhttp-scraper$ native-image --language:js -jar build/libs/rawhttp-scraper-1.0-SNAPSHOT.jar
Build on Server(pid: 9444, port: 38979)
   classlist:   2,467.19 ms
       (cap):     704.03 ms
       setup:   1,473.87 ms
Detected unnecessary RecomputeFieldValue.ArrayBaseOffset com.oracle.svm.core.jdk.Target_java_nio_DirectByteBuffer.arrayBaseOffset substitution field for java.nio.DirectByteBuffer.arrayBaseOffset. The annotated field can be removed. This ArrayBaseOffset computation can be detected automatically. Use option -H:+UnsafeAutomaticSubstitutionsLogLevel=2 to print all automatically detected substitutions. 
  (typeflow):   2,724.15 ms
   (objects):   2,089.71 ms
  (features):     317.99 ms
    analysis:   5,277.46 ms
16 method(s) included for runtime compilation
    universe:     391.14 ms
     (parse):     626.01 ms
    (inline):     742.06 ms
   (compile):   2,715.60 ms
     compile:   4,496.92 ms
       image:     590.58 ms
       write:     142.51 ms
     [total]:  14,882.76 ms
~/projects/gh-589/rawhttp-scraper$ ./rawhttp-scraper-1.0-SNAPSHOT samples/requests/google.com samples/js/parseJson.js
Exception in thread "main" java.lang.NullPointerException
	at com.athaydes.rawhttp.scraper.RawHttpScraper.run(RawHttpScraper.java:39)
	at com.athaydes.rawhttp.scraper.RawHttpScraper.runRequestFromFile(RawHttpScraper.java:82)
	at com.athaydes.rawhttp.scraper.RawHttpScraper.main(RawHttpScraper.java:61)
	at com.oracle.svm.core.JavaMainWrapper.run(JavaMainWrapper.java:177)

[renatoathaydes]

You need to checkout branch 'works-on-jvm-but-not-graalvm'. I've been using the GraalVM Polyglot API now as that works, but the bug is happening with the ScriptManager API.

[cstancu]

I tried the works-on-jvm-but-not-graalvm branch. Same result:

~/projects/gh-589/rawhttp-scraper$ git checkout works-on-jvm-but-not-graalvm
Already on 'works-on-jvm-but-not-graalvm'
Your branch is up-to-date with 'origin/works-on-jvm-but-not-graalvm'.
~/projects/gh-589/rawhttp-scraper$ git status
On branch works-on-jvm-but-not-graalvm
Your branch is up-to-date with 'origin/works-on-jvm-but-not-graalvm'.
nothing to commit, working directory clean
~/projects/gh-589/rawhttp-scraper$ ./gradlew clean jar

BUILD SUCCESSFUL in 0s
3 actionable tasks: 3 executed
~/projects/gh-589/rawhttp-scraper$ native-image --language:js -jar build/libs/rawhttp-scraper-1.0-SNAPSHOT.jar
Build on Server(pid: 9444, port: 38979)
   classlist:   1,620.25 ms
       (cap):     517.14 ms
       setup:   1,095.88 ms
Detected unnecessary RecomputeFieldValue.ArrayBaseOffset com.oracle.svm.core.jdk.Target_java_nio_DirectByteBuffer.arrayBaseOffset substitution field for java.nio.DirectByteBuffer.arrayBaseOffset. The annotated field can be removed. This ArrayBaseOffset computation can be detected automatically. Use option -H:+UnsafeAutomaticSubstitutionsLogLevel=2 to print all automatically detected substitutions. 
  (typeflow):   2,204.37 ms
   (objects):   1,982.74 ms
  (features):     290.55 ms
    analysis:   4,606.59 ms
16 method(s) included for runtime compilation
    universe:     271.27 ms
     (parse):     365.43 ms
    (inline):     841.42 ms
   (compile):   1,934.96 ms
     compile:   3,458.83 ms
       image:     484.41 ms
       write:     112.36 ms
     [total]:  11,675.41 ms
~/projects/gh-589/rawhttp-scraper$ native-image --version
GraalVM Version 1.0.0-rc5
~/projects/gh-589/rawhttp-scraper$ ./rawhttp-scraper-1.0-SNAPSHOT samples/requests/google.com samples/js/parseJson.js
Exception in thread "main" java.lang.NullPointerException
	at com.athaydes.rawhttp.scraper.RawHttpScraper.run(RawHttpScraper.java:38)
	at com.athaydes.rawhttp.scraper.RawHttpScraper.runRequestFromFile(RawHttpScraper.java:74)
	at com.athaydes.rawhttp.scraper.RawHttpScraper.main(RawHttpScraper.java:53)
	at com.oracle.svm.core.JavaMainWrapper.run(JavaMainWrapper.java:177)
~/projects/gh-589/rawhttp-scraper$ 

As @olpaw said the NPE at the end is something that we need to fix, but I don't get your initial reported issue.

[renatoathaydes]

I realized just now that I was using RC4, which I downloaded just a few days ago!
Got RC5, and now I have the same result as you:

➜  rawhttp-scraper git:(works-on-jvm-but-not-graalvm) ./gradlew clean jar
Starting a Gradle Daemon, 1 incompatible Daemon could not be reused, use --status for details

BUILD SUCCESSFUL in 6s
3 actionable tasks: 3 executed
➜  rawhttp-scraper git:(works-on-jvm-but-not-graalvm) native-image --version
GraalVM Version 1.0.0-rc5
➜  rawhttp-scraper git:(works-on-jvm-but-not-graalvm) native-image --language:js -jar build/libs/rawhttp-scraper-1.0-SNAPSHOT.jar
Build on Server(pid: 97042, port: 60133)*
   classlist:   5,096.66 ms
       (cap):   1,945.20 ms
       setup:   5,021.85 ms
Detected unnecessary RecomputeFieldValue.ArrayBaseOffset com.oracle.svm.core.jdk.Target_java_nio_DirectByteBuffer.arrayBaseOffset substitution field for java.nio.DirectByteBuffer.arrayBaseOffset. The annotated field can be removed. This ArrayBaseOffset computation can be detected automatically. Use option -H:+UnsafeAutomaticSubstitutionsLogLevel=2 to print all automatically detected substitutions. 
  (typeflow):   5,125.43 ms
   (objects):   5,044.46 ms
  (features):     870.77 ms
    analysis:  11,312.78 ms
22 method(s) included for runtime compilation
    universe:     575.12 ms
     (parse):   1,205.79 ms
    (inline):   1,262.92 ms
   (compile):  10,142.31 ms
     compile:  13,314.78 ms
       image:   2,325.76 ms
       write:   1,436.90 ms
     [total]:  39,162.61 ms
➜  rawhttp-scraper git:(works-on-jvm-but-not-graalvm) ✗ ./rawhttp-scraper-1.0-SNAPSHOT samples/requests/get-speakers samples/js/printResponse.js 
Exception in thread "main" java.lang.NullPointerException
	at com.athaydes.rawhttp.scraper.RawHttpScraper.run(RawHttpScraper.java:38)
	at com.athaydes.rawhttp.scraper.RawHttpScraper.runRequestFromFile(RawHttpScraper.java:74)
	at com.athaydes.rawhttp.scraper.RawHttpScraper.main(RawHttpScraper.java:53)
	at com.oracle.svm.core.JavaMainWrapper.run(JavaMainWrapper.java:177)
➜  rawhttp-scraper git:(works-on-jvm-but-not-graalvm) ✗ 

[abcfy2]

I have the same issue when I use my vert.x program: https://github.com/dteam-top/dgate

$ echo $JAVA_HOME
/home/fengyu/graalvm-ce-1.0.0-rc6/

$ ~/graalvm-ce-1.0.0-rc6/bin/java -version
openjdk version "1.8.0_172"
OpenJDK Runtime Environment (build 1.8.0_172-20180625212755.graaluser.jdk8u-src-tar-g-b11)
GraalVM 1.0.0-rc6 (build 25.71-b01-internal-jvmci-0.48, mixed mode)

$ gradle shadowjar

$ ~/graalvm-ce-1.0.0-rc6/bin/java -jar build/libs/dgate-0.1.5-fat.jar  # works very well

$  ~/graalvm-ce-1.0.0-rc6/bin/native-image -jar build/libs/dgate-0.1.5-fat.jar 
Build on Server(pid: 22897, port: 43231)*
[dgate-0.1.5-fat:22897]    classlist:  12,065.61 ms
[dgate-0.1.5-fat:22897]        (cap):   1,423.68 ms
[dgate-0.1.5-fat:22897]        setup:   2,700.90 ms
[dgate-0.1.5-fat:22897]   (typeflow):  15,362.86 ms
[dgate-0.1.5-fat:22897]    (objects):   4,325.05 ms
[dgate-0.1.5-fat:22897]   (features):      81.41 ms
[dgate-0.1.5-fat:22897]     analysis:  19,948.87 ms
[dgate-0.1.5-fat:22897]     universe:     554.48 ms
[dgate-0.1.5-fat:22897]      (parse):   1,969.18 ms
[dgate-0.1.5-fat:22897]     (inline):   2,250.13 ms
[dgate-0.1.5-fat:22897]    (compile):   9,366.35 ms
[dgate-0.1.5-fat:22897]      compile:  14,265.47 ms
[dgate-0.1.5-fat:22897]        image:   1,685.53 ms
[dgate-0.1.5-fat:22897]        write:     354.49 ms
[dgate-0.1.5-fat:22897]      [total]:  51,684.09 ms

$ ldd dgate-0.1.5-fat 
	linux-vdso.so.1 (0x00007fffc05a4000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fcf276f6000)
	librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fcf274ee000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fcf270fd000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fcf2828c000)

$ ./dgate-0.1.5-fat 
Exception in thread "main" java.lang.NullPointerException
	at io.vertx.core.impl.launcher.CommandLineUtils.getJar(CommandLineUtils.java:41)
	at io.vertx.core.impl.launcher.VertxCommandLauncher.getCommandLinePrefix(VertxCommandLauncher.java:298)
	at io.vertx.core.impl.launcher.VertxCommandLauncher.printGlobalUsage(VertxCommandLauncher.java:276)
	at io.vertx.core.impl.launcher.VertxCommandLauncher.dispatch(VertxCommandLauncher.java:393)
	at io.vertx.core.impl.launcher.VertxCommandLauncher.dispatch(VertxCommandLauncher.java:328)
	at top.dteam.dgate.Launcher.main(Launcher.java:24)
	at com.oracle.svm.core.JavaMainWrapper.run(JavaMainWrapper.java:163)

[cstancu]

The original error has been fixed. When you build an image with native-image --language:js ... the Graal.js engine is included and returned by new ScriptEngineManager().getEngineByName("js"). This fix will be available in RC14.