GraalVM native-image build is broken

[poslegm]

• Version: 2.1.0

Steps

CLASSPATH=$(coursier fetch org.scalameta:scalafmt-cli_2.12:2.1.0 | paste -s -d: -)
native-image -cp $CLASSPATH org.scalafmt.cli.Cli -H:+ReportUnsupportedElementsAtRuntime --initialize-at-build-time --no-fallback --allow-incomplete-classpath

Problem

GraalVM output:

Build on Server(pid: 325943, port: 42437)
[org.scalafmt.cli.cli:325943]    classlist:  13,856.86 ms
[org.scalafmt.cli.cli:325943]        (cap):   1,389.98 ms
[org.scalafmt.cli.cli:325943]        setup:   1,972.81 ms
[org.scalafmt.cli.cli:325943]     analysis: 109,416.49 ms
Error: com.oracle.graal.pointsto.constraints.UnsupportedFeatureException: Invoke with MethodHandle argument could not be reduced to at most a single call: java.lang.invoke.LambdaForm$MH/1643382681.invoke_MT(Object, Object)
Detailed message:
Trace:
        at parsing coursierapi.shaded.scala.runtime.Statics.releaseFence(Statics.java:148)
Call path from entry point to coursierapi.shaded.scala.runtime.Statics.releaseFence():
        at coursierapi.shaded.scala.runtime.Statics.releaseFence(Statics.java:148)
        at coursierapi.shaded.scala.collection.immutable.$colon$colon.<init>(List.scala:593)
        at coursierapi.shaded.scala.collection.immutable.Set$Set4.iterator(Set.scala:243)
        at coursierapi.shaded.scala.util.hashing.MurmurHash3.unorderedHash(MurmurHash3.scala:95)
        at coursierapi.shaded.scala.collection.immutable.HashSet.hashCode(HashSet.scala:184)
        at java.util.HashMap.hash(HashMap.java:339)
        at java.util.HashMap.get(HashMap.java:557)
        at com.oracle.svm.jni.access.JNIReflectionDictionary.getClassObjectByName(JNIReflectionDictionary.java:123)
        at com.oracle.svm.jni.functions.JNIFunctions.FindClass(JNIFunctions.java:326)
        at com.oracle.svm.core.code.IsolateEnterStub.JNIFunctions_FindClass_3ec1032c6cb9443725d1e68194130533bfb04076(generated:0)

Error: Use -H:+ReportExceptionStackTraces to print stacktrace of underlying exception
Error: Image build request failed with exit status 1

Notes

Build was broken after #1511

[olafurpg]

Thank you for reporting! Did you try passing in a reflection file? See

• https://github.com/scalacenter/bloop/blob/0536c40217da480c4ae39f3b1cf47be77f144f14/build.sbt#L227-L241
• https://github.com/scalacenter/bloop/blob/0536c40217da480c4ae39f3b1cf47be77f144f14/bloopgun/src/main/graal/reflection.json

Even if you get it working, I believe it might be better to link a different main function that doesn't do runtime dependency resolution and classloading. It would only use the ScalafmtCoreRunner, not ScalafmtDynamicRunner

scalafmt/scalafmt-cli/src/main/scala/org/scalafmt/cli/Cli.scala

Lines 91 to 93 in ba6ebc3

	case None => ScalafmtCoreRunner
	case Some(v) if v == Versions.version =>
	ScalafmtCoreRunner

This main function could error in cases when the version setting in .scalafmt.conf doesn't match the Scalafmt version in the binary.

[olafurpg]

BTW, coursier has a -p flag to construct a classpath, the paste command should not be needed

CLASSPATH=$(coursier fetch org.scalameta:scalafmt-cli_2.12:2.1.0 -p)

[poslegm]

@mroth @olafurpg JFYI:

sbt cli/assembly is broken because coursier/interface depends on scala-reflect 2.13.0 and scalafmt depends on scala-reflect 2.12.8:

[error] (cli / assembly) deduplicate: different file contents found in the following:
[error] /Users/poslegm/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/io/get-coursier/interface/0.0.12/interface-0.0.12.jar:reflect.properties
[error] /Users/poslegm/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-reflect/2.12.8/scala-reflect-2.12.8.jar:reflect.properties

I hope that #1434 can solve this.

UPD: I built scalafmt for 2.13 and assembly works good

[poslegm]

Also I tried to play with Graal reflection configs but unsuccessful 😞

Judging by the stacktrace problem may be in https://github.com/olafurpg/metaconfig/blob/master/metaconfig-core/shared/src/main/scala/metaconfig/generic/Field.scala#L37 (StaticAnnotation in lambda)

Error: com.oracle.graal.pointsto.constraints.UnsupportedFeatureException: Invoke with MethodHandle argument could not be reduced to at most a single call: java.lang.invoke.MethodHandleImpl$BindCaller.prepareForInvoker(MethodHandle)
Trace:
        at parsing java.lang.invoke.MethodHandleImpl$BindCaller.makeInjectedInvoker(MethodHandleImpl.java:1166)
Call path from entry point to java.lang.invoke.MethodHandleImpl$BindCaller.makeInjectedInvoker(Class):
        at java.lang.invoke.MethodHandleImpl$BindCaller.makeInjectedInvoker(MethodHandleImpl.java:1141)
        at java.lang.invoke.MethodHandleImpl$BindCaller.access$300(MethodHandleImpl.java:1122)
        at java.lang.invoke.MethodHandleImpl$BindCaller$1.computeValue(MethodHandleImpl.java:1175)
        at java.lang.invoke.MethodHandleImpl$BindCaller$1.computeValue(MethodHandleImpl.java:1173)
        at com.oracle.svm.core.jdk.Target_java_lang_ClassValue.get(JavaLangSubstitutions.java:514)
        at java.lang.invoke.MethodHandleImpl$BindCaller.bindCaller(MethodHandleImpl.java:1136)
        at java.lang.invoke.MethodHandleImpl.bindCaller(MethodHandleImpl.java:1117)
        at java.lang.invoke.MethodHandles$Lookup.maybeBindCaller(MethodHandles.java:1681)
        at java.lang.invoke.MethodHandles$Lookup.getDirectMethodCommon(MethodHandles.java:1669)
        at java.lang.invoke.MethodHandles$Lookup.getDirectMethod(MethodHandles.java:1605)
        at java.lang.invoke.MethodHandles$Lookup.findStatic(MethodHandles.java:781)
        at java.lang.invoke.BoundMethodHandle$Factory.makeCbmhCtor(BoundMethodHandle.java:818)
        at java.lang.invoke.BoundMethodHandle$Factory.makeCtors(BoundMethodHandle.java:763)
        at java.lang.invoke.BoundMethodHandle$SpeciesData.<init>(BoundMethodHandle.java:349)
        at java.lang.invoke.BoundMethodHandle$SpeciesData$1.apply(BoundMethodHandle.java:389)
        at java.lang.invoke.BoundMethodHandle$SpeciesData$1.apply(BoundMethodHandle.java:383)
        at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660)
        at java.lang.invoke.BoundMethodHandle$SpeciesData.get(BoundMethodHandle.java:383)
        at java.lang.invoke.BoundMethodHandle$SpeciesData.extendWith(BoundMethodHandle.java:378)
        at java.lang.invoke.BoundMethodHandle$Species_L.copyWithExtendL(BoundMethodHandle.java:233)
        at java.lang.invoke.MethodHandle.asCollector(MethodHandle.java:1005)
        at java.lang.invoke.MethodHandleImpl$AsVarargsCollector.asTypeUncached(MethodHandleImpl.java:502)
        at java.lang.invoke.MethodHandle.asType(MethodHandle.java:761)
        at java.lang.invoke.Invokers.checkGenericType(Invokers.java:321)
        at java.lang.invoke.LambdaForm$MH/1078130969.invoke_MT(LambdaForm$MH)
        at scala.runtime.Statics.releaseFence(Statics.java:148)
        at scala.collection.immutable.List.map(List.scala:231)
        at metaconfig.generic.Field.toString(Field.scala:37) <--- THIS
        at java.lang.String.valueOf(String.java:2994)
        at java.lang.StringBuilder.append(StringBuilder.java:131)
        at com.oracle.svm.core.amd64.AMD64CPUFeatureAccess.verifyHostSupportsArchitecture(AMD64CPUFeatureAccess.java:179)
        at com.oracle.svm.core.JavaMainWrapper.runCore(JavaMainWrapper.java:129)
        at com.oracle.svm.core.JavaMainWrapper.run(JavaMainWrapper.java:186)
        at com.oracle.svm.core.code.IsolateEnterStub.JavaMainWrapper_run_5087f5482cc9a6abc971913ece43acb471d2631b(generated:0)

But I am not sure

[olafurpg]

Interesting, I would be open to merge a metaconfig PR refactoring the lambda into

    val annots = annotations.mkString("@", ", @", "")

if that helps.

[olafurpg]

I'm able to link scalafmt with the recent 19.3 release without changes to scalafmt or metaconfig!

[poslegm]

I confirm that. Latest version of scalafmt successfully builds with GraalVM v19.3 and command

CLASSPATH=$(coursier fetch org.scalameta:scalafmt-cli_2.13:2.2.2 -p)
native-image -cp $CLASSPATH org.scalafmt.cli.Cli -H:+ReportUnsupportedElementsAtRuntime --initialize-at-build-time --no-fallback --allow-incomplete-classpath

@mroth try it in your build pipeline, please

[olafurpg]

I'm currently working on improving the native-image support (better error messages for version mismatch, improved performance).

@mroth FYI inspired by your project, would also like to update our CI to build binaries for linux/macos on every merge into master!

[mroth]

@mroth try it in your build pipeline, please

With this and a few changes it seems to be working now. I haven't tested the actual binaries (I'm currently at the airport in China about to board a flight) but it looks like its succeeding. Check out the latest at mroth/scalafmt-native#6 and let me know what you think, I can get this merged this weekend or early next week and release if all looks good.

@mroth FYI inspired by your project, would also like to update our CI to build binaries for linux/macos on every merge into master!

Awesome! Let me know how I can help.

[olafurpg]

Fixed in #1561.