Introduce LambdaCapturingTypeBuildItem

[gastaldi]

This introduces support for the lambdaCapturingType configuration introduced in oracle/graal@b455f23

Fixes #24861

[gastaldi]

It's marked as draft until a test is available 😉

[gastaldi]

I've been also thinking of adding a new String[] lambdaCapturingTypes() method to the @RegisterForReflection annotation, WDYT?

[galderz]

So far it looks ok to me.

[zakkak]

LGTM as well. Thanks @gastaldi

[zhfeng]

Hi @gastaldi ,

I add a test for lambda capturing. and it does not work with @RegisterForReflection but it work with serialization-config.json. Please check it to see if I have something missed.

Thanks!

[gastaldi]

@zhfeng that is really weird indeed. I debugged and the code produced in the generated AutoFeature class in https://github.com/gastaldi/quarkus/blob/d6e95a72ed9dffe4fcc578286785b12bdd880900/core/deployment/src/main/java/io/quarkus/deployment/steps/NativeImageAutoFeatureStep.java#L57 is the same as the JSON parser uses in the serialization-config.json:
https://github.com/oracle/graal/blob/master/substratevm/src/com.oracle.svm.core/src/com/oracle/svm/core/configure/SerializationConfigurationParser.java#L96

Any ideas?

[zhfeng]

@gastaldi I have no idea but I just tried to get an Exception when using seriliazion-config.json. It looks like

[1/7] Initializing...                                                                                    (0.0s @ 0.49GB)
Fatal error: java.lang.NullPointerException
	at java.base/java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:936)
	at com.oracle.svm.reflect.hosted.ReflectionDataBuilder.registerMethods(ReflectionDataBuilder.java:149)
	at com.oracle.svm.reflect.hosted.ReflectionDataBuilder.lambda$register$1(ReflectionDataBuilder.java:140)
	at com.oracle.svm.hosted.ConditionalConfigurationRegistry.registerConditionalConfiguration(ConditionalConfigurationRegistry.java:43)
	at com.oracle.svm.reflect.hosted.ReflectionDataBuilder.register(ReflectionDataBuilder.java:140)
	at org.graalvm.sdk/org.graalvm.nativeimage.hosted.RuntimeReflection.register(RuntimeReflection.java:83)
	at com.oracle.svm.reflect.serialize.hosted.SerializationBuilder.registerLambdaCapturingClass(SerializationFeature.java:430)
	at com.oracle.svm.core.configure.SerializationConfigurationParser.parseSerializationDescriptorObject(SerializationConfigurationParser.java:96)
	at com.oracle.svm.core.configure.SerializationConfigurationParser.parseSerializationTypes(SerializationConfigurationParser.java:81)
	at com.oracle.svm.core.configure.SerializationConfigurationParser.parseNewConfiguration(SerializationConfigurationParser.java:74)
	at com.oracle.svm.core.configure.SerializationConfigurationParser.parseAndRegister(SerializationConfigurationParser.java:58)
	at com.oracle.svm.core.configure.ConfigurationParser.parseAndRegister(ConfigurationParser.java:71)
	at com.oracle.svm.hosted.config.ConfigurationParserUtils.doParseAndRegister(ConfigurationParserUtils.java:125)
	at com.oracle.svm.hosted.config.ConfigurationParserUtils.lambda$parseAndRegisterConfigurations$3(ConfigurationParserUtils.java:111)

I hope it is helpful!

[zhfeng]

I think our handle lambda capuring class is right but it might be something with registering Serialization class? see https://github.com/oracle/graal/blob/master/substratevm/src/com.oracle.svm.core/src/com/oracle/svm/core/configure/SerializationConfigurationParser.java#L98-L100

It could be different with graalvm < 22.1 ?

[gastaldi]

@gastaldi I have no idea but I just tried to get an Exception when using serialization-config.json.

I saw this same exception during my tests, that's why I wrapped the calls in a try-catch block, it sounds like a bug in GraalVM.

I think our handle lambda capturing class is right but it might be something with registering Serialization class?

That same method is called when the ReflectiveClassBuildItem is produced and the org.graalvm.nativeimage.hosted.RuntimeSerialization#register() method is called

It could be different with graalvm < 22.1?

I don't know if serializing lambdas were supported in pre-22.1 versions, but I wouldn't hold my breath, since Quarkus recommends 22.1 :)

[zhfeng]

@gastaldi Hmm, it looks like SerializationConfigurationParser invokes https://github.com/oracle/graal/blob/566d7bf7c67dace1a3c7e1679aea918b6df2c9b9/sdk/src/org.graalvm.nativeimage/src/org/graalvm/nativeimage/hosted/RuntimeSerialization.java#L99

[zhfeng]

@gastaldi by using serialization-config.json

    919 classes,   936 fields, and 5,138 methods registered for reflection
      70 classes,    90 fields, and    57 methods registered for JNI access

but by using @RegisterForReflection, it seems that we are missing some reflections?

     919 classes,   936 fields, and 5,126 methods registered for reflection
      70 classes,    90 fields, and    57 methods registered for JNI access

[gastaldi]

That's a good point. I get different results but it looks like the same 12 methods are missing. With @RegisterForReflection:

     968 classes,   423 fields, and 5,296 methods registered for reflection
      65 classes,    78 fields, and    55 methods registered for JNI access

With -H:SerializationConfigurationResources=serialization-config.json:

     968 classes,   423 fields, and 5,308 methods registered for reflection
      65 classes,    78 fields, and    55 methods registered for JNI access

I wonder which methods are these

[gastaldi]

@zhfeng these are the missing methods:

private final java.lang.Object java.util.Comparator$$Lambda$283/0x00000003025fcf50.writeReplace()
private final java.lang.Object java.util.Comparator$$Lambda$478/0x0000000302670a00.writeReplace()
private final java.lang.Object java.util.Comparator$$Lambda$479/0x0000000302670ca0.writeReplace()
private final java.lang.Object java.util.Comparator$$Lambda$480/0x0000000302670f40.writeReplace()
private final java.lang.Object java.util.Comparator$$Lambda$481/0x0000000302671470.writeReplace()
private final java.lang.Object java.util.Comparator$$Lambda$482/0x0000000302671710.writeReplace()
private final java.lang.Object java.util.Comparator$$Lambda$483/0x00000003026719b0.writeReplace()
private final java.lang.Object java.util.Comparator$$Lambda$484/0x0000000302671c50.writeReplace()
private final java.lang.Object java.util.Comparator$$Lambda$485/0x0000000302671ef0.writeReplace()
private final java.lang.Object java.util.Comparator$$Lambda$486/0x0000000302672190.writeReplace()
private final java.lang.Object java.util.Comparator$$Lambda$80/0x00000003000b0bc8.writeReplace()
private final java.lang.Object java.util.Comparator$$Lambda$85/0x00000003000b1888.writeReplace()

[zhfeng]

@gastaldi nice investigating! - how did you get them?

And I think that could be a reason for @RegisterForReflection not working. These missing methods should be registered for reflection by lambda capturing.

[gastaldi]

@gastaldi nice investigating! - how did you get them?

I created another Feature:

@AutomaticFeature
public class GetMethodsAutoFeature implements Feature {

    @Override
    public void afterAnalysis(AfterAnalysisAccess access) {
        RuntimeReflectionSupport support = ImageSingletons.lookup(RuntimeReflectionSupport.class);
        try {
            try (OutputStream os = Files
                    .newOutputStream(Path.of("/Users/ggastald/workspace/quarkus/integration-tests/main/count.txt"));
                    PrintStream ps = new PrintStream(os, true)) {
                support.getReflectionExecutables().stream().map(Executable::toString)
                        .sorted(String.CASE_INSENSITIVE_ORDER)
                        .forEach(ex -> ps.println(ex));
            }
        } catch (IOException e) {
            throw new RuntimeException(e);
        }

    }
}

And compared the generated files on each run :)

And I think that could be a reason for @RegisterForReflection not working. These missing methods should be registered for reflection by lambda capturing.

Probably, still not sure what's missing in the generated bytecode

[zhfeng]

@galderz It looks like we register lambda capuring class too later? after SerializationFeature?

see SerializationFeature::beforeAnalysis() which handles to register lambda methods from capturingClasses.

https://github.com/oracle/graal/blob/vm-ee-22.1.0/substratevm/src/com.oracle.svm.reflect/src/com/oracle/svm/reflect/serialize/hosted/SerializationFeature.java#L223-L232

[zhfeng]

Is there any way to let io.quarkus.runner.AutoFeature run before SerializationFeature?

[zhfeng]

@gastaldi can we register the lambda capuring class at here?

/**
     * Handler for initializations after all features have been registered and all options have been
     * parsed; but before any initializations for the static analysis have happened.
     *
     * @param access The supported operations that the feature can perform at this time
     *
     * @since 19.0
     */
    default void afterRegistration(AfterRegistrationAccess access) {
    }

[zhfeng]

@gastaldi it works with duringSetup. please check gastaldi#17

[zhfeng]

The reason is that we have to register these lambdaCapuringClasses before SerializationFeature doing the static analysis.

[gastaldi]

Nice catch @zhfeng !

[zhfeng]

@gastaldi @gsmet The only thing I concern is the ordering of Feature, so io.quarkus.runner.AutoFeature is always running after SerializationFeature?

[gastaldi]

@zhfeng I don't know how GraalVM orders the Feature implementations it finds, if you can find that in the GraalVM sources maybe that would answer it.

But does it really matter now that you moved the registration to duringSetup?

[zhfeng]

@gastaldi SerializationFeature adds RuntimeSerializationSupport into ImageSingletons in duringSetup

https://github.com/oracle/graal/blob/vm-ee-22.1.0/substratevm/src/com.oracle.svm.reflect/src/com/oracle/svm/reflect/serialize/hosted/SerializationFeature.java#L117

I think we are using it to register the lambdaCapuringClass.

[zakkak]

@gastaldi @gsmet The only thing I concern is the ordering of Feature, so io.quarkus.runner.AutoFeature is always running after SerializationFeature?

TLDR: @zhfeng SerializationFeature will run afterbefore AutoFeature because we already declare it as a requirement.

@zhfeng I don't know how GraalVM orders the Feature implementations it finds, if you can find that in the GraalVM sources maybe that would answer it.

But does it really matter now that you moved the registration to duringSetup?

TLDR: @gastaldi yes it still matters.

Hi @zhfeng and @gastaldi I will try to summarize what's happening and please correct me if I misunderstood something.

1. You are trying to register "lambda capturing types" for serialization using org.graalvm.nativeimage.impl.RuntimeSerializationSupport#registerLambdaCapturingClass and specifically the com.oracle.svm.reflect.serialize.hosted.SerializationBuilder#registerLambdaCapturingClass implementation.
2. In order to do this SerializationBuilder needs to be already added to the imageSingletons which is done in com.oracle.svm.reflect.serialize.hosted.SerializationFeature#duringSetup so the registration of the lambda capturing types needs to happen after the setup phase.
3. We also know that the registration of the lambda capturing types needs to happen before com.oracle.svm.reflect.serialize.hosted.SerializationFeature#beforeAnalysis gets invoked since that method goes through the capturingClasses and registers lambda methods, so it needs to find the lambda capturing types in there. As a result the registration of the lambda capturing types needs to happen before the beforeAnalysis phase.

Since there is no other "handler" to run between duringSetup and beforeAnalysis it looks like we need to run our part in duringSetup but after SerializationFeature or in beforeAnalysis but before SerializationFeature. In order to achieve this we need to define a dependency between our AutoFeature and the SerializationFeature to do this one needs to override org.graalvm.nativeimage.hosted.Feature#getRequiredFeatures in one of the two features. So in this case to run our part in duringSetup but after SerializationFeature we need to add something like the following in AutoFeature:

    public List<Class<? extends Feature>> getRequiredFeatures() {
        return Collections.singletonList(SerializationFeature.class);
    }

Which we already do in

quarkus/core/deployment/src/main/java/io/quarkus/deployment/steps/NativeImageAutoFeatureStep.java

Lines 537 to 548 in b32d4fc

	//register serialization feature as requested
	MethodCreator requiredFeatures = file.getMethodCreator("getRequiredFeatures", "java.util.List");
	TryBlock requiredCatch = requiredFeatures.tryBlock();
	ResultHandle serializationFeatureClass = requiredCatch
	.loadClassFromTCCL("com.oracle.svm.reflect.serialize.hosted.SerializationFeature");
	ResultHandle requiredFeaturesList = requiredCatch.invokeStaticMethod(
	ofMethod("java.util.Collections", "singletonList", List.class, Object.class),
	serializationFeatureClass);
	requiredCatch.returnValue(requiredFeaturesList);

(note that adding AutoFeature as a requirement for SerializationFeature would require altering GraalVM's source code, so it's a no-go)

HTH

[zhfeng]

Thanks a lot @zakkak and it is much more clear now!

[zhfeng]

I think it should be good to add this new feature in the next Quarkus release. Is there anything we need to do before merging?

[gastaldi]

@zhfeng We need someone that has commit rights to review and approve this. /cc @gsmet

[gsmet]

Nice stuff. Let's get it in and we can tweak it later if needed.