Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

JAVA-2940: Add GraalVM native image build configurations #1560

Merged
merged 1 commit into from
Jul 30, 2021

Conversation

absurdfarce
Copy link
Contributor

@absurdfarce absurdfarce commented Jul 2, 2021

Spun off of work done on JAVA-2940

Developed and tested as a feature branch off of #1559. Moved over to 4.11.x under the assumption we might want to fix this with a 4.11 release as well. Can always move it back if we decide not to do so.

@@ -0,0 +1,214 @@
[
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"allPublicConstructors": true,
"allDeclaredMethods": true,
"allPublicMethods": true
},
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another set of dependencies, this time around JNR, discovered in earlier testing. I'm not at all sure these still apply especially since JAVA-2663 (#1439 specifically). Need some more testing here as well.

"fields": [
{"name": "consumerIndex", "allowUnsafeAccess": true}
]
}
Copy link
Contributor Author

@absurdfarce absurdfarce Jul 2, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shaded jctools config from an actual Netty issue (netty/netty#10376)

@adutra
Copy link
Contributor

adutra commented Jul 7, 2021

I think that since #1559 is now linked to JAVA-2950, we could consider that this PR is the "fix" (as in: remaining tweaks after the DependencyCheck issue is solved) for JAVA-2940, wdyt?

"methods": [
{ "name": "<init>", "parameterTypes": [] }
]
},
Copy link
Contributor Author

@absurdfarce absurdfarce Jul 23, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Without this exclusion:

Exception in thread "main" java.lang.IllegalArgumentException: Class NioSocketChannel does not have a public non-arg constructor                                                  
        at io.netty.channel.ReflectiveChannelFactory.<init>(ReflectiveChannelFactory.java:36)                                                                                     
        at io.netty.bootstrap.AbstractBootstrap.channel(AbstractBootstrap.java:110)                                                                                               
        at com.datastax.oss.driver.internal.core.channel.ChannelFactory.connect(ChannelFactory.java:192)                                                                          
        at com.datastax.oss.driver.internal.core.channel.ChannelFactory.connect(ChannelFactory.java:167)                                                                          
        at com.datastax.oss.driver.internal.core.channel.ChannelFactory.connect(ChannelFactory.java:148)                                                                          
        at com.datastax.oss.driver.internal.core.control.ControlConnection$SingleThreaded.connect(ControlConnection.java:363)                                                     
        at com.datastax.oss.driver.internal.core.control.ControlConnection$SingleThreaded.init(ControlConnection.java:303)                                                        
        at com.datastax.oss.driver.internal.core.control.ControlConnection$SingleThreaded.access$1100(ControlConnection.java:243)                                                 
        at com.datastax.oss.driver.internal.core.control.ControlConnection.lambda$init$0(ControlConnection.java:122)                                                              
        at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)                                                                                                      
        at io.netty.util.concurrent.PromiseTask.run(PromiseTask.java:106)                                                                                                         
        at io.netty.channel.DefaultEventLoop.run(DefaultEventLoop.java:54)                                                                                                        
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)                                                                           
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)                                                                                              
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)                                                                                  
        at java.lang.Thread.run(Thread.java:748)                                                                                                                                  
        at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:553)                                                                                        
        at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:192)                                                                       
Caused by: java.lang.NoSuchMethodException: io.netty.channel.socket.nio.NioSocketChannel.<init>()                                                                                 
        at java.lang.Class.getConstructor0(DynamicHub.java:3082)                                                                                                                  
        at java.lang.Class.getConstructor(DynamicHub.java:1825)                                                                                                                   
        at io.netty.channel.ReflectiveChannelFactory.<init>(ReflectiveChannelFactory.java:34)
        ... 17 more

Not abundantly clear why this would be; NioSocketChannel very definitely does have a no-arg constructor. It's possible there's another Graal substitution at play here, although if that's the case I haven't been able to identify where it lives.

Note that Helidon had to address something similar: see helidon-io/helidon#1722

"methods": [
{ "name": "toLeakAwareBuffer", "parameterTypes": ["io.netty.buffer.ByteBuf"] }
]
},
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Without this exclusion:

Exception in thread "main" java.lang.ExceptionInInitializerError
        at com.oracle.svm.core.classinitialization.ClassInitializationInfo.initialize(ClassInitializationInfo.java:315)                                                          
        at java.lang.Class.ensureInitialized(DynamicHub.java:553)
        at com.oracle.svm.core.classinitialization.ClassInitializationInfo.initialize(ClassInitializationInfo.java:260)                                                          
        at io.netty.buffer.ByteBufUtil.<clinit>(ByteBufUtil.java:86)
        at com.oracle.svm.core.classinitialization.ClassInitializationInfo.invokeClassInitializer(ClassInitializationInfo.java:375)                                              
        at com.oracle.svm.core.classinitialization.ClassInitializationInfo.initialize(ClassInitializationInfo.java:295)                                                          
        at io.netty.buffer.ByteBufAllocator.<clinit>(ByteBufAllocator.java:24)
        at com.oracle.svm.core.classinitialization.ClassInitializationInfo.invokeClassInitializer(ClassInitializationInfo.java:375)                                              
        at com.oracle.svm.core.classinitialization.ClassInitializationInfo.initialize(ClassInitializationInfo.java:295)                                                          
        at com.datastax.oss.driver.internal.core.context.DefaultNettyOptions.allocator(DefaultNettyOptions.java:144)                                                             
        at com.datastax.oss.driver.internal.core.channel.ChannelFactory.connect(ChannelFactory.java:193)                                                                         
        at com.datastax.oss.driver.internal.core.channel.ChannelFactory.connect(ChannelFactory.java:167)                                                                         
        at com.datastax.oss.driver.internal.core.channel.ChannelFactory.connect(ChannelFactory.java:148)                                                                         
        at com.datastax.oss.driver.internal.core.control.ControlConnection$SingleThreaded.connect(ControlConnection.java:363)                                                    
        at com.datastax.oss.driver.internal.core.control.ControlConnection$SingleThreaded.init(ControlConnection.java:303)                                                       
        at com.datastax.oss.driver.internal.core.control.ControlConnection$SingleThreaded.access$1100(ControlConnection.java:243)                                                
        at com.datastax.oss.driver.internal.core.control.ControlConnection.lambda$init$0(ControlConnection.java:122)                                                             
        at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
        at io.netty.util.concurrent.PromiseTask.run(PromiseTask.java:106)
        at io.netty.channel.DefaultEventLoop.run(DefaultEventLoop.java:54)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)                                                                          
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)                                                                                 
        at java.lang.Thread.run(Thread.java:748)
        at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:553)                                                                                       
        at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:192)                                                                      
Caused by: java.lang.IllegalArgumentException: Can't find '[toLeakAwareBuffer]' in io.netty.buffer.AbstractByteBufAllocator                                                      
        at io.netty.util.ResourceLeakDetector.addExclusions(ResourceLeakDetector.java:576)                                                                                       
        at io.netty.buffer.AbstractByteBufAllocator.<clinit>(AbstractByteBufAllocator.java:36)                                                                                   
        at com.oracle.svm.core.classinitialization.ClassInitializationInfo.invokeClassInitializer(ClassInitializationInfo.java:375)                                              
        at com.oracle.svm.core.classinitialization.ClassInitializationInfo.initialize(ClassInitializationInfo.java:295)                                                          
        ... 25 more

Haven't dug into this one much to see if there's anything we can do here.

"methods": [
{ "name": "touch", "parameterTypes": ["java.lang.Object", "java.lang.Object"] }
]
},
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Without this exclusion:

WARNING: An exception was thrown by com.datastax.oss.driver.internal.core.channel.ConnectInitHandler$$Lambda$df350d7c55c0f44e72fb6470156ed35696eba329.operationComplete() [3/1826]
java.lang.ExceptionInInitializerError                                                                                                                                             
        at com.oracle.svm.core.classinitialization.ClassInitializationInfo.initialize(ClassInitializationInfo.java:315)                                                           
        at io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)                                                                                         
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:786)
        at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:758)
        at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:808)
        at io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1025)
        at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:294)
        at com.datastax.oss.driver.internal.core.channel.ChannelHandlerRequest.send(ChannelHandlerRequest.java:75)
        at com.datastax.oss.driver.internal.core.channel.ProtocolInitHandler$InitRequest.send(ProtocolInitHandler.java:193)
        at com.datastax.oss.driver.internal.core.channel.ProtocolInitHandler.onRealConnect(ProtocolInitHandler.java:124)
        at com.datastax.oss.driver.internal.core.channel.ConnectInitHandler.lambda$connect$0(ConnectInitHandler.java:57)
        at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578)
        at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:571)
        at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:550)
        at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:491)
        at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:616)
        at io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:605)
        at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104)
        at io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84)
        at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:300)
        at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:335)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:707)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.lang.Thread.run(Thread.java:748)
        at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:553)
        at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:192)
Caused by: java.lang.IllegalArgumentException: Can't find '[touch]' in io.netty.util.ReferenceCountUtil
        at io.netty.util.ResourceLeakDetector.addExclusions(ResourceLeakDetector.java:576)
        at io.netty.util.ReferenceCountUtil.<clinit>(ReferenceCountUtil.java:30)
        at com.oracle.svm.core.classinitialization.ClassInitializationInfo.invokeClassInitializer(ClassInitializationInfo.java:375)
        at com.oracle.svm.core.classinitialization.ClassInitializationInfo.initialize(ClassInitializationInfo.java:295)

Also haven't dug into the underlying causes for this one yet.

@absurdfarce absurdfarce requested a review from adutra July 26, 2021 16:16
@adutra
Copy link
Contributor

adutra commented Jul 27, 2021

I've been playing with this today and I have a few additional remarks:

LZ4 Compression

I am trying to validate compression with lz4. It seems that it is not working:

java.lang.IllegalStateException: Could not find the LZ4 library on the classpath (the driver declares it as an optional dependency, so you need to declare it explicitly)
        at com.datastax.oss.driver.internal.core.protocol.Lz4Compressor.<init>(Lz4Compressor.java:50)
        at com.datastax.oss.driver.internal.core.protocol.Lz4Compressor.<init>(Lz4Compressor.java:39)
        at com.datastax.oss.driver.internal.core.protocol.BuiltInCompressors.newInstance(BuiltInCompressors.java:44)

Indeed, even if BuiltInCompressors is properly substituted, LZ4 itself does some more reflection to find out the right classes to instantiate. This isn't working. I had to add a whole bunch of exceptions in reflection.json:

  {
    "name" : "net.jpountz.lz4.LZ4Compressor",
    "allDeclaredConstructors": true,
    "allPublicConstructors": true,
    "allDeclaredMethods": true,
    "allPublicMethods": true
  },
  {
    "name" : "net.jpountz.lz4.LZ4JNICompressor",
    "allDeclaredConstructors": true,
    "allPublicConstructors": true,
    "allDeclaredMethods": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name" : "net.jpountz.lz4.LZ4JavaSafeCompressor",
    "allDeclaredConstructors": true,
    "allPublicConstructors": true,
    "allDeclaredMethods": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name" : "net.jpountz.lz4.LZ4JavaUnsafeCompressor",
    "allDeclaredConstructors": true,
    "allPublicConstructors": true,
    "allDeclaredMethods": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name" : "net.jpountz.lz4.LZ4HCJavaSafeCompressor",
    "allDeclaredConstructors": true,
    "allPublicConstructors": true,
    "allDeclaredMethods": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name" : "net.jpountz.lz4.LZ4HCJavaUnsafeCompressor",
    "allDeclaredConstructors": true,
    "allPublicConstructors": true,
    "allDeclaredMethods": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name" : "net.jpountz.lz4.LZ4JavaSafeSafeDecompressor",
    "allDeclaredConstructors": true,
    "allPublicConstructors": true,
    "allDeclaredMethods": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name" : "net.jpountz.lz4.LZ4JavaSafeFastDecompressor",
    "allDeclaredConstructors": true,
    "allPublicConstructors": true,
    "allDeclaredMethods": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name" : "net.jpountz.lz4.LZ4JavaUnsafeSafeDecompressor",
    "allDeclaredConstructors": true,
    "allPublicConstructors": true,
    "allDeclaredMethods": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name" : "net.jpountz.lz4.LZ4JavaUnsafeFastDecompressor",
    "allDeclaredConstructors": true,
    "allPublicConstructors": true,
    "allDeclaredMethods": true,
    "allPublicMethods": true,
    "allPublicFields": true
  }

We need all of the above unfortunately, including the abstract superclass LZ4Compressor :-(

TypeVariable

There is something fishy with TypeVariable, the proxy config is not working it seems:

Error: com.oracle.svm.hosted.substitute.DeletedElementException: Unsupported method java.lang.Class.getConstantPool() is reachable: The declaring class of this element has been substituted, but this element is not present in the substitution class
To diagnose the issue, you can add the option --report-unsupported-elements-at-runtime. The unsupported element is then reported at run time when it is accessed the first time.
Trace: 
        at parsing java.lang.System$2.getConstantPool(System.java:1230)
Call path from entry point to java.lang.System$2.getConstantPool(Class): 
        at java.lang.System$2.getConstantPool(System.java:1230)
        at sun.reflect.annotation.TypeAnnotationParser.parseAllTypeAnnotations(TypeAnnotationParser.java:323)
        at sun.reflect.annotation.TypeAnnotationParser.fetchBounds(TypeAnnotationParser.java:299)
        at sun.reflect.annotation.TypeAnnotationParser.parseAnnotatedBounds(TypeAnnotationParser.java:244)
        at sun.reflect.annotation.TypeAnnotationParser.parseAnnotatedBounds(TypeAnnotationParser.java:237)
        at sun.reflect.generics.reflectiveObjects.TypeVariableImpl.getAnnotatedBounds(TypeVariableImpl.java:241)
        at com.oracle.svm.reflect.TypeVariable_getAnnotatedBounds_88ea462da2b17ea45028db307519aaeeec9a4036_58.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.datastax.oss.driver.shaded.guava.common.reflect.Types$TypeVariableInvocationHandler.invoke(Types.java:387)

I still need to investigate.

ServiceLoader

I manually validated that this is working by running the following code:

    ServiceLoader<BlockHoundIntegration> serviceLoader = ServiceLoader.load(BlockHoundIntegration.class);
    boolean driverBlockHoundIntegrationFound = StreamSupport.stream(serviceLoader.spliterator(),
        false)
        .map(integration -> integration.getClass().getSimpleName())
        .anyMatch(name -> name.contains("DriverBlockHoundIntegration"));
    System.out.println("DriverBlockHoundIntegration registered = " + driverBlockHoundIntegrationFound);

@adutra
Copy link
Contributor

adutra commented Jul 27, 2021

Also @absurdfarce would you mind rebasing this PR on top of 4.11.x? Otherwise we are going to hit JAVA-2950 during tests.

@adutra
Copy link
Contributor

adutra commented Jul 27, 2021

Regarding lz4, what's even worse is that we cannot unconditionally add those LZ4 classes to reflection.json. If LZ4 is not present during image build, the build will fail. I think we need something like an Lz4Feature :-(

@adutra
Copy link
Contributor

adutra commented Jul 27, 2021

Regarding lz4, what's even worse is that we cannot unconditionally add those LZ4 classes to reflection.json. If LZ4 is not present during image build, the build will fail. I think we need something like an Lz4Feature :-(

Update: in fact the build succeeds, but generates warnings:

WARNING: Could not resolve net.jpountz.lz4.LZ4Compressor for reflection configuration. Reason: java.lang.ClassNotFoundException: net.jpountz.lz4.LZ4Compressor.
WARNING: Could not resolve net.jpountz.lz4.LZ4JNICompressor for reflection configuration. Reason: java.lang.ClassNotFoundException: net.jpountz.lz4.LZ4JNICompressor.
WARNING: Could not resolve net.jpountz.lz4.LZ4JavaSafeCompressor for reflection configuration. Reason: java.lang.ClassNotFoundException: net.jpountz.lz4.LZ4JavaSafeCompressor.
WARNING: Could not resolve net.jpountz.lz4.LZ4JavaUnsafeCompressor for reflection configuration. Reason: java.lang.ClassNotFoundException: net.jpountz.lz4.LZ4JavaUnsafeCompressor.
WARNING: Could not resolve net.jpountz.lz4.LZ4HCJavaSafeCompressor for reflection configuration. Reason: java.lang.ClassNotFoundException: net.jpountz.lz4.LZ4HCJavaSafeCompressor.
WARNING: Could not resolve net.jpountz.lz4.LZ4HCJavaUnsafeCompressor for reflection configuration. Reason: java.lang.ClassNotFoundException: net.jpountz.lz4.LZ4HCJavaUnsafeCompressor.
WARNING: Could not resolve net.jpountz.lz4.LZ4JavaSafeSafeDecompressor for reflection configuration. Reason: java.lang.ClassNotFoundException: net.jpountz.lz4.LZ4JavaSafeSafeDecompressor.
WARNING: Could not resolve net.jpountz.lz4.LZ4JavaSafeFastDecompressor for reflection configuration. Reason: java.lang.ClassNotFoundException: net.jpountz.lz4.LZ4JavaSafeFastDecompressor.
WARNING: Could not resolve net.jpountz.lz4.LZ4JavaUnsafeSafeDecompressor for reflection configuration. Reason: java.lang.ClassNotFoundException: net.jpountz.lz4.LZ4JavaUnsafeSafeDecompressor.
WARNING: Could not resolve net.jpountz.lz4.LZ4JavaUnsafeFastDecompressor for reflection configuration. Reason: java.lang.ClassNotFoundException: net.jpountz.lz4.LZ4JavaUnsafeFastDecompressor.

@adutra
Copy link
Contributor

adutra commented Jul 27, 2021

More on the TypeVariable error: It seems related to the specific GraalVM version I'm using.

  • graalvm-ce-java8-21.0.0.2 : DOES NOT WORK, spits that weird error.
  • graalvm-ce-java11-21.2.0 : works
  • graalvm-ee-java8-21.2.0: works

I guess we can consider that a bug in that GraalVM version and shovel it under the rug.

@adutra adutra changed the title Include Graal native image config files for default options JAVA-2940: Add GraalVM native image build configurations Jul 29, 2021
@adutra adutra added this to the 4.11.3 milestone Jul 29, 2021
@adutra adutra force-pushed the graal_native_image_build_configs branch from ae92a06 to 93a394f Compare July 29, 2021 16:32
@adutra adutra changed the base branch from 4.11.x to 4.x July 29, 2021 16:32
@adutra adutra modified the milestones: 4.11.3, 4.13.0 Jul 29, 2021
@adutra adutra force-pushed the graal_native_image_build_configs branch 2 times, most recently from 0566186 to 3ed3f4d Compare July 30, 2021 14:11
@adutra adutra force-pushed the graal_native_image_build_configs branch from 3dfe5c3 to 76c9ca6 Compare July 30, 2021 14:40
@adutra adutra merged commit 4270f93 into 4.x Jul 30, 2021
@adutra adutra deleted the graal_native_image_build_configs branch July 30, 2021 14:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants