[Java] Integration test fails with UnsupportedOperationException

[asfimport]

Running the java integration test fails with an exception:

$ java -classpath ~/arrow/java/tools/target/arrow-tools-0.14.0-SNAPSHOT-jar-with-dependencies.jar -Dio.netty.tryReflectionSetAccessible=false org.apache.arrow.tools.Integration -a ~/tmp/1832930b_simple.json_as_file -j ~/arrow/integration/data/simple.json -c VALIDATE

...

Incompatible files
sun.misc.Unsafe or java.nio.DirectByteBuffer.<init>(long, int) not available
08:55:43.597 [main] ERROR org.apache.arrow.tools.Integration - Incompatible files
java.lang.UnsupportedOperationException: sun.misc.Unsafe or java.nio.DirectByteBuffer.<init>(long, int) not available
	at io.netty.util.internal.PlatformDependent.directBuffer(PlatformDependent.java:399)
	at io.netty.buffer.NettyArrowBuf.getDirectBuffer(NettyArrowBuf.java:233)
	at io.netty.buffer.NettyArrowBuf.nioBuffer(NettyArrowBuf.java:223)
	at io.netty.buffer.ArrowBuf.nioBuffer(ArrowBuf.java:245)
	at org.apache.arrow.vector.ipc.message.ArrowRecordBatch.computeBodyLength(ArrowRecordBatch.java:211)
	at org.apache.arrow.vector.ipc.message.MessageSerializer.serialize(MessageSerializer.java:175)
	at org.apache.arrow.vector.ipc.ArrowWriter.writeRecordBatch(ArrowWriter.java:119)
	at org.apache.arrow.vector.ipc.ArrowFileWriter.writeRecordBatch(ArrowFileWriter.java:61)
	at org.apache.arrow.vector.ipc.ArrowWriter.writeBatch(ArrowWriter.java:107)
	at org.apache.arrow.tools.Integration$Command$2.execute(Integration.java:171)
	at org.apache.arrow.tools.Integration.run(Integration.java:118)
	at org.apache.arrow.tools.Integration.main(Integration.java:69)

Looking through netty's source, it looks like this exception is emitted here.

$ apt search jdk | grep installed
default-jre/bionic-updates,bionic-security,now 2:1.11-68ubuntu1~18.04.1 amd64 [installed,automatic]
default-jre-headless/bionic-updates,bionic-security,now 2:1.11-68ubuntu1~18.04.1 amd64 [installed,automatic]
libslf4j-java/bionic,bionic,now 1.7.25-3 all [installed,automatic]
openjdk-11-jdk/bionic-updates,bionic-security,now 11.0.3+7-1ubuntu2~18.04.1 amd64 [installed]
openjdk-11-jdk-headless/bionic-updates,bionic-security,now 11.0.3+7-1ubuntu2~18.04.1 amd64 [installed,automatic]
openjdk-11-jre/bionic-updates,bionic-security,now 11.0.3+7-1ubuntu2~18.04.1 amd64 [installed,automatic]
openjdk-11-jre-headless/bionic-updates,bionic-security,now 11.0.3+7-1ubuntu2~18.04.1 amd64 [installed,automatic]

Reporter: Ben Kietzman / @bkietz
Assignee: Bryan Cutler / @BryanCutler

PRs and other links:

• GitHub Pull Request #4522

_{Note: This issue was originally created as ARROW-5412. Please see the migration documentation for further details.}

[asfimport]

Ben Kietzman / @bkietz:
Tried again with Oracle JDK 11, got the same error:

$ apt search jdk | grep installed
libslf4j-java/bionic,bionic,now 1.7.25-3 all [installed,automatic]
oracle-java11-installer/bionic,now 11.0.2-2~linuxuprising1 amd64 [installed]
oracle-java11-set-default/bionic,now 11.0.2-2~linuxuprising1 amd64 [installed,automatic]

[asfimport]

Wes McKinney / @wesm:
@siddharthteotia @pravindra do you know what this might be about? I haven't seen this issue myself

[asfimport]

Wes McKinney / @wesm:
It's possible this is fixed by using a newer version of Netty, can you try bumping the version here to the latest 4.1.x version?

https://github.com/apache/arrow/blob/master/java/pom.xml#L36

[asfimport]

Ben Kietzman / @bkietz:
Same error is emitted with Netty 4.1.36

[asfimport]

Bryan Cutler / @BryanCutler:
Yeah, sounds like this could be related to the change in ARROW-3191 described here http://mail-archives.apache.org/mod_mbox/arrow-dev/201904.mbox/%3cCAKHW1aoU7R7wcyprd5SYmfnmC3WtC=zNnTbpB=jV3JLE=Q+TGg@mail.gmail.com%3e

@bkietz have you tried an older JDK, maybe 8? Any suggestions @siddharthteotia?

[asfimport]

Ben Kietzman / @bkietz:
Tried again with Oracle JDK 8, got the same error:

$ apt search jdk | grep installed
jdk1.8/now 1.8.0211-1 amd64 [installed,local]
libslf4j-java/bionic,bionic,now 1.7.25-3 all [installed,auto-removable]

[asfimport]

Bryan Cutler / @BryanCutler:
I also saw the same error with JDK 8 and 9. I got it passing for Java by adding -Dio.netty.tryReflectionSetAccessible=true to the cmd. Let me put up a PR for the fix.

[asfimport]

Bryan Cutler / @BryanCutler:
Actually, I got it working with JDK 8 without the above conf. I think previously, I had compiled with JDK 8 and tested with JDK 9 which led to the netty error. I'll try again with just JDK 9.

[asfimport]

Bryan Cutler / @BryanCutler:
Issue resolved by pull request 4522
#4522

[asfimport]

Jim Northrup:
setting the system properties as first line of main() succeeds for jdk12 where latest release as-of-this-comment still throws the exception cited.

@JvmStatic
fun main(vararg args: String) = runBlocking{
  System.setProperty( "io.netty.tryReflectionSetAccessible","true") 
  go(*args) 
}