[Java] Make arrow-memory-netty work with latest netty-handler.jar (4.1.94)

[mrmadira]

Describe the enhancement requested

Latest arrow-memory-netty jar fails to work with latest netty-handler.jar with the following exception.

java.lang.NoSuchMethodError: io/netty/buffer/PooledByteBufAllocatorL$InnerAllocator.threadCache()Lio/netty/buffer/PoolThreadCache; (loaded from file:/opt/ibm/spark/jars/arrow-memory-netty-7.0.0.jar by jdk.internal.loader.ClassLoaders$AppClassLoader@5f26aa33) called from class io.netty.buffer.PooledByteBufAllocatorL$InnerAllocator (loaded from file:/opt/ibm/spark/jars/arrow-memory-netty-7.0.0.jar by jdk.internal.loader.ClassLoaders$AppClassLoader@5f26aa33).
	at io.netty.buffer.PooledByteBufAllocatorL$InnerAllocator.newDirectBufferL(PooledByteBufAllocatorL.java:164)
	at io.netty.buffer.PooledByteBufAllocatorL$InnerAllocator.directBuffer(PooledByteBufAllocatorL.java:214)
	at io.netty.buffer.PooledByteBufAllocatorL.allocate(PooledByteBufAllocatorL.java:58)
	at org.apache.arrow.memory.NettyAllocationManager.<init>(NettyAllocationManager.java:77)
	at org.apache.arrow.memory.NettyAllocationManager.<init>(NettyAllocationManager.java:84)
	at org.apache.arrow.memory.NettyAllocationManager$1.create(NettyAllocationManager.java:34)
	at org.apache.arrow.memory.BaseAllocator.newAllocationManager(BaseAllocator.java:315)
	at org.apache.arrow.memory.BaseAllocator.newAllocationManager(BaseAllocator.java:310)
	at org.apache.arrow.memory.BaseAllocator.bufferWithoutReservation(BaseAllocator.java:298)
	at org.apache.arrow.memory.BaseAllocator.buffer(BaseAllocator.java:276)
	at org.apache.arrow.memory.BaseAllocator.buffer(BaseAllocator.java:240)
	at org.apache.arrow.vector.ipc.message.MessageSerializer.readMessageBody(MessageSerializer.java:725)
	at org.apache.arrow.vector.ipc.message.MessageSerializer.deserializeRecordBatch(MessageSerializer.java:363)
	at org.apache.spark.sql.execution.arrow.ArrowConverters$.loadBatch(ArrowConverters.scala:189)
	at org.apache.spark.sql.execution.arrow.ArrowConverters$$anon$2.nextBatch(ArrowConverters.scala:165)
	at org.apache.spark.sql.execution.arrow.ArrowConverters$$anon$2.<init>(ArrowConverters.scala:144)
	at org.apache.spark.sql.execution.arrow.ArrowConverters$.fromBatchIterator(ArrowConverters.scala:143)
	at org.apache.spark.sql.execution.arrow.ArrowConverters$.$anonfun$toDataFrame$1(ArrowConverters.scala:203)
	at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2(RDD.scala:855)
	at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2$adapted(RDD.scala:855)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:329)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:329)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
	at org.apache.spark.scheduler.Task.run(Task.scala:136)
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1504)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:839)

We believe this change in netty-handler is where the integration breaks.: netty/netty@9412424

Request for new version of arrow-memory-netty that will work with latest netty-handler

Component(s)

Java

[danepitkin]

Hey @mrmadira , a new Arrow v13 release is in process right now that will release arrow-memory-netty v13.0.0, which will be pinned to netty v4.1.94.final.

[danepitkin]

Here's the PR that upgrades netty in Arrow: #36211

[danepitkin]

FYI, the Arrow v13 release will actually be pinned to v4.1.96 and will have support for <4.1.94 and >= 4.1.96.

See PR #36926

[danepitkin]

I'll go ahead and close this issue since we won't support netty 4.1.94, but the next release will support the latest version 4.1.96.