Unable to connect to Docker based Dremio from Kestra

[shrutimantri]

Expected Behavior

We should be able to connect to Dremio server running locally on Docker, and query the tables on it.

Actual Behaviour

Unable to connect to Dremio server running locally on Docker from Kestra on Docker.

Steps To Reproduce

1. Run docker desktop

2. docker pull dremio/dremio-oss

3. docker run -p 9047:9047 -p 31010:31010 -p 45678:45678 dremio/dremio-oss

4. You can access the dremio server on local browser using: http://localhost:9047

5. Create a sample table on the dremio server.

6. You can connect to this dremio server using SQL clients like DB Visualizer using: "jdbc:dremio:direct=localhost:31010" and the appropriate username and password.
   

7. But we are unable to connect to this Dremio server using Kestra on Docker:

id: dremio-task
namespace: company.team
tasks:
  - id: dremio-query
    type: "io.kestra.plugin.jdbc.dremio.Query"
    url: jdbc:dremio:direct=<local_ip>:31010
    username: <username>
    password: <passwd>
    sql: select * FROM "Samples"."samples.dremio.com"."SF_incidents2016.json"
    fetchOne: true

We get the error:

2024-02-17 17:39:08.340Failure in connecting to Dremio: cdjd.com.dremio.exec.rpc.RpcException: HANDSHAKE_COMMUNICATION : [USER]: Channel closed null <--> null (user client)
2024-02-17 17:39:08.340HANDSHAKE_COMMUNICATION : [USER]: Channel closed null <--> null (user client)
[USER]: Channel closed null <--> null (user client)
2024-02-17 17:39:08.340java.sql.SQLException: Failure in connecting to Dremio: cdjd.com.dremio.exec.rpc.RpcException: HANDSHAKE_COMMUNICATION : [USER]: Channel closed null <--> null (user client)
	at com.dremio.jdbc.impl.DremioExceptionMapper.map(DremioExceptionMapper.java:81)
	at com.dremio.jdbc.impl.DremioConnectionImpl.<init>(DremioConnectionImpl.java:115)
	at com.dremio.jdbc.impl.DremioJdbc41Factory.newConnection(DremioJdbc41Factory.java:72)
	at com.dremio.jdbc.impl.DremioFactory.newConnection(DremioFactory.java:67)
	at cdjd.org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:138)
	at com.dremio.jdbc.Driver.connect(Driver.java:70)
	at java.sql/java.sql.DriverManager.getConnection(Unknown Source)
	at java.sql/java.sql.DriverManager.getConnection(Unknown Source)
	at io.kestra.plugin.jdbc.JdbcConnectionInterface.connection(JdbcConnectionInterface.java:63)
	at io.kestra.plugin.jdbc.AbstractJdbcQuery.run(AbstractJdbcQuery.java:77)
	at io.kestra.plugin.jdbc.dremio.Query.run(Query.java:55)
	at io.kestra.plugin.jdbc.dremio.Query.run(Query.java:20)
	at io.kestra.core.runners.Worker$WorkerThread.run(Worker.java:729)
Caused by: cdjd.com.dremio.exec.rpc.RpcException: HANDSHAKE_COMMUNICATION : [USER]: Channel closed null <--> null (user client)
	at cdjd.com.dremio.exec.client.DremioClient$FutureHandler.connectionFailed(DremioClient.java:826)
	at cdjd.com.dremio.sabot.rpc.user.QueryResultHandler$ChannelClosedHandler.connectionFailed(QueryResultHandler.java:387)
	at cdjd.com.dremio.exec.rpc.BasicClient$ConnectionMultiListener$HandshakeSendListener.failed(BasicClient.java:401)
	at cdjd.com.dremio.exec.rpc.RequestIdMap$RpcListener.setException(RequestIdMap.java:150)
	at cdjd.com.dremio.exec.rpc.RequestIdMap$SetExceptionProcedure.apply(RequestIdMap.java:76)
	at cdjd.com.dremio.exec.rpc.RequestIdMap$SetExceptionProcedure.apply(RequestIdMap.java:66)
	at cdjd.com.carrotsearch.hppc.IntObjectHashMap.forEach(IntObjectHashMap.java:692)
	at cdjd.com.dremio.exec.rpc.RequestIdMap.channelClosed(RequestIdMap.java:62)
	at cdjd.com.dremio.exec.rpc.RemoteConnection.channelClosed(RemoteConnection.java:200)
	at cdjd.com.dremio.exec.rpc.RpcBus$ChannelClosedHandler.operationComplete(RpcBus.java:205)
	at cdjd.com.dremio.exec.rpc.RpcBus$ChannelClosedHandler.operationComplete(RpcBus.java:191)
	at cdjd.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:577)
	at cdjd.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:551)
	at cdjd.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:490)
	at cdjd.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:615)
	at cdjd.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:604)
	at cdjd.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104)
	at cdjd.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84)
	at cdjd.io.netty.channel.AbstractChannel$CloseFuture.setClosed(AbstractChannel.java:1158)
	at cdjd.io.netty.channel.AbstractChannel$AbstractUnsafe.doClose0(AbstractChannel.java:760)
	at cdjd.io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:736)
	at cdjd.io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:607)
	at cdjd.io.netty.channel.DefaultChannelPipeline$HeadContext.close(DefaultChannelPipeline.java:1352)
	at cdjd.io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:622)
	at cdjd.io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:606)
	at cdjd.io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:472)
	at cdjd.com.dremio.exec.rpc.RpcExceptionHandler.exceptionCaught(RpcExceptionHandler.java:37)
	at cdjd.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:302)
	at cdjd.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:281)
	at cdjd.io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:273)
	at cdjd.io.netty.channel.DefaultChannelPipeline$HeadContext.exceptionCaught(DefaultChannelPipeline.java:1377)
	at cdjd.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:302)
	at cdjd.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:281)
	at cdjd.io.netty.channel.DefaultChannelPipeline.fireExceptionCaught(DefaultChannelPipeline.java:907)
	at cdjd.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.handleReadException(AbstractNioByteChannel.java:125)
	at cdjd.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:174)
	at cdjd.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
	at cdjd.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
	at cdjd.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
	at cdjd.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
	at cdjd.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
	at cdjd.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at java.base/java.lang.Thread.run(Unknown Source)
Caused by: cdjd.com.dremio.exec.rpc.ChannelClosedException: [USER]: Channel closed null <--> null (user client)
	at cdjd.com.dremio.exec.rpc.RpcBus$ChannelClosedHandler.operationComplete(RpcBus.java:203)
	... 33 more

Environment Information

• Kestra Version: 0.14.4
• Plugin version: 0.14.4
• Operating System (OS / Docker / Kubernetes): Docker
• Java Version (If not docker):

Example flow

id: dremio-task
namespace: company.team
tasks:
  - id: dremio-query
    type: "io.kestra.plugin.jdbc.dremio.Query"
    url: jdbc:dremio:direct=<local_ip>:31010
    username: <username>
    password: <passwd>
    sql: select * FROM "Samples"."samples.dremio.com"."SF_incidents2016.json"
    fetchOne: true