The remote port forwarding on the server cannot be released, causing a large number of CLOSE_WAIT states.

[wumengchao]

Version

2.14.0

Bug description

Discovered an issue that is difficult to reproduce and troubleshoot. The phenomenon is as follows: After creating port forwarding on the client through SSH -R, the client's network disconnects for uncertain reasons. However, the server's port cannot be released, and when testing requests to the server's port, a large number of CLOSE_WAITs appear.

Actual behavior

After creating port forwarding on the client through SSH -R, the client's network disconnects for uncertain reasons. However, the server's port cannot be released, and when testing requests to the server's port, a large number of CLOSE_WAITs appear.

Expected behavior

Hope to be provided with troubleshooting methods, and ideally a quick resolution.

Relevant log output

No response

Other information

No response

[wumengchao]

WARN o.apache.sshd.netty.NettyIoSession - channelActive(session=NettyIoSession[local=/127.0.0.1:8443, remote=/127.0.0.1:5023]): could not create SSH session (java.lang.IllegalStateException); closing
java.lang.IllegalStateException: Channel id=82 not registered because session is being closed: TcpipClientChannel[id=82, recipient=-1]-ServerSessionImpl[deploy@/127.0.0.1:3471]
at org.apache.sshd.common.channel.AbstractChannel.handleChannelRegistrationResult(AbstractChannel.java:471)
at org.apache.sshd.common.session.helpers.AbstractConnectionService.registerChannel(AbstractConnectionService.java:424)
at org.apache.sshd.common.forward.DefaultForwarder$StaticIoHandler.sessionCreated(DefaultForwarder.java:1024)
at org.apache.sshd.netty.NettyIoSession.channelActive(NettyIoSession.java:254)
at org.apache.sshd.netty.NettyIoSession$Adapter.channelActive(NettyIoSession.java:356)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:260)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:238)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelActive(AbstractChannelHandlerContext.java:231)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelActive(DefaultChannelPipeline.java:1395)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:258)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:238)
at io.netty.channel.DefaultChannelPipeline.fireChannelActive(DefaultChannelPipeline.java:894)
at io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:521)
at io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:428)
at io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:485)
at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:994)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:1570)

[wumengchao]

Port 8443 process has ended, but the port forwarded by mina-sshd is still occupied and cannot be released.

[tomaswolf]

Can you draw me a picture similar to the ones in https://github.com/apache/mina-sshd/blob/master/docs/technical/tcpip-forwarding.md ? Which server corresponds to "whatever", which port is "A" and "B", and what is your exact ssh command?

The exception message says the session was being closed. So why is it being closed? There must be some earlier problem. Can you run that server with debug logging?

Finally, I see the server is using the Netty transport back-end. Can you try running it with the NIO2 back-end instead? Does the problem also exist with NIO2?