[Bug] Multiple deadlocks were detected during the client shutdown process.

[YanYunyang]

Before Creating the Bug Report

• I found a bug, not just asking a question, which should be created in GitHub Discussions.

• I have searched the GitHub Issues and GitHub Discussions of this repository and believe that this is not a duplicate.

• I have confirmed that this bug belongs to the current repository, not other repositories of RocketMQ.

Runtime platform environment

4.19.90-23-42.v2101.ky10.x86_64

RocketMQ version

5.2.0

JDK Version

openjdk 1.8

Describe the Bug

During the client shutdown process, deadlocks were detected, which resolved after a period of time.
This phenomenon was observed using jvisualvm and jconsole.
The deadlock occurred between the client shutdown thread and the Netty worker threads.
Code analysis:

1. NettyRemotingClient holds the lockChannelTables lock, which is responsible for guarding access to channelTables. channelTables caches all channels encapsulated in ChannelWrapper.
2. The inner class ChannelWrapper within NettyRemotingClientholds a read-write locklock`, responsible for concurrent access to the channel.
3. The inner class NettyConnectManageHandler in NettyRemotingClient is responsible for handling events like close, connect, and channelInactive. When a channel becomes unavailable, it executes the close or channelInactive methods to remove the channel from channelTables.

The execution path where deadlock occurs.

Steps to Reproduce

1.Creating multiple consumer clients and shutting them down one by one. The more clients are created, the more likely a deadlock will be triggered.
2.Open jconsole and continuously click on deadlock detection.

What Did You Expect to See?

deadlocks were detected.

What Did You See Instead?

deadlocks were detected.

Additional Context

No response