Automatic recovery fails if the channel is disposed

[MatElcome]

Describe the bug

If the connection to the RabbitMQ server is interrupted (eg. networking issue, server restart), the client library will try to recover and recreate any open channels (provided the AutomaticRecoveryEnabled = true option is used). However, the recovery will fail if a channel is disposed in user code before the recovery operation succeeds.

This issue is present in version 7.0.0-rc.6 of RabbitMQ.Client - it doesn't seem to affect version 6.8.1

Reproduction steps

1. Start a rabbitmq server using docker

docker run -d -h rabbit-test --name rabbit-test -p 5672:5672 rabbitmq:3-alpine

2. Create a new csharp console project, add a reference to RabbitMQ.Client version 7.0.0-rc.6
3. Add the following code:

RabbitMQ.Client.ConnectionFactory connectionFactory = new()
{
    AutomaticRecoveryEnabled = true,
    UserName = "guest",
};

string exchangeName = "test-exchange";

// Initial setup: create exchange
{
    using var connection = await connectionFactory.CreateConnectionAsync();
    using var channel = await connection.CreateChannelAsync();
    await channel.ExchangeDeclareAsync(exchangeName, "topic", true, false);
}

// Send message loop
{
    using var connection = await connectionFactory.CreateConnectionAsync();
    for (int i = 0; i < 300; i++)
    {
        try
        {
            using var channel = await connection.CreateChannelAsync(); // New channel for each message
            await Task.Delay(1000);
            await channel.BasicPublishAsync(exchangeName, "", new RabbitMQ.Client.BasicProperties(), System.Text.Encoding.UTF8.GetBytes("test"));
            Console.WriteLine($"Sent message {i}");
        }
        catch (Exception ex)
        {
            Console.WriteLine($"Failed to send message {i}: {ex.Message}");
            await Task.Delay(1000);
        }
    }
}

4. Restart the rabbitmq server to trigger autorecovery

docker restart rabbit-test

Expected behavior

Expected output:

Sent message 0
Sent message 1
Sent message 2
Sent message 3
Failed to send message 4: Already closed: The AMQP operation was interrupted: AMQP close-reason, initiated by Peer, code=320, text='CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'', classId=0, methodId=0
Failed to send message 5: Already closed: The AMQP operation was interrupted: AMQP close-reason, initiated by Peer, code=320, text='CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'', classId=0, methodId=0
...
Failed to send message 17: Already closed: The AMQP operation was interrupted: AMQP close-reason, initiated by Peer, code=320, text='CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'', classId=0, methodId=0
Sent message 18
Sent message 19

Actual output:

Sent message 0
Sent message 1
Sent message 2
Sent message 3
Sent message 4
Failed to send message 5: Already closed: The AMQP operation was interrupted: AMQP close-reason, initiated by Peer, code=320, text='CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'', classId=0, methodId=0
Failed to send message 6: Already closed: The AMQP operation was interrupted: AMQP close-reason, initiated by Peer, code=320, text='CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'', classId=0, methodId=0
...
Failed to send message 23: Already closed: The AMQP operation was interrupted: AMQP close-reason, initiated by Peer, code=320, text='CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'', classId=0, methodId=0
Failed to send message 24: Already closed: The AMQP operation was interrupted: AMQP close-reason, initiated by Application, code=541, text='FailedAutoRecovery', classId=0, methodId=0
Failed to send message 25: Already closed: The AMQP operation was interrupted: AMQP close-reason, initiated by Application, code=541, text='FailedAutoRecovery', classId=0, methodId=0
Failed to send message 26: Already closed: The AMQP operation was interrupted: AMQP close-reason, initiated by Application, code=541, text='FailedAutoRecovery', classId=0, methodId=0
...

Notice how in message 24 onwards, the error message is FailedAutoRecovery.

Additional context

The following exception is thrown during autorecovery:

System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'RabbitMQ.Client.Impl.AutorecoveringChannel'.
   at RabbitMQ.Client.Impl.AutorecoveringChannel.<ThrowIfDisposed>g__ThrowDisposed|87_0()
   at RabbitMQ.Client.Impl.AutorecoveringChannel.AutomaticallyRecoverAsync(AutorecoveringConnection conn, Boolean recoverConsumers, Boolean recordedEntitiesSemaphoreHeld, CancellationToken cancellationToken)
   at RabbitMQ.Client.Framing.Impl.AutorecoveringConnection.RecoverChannelsAndItsConsumersAsync(Boolean recordedEntitiesSemaphoreHeld, CancellationToken cancellationToken)
   at RabbitMQ.Client.Framing.Impl.AutorecoveringConnection.TryPerformAutomaticRecoveryAsync(CancellationToken cancellationToken)

[lukebakken]

@MatElcome thank you for the complete issue report!!! I will release a new RC with a fix shortly.

[lukebakken]

@MatElcome - please test out RC 7 when it is published, which should be within the next 30 minutes -

https://github.com/rabbitmq/rabbitmq-dotnet-client/releases/tag/v7.0.0-rc.7

https://github.com/rabbitmq/rabbitmq-dotnet-client/actions/runs/10184587833

Thank you very much!

[lukebakken]

@MatElcome FYI - hot off of the presses - https://www.nuget.org/packages/RabbitMQ.Client/7.0.0-rc.7

[MatElcome]

@lukebakken Perfect, RC 7 resolves the issue.

Thanks for such a quick fix - its greatly appreciated!

[lukebakken]

@MatElcome thank YOU for testing the RC versions!