CosmosClient::Dispose throws SynchronizationLockException

[ohads-MSFT]

Describe the bug
CosmosClient::Dispose() throws SynchronizationLockException

To Reproduce
We don't know exactly how to reproduce, but it happens on ASP.NET app shutdown. We have an IDisposable Cosmos Client registered to the ASP.NET DI that when disposed by the ASP.NET host, disposes all the different clients it created in a loop:

foreach (var client in _clients)
{
    client.Value.Value.Dispose();
}

Note that .Value is Lazy<T> so I suspect it could be that some of the clients above are actually created (new CosmosClient()) during that loop .

Expected behavior
Dispose() should not throw exceptions.

Actual behavior
CosmosClient::Dispose() sometime throws as explained above (see full screenshot below)

Environment summary
SDK Version: Microsoft.Azure.Cosmos 3.32.0 (latest at the time of writing)
OS Version: linux (mcr.microsoft.com/dotnet/aspnet:7.0.2-cbl-mariner2.0-distroless)

Additional context

Unhandled exception. System.Threading.SynchronizationLockException: The lock is being disposed while still being used. 
It either is being held by a thread and/or has active waiters waiting to acquire the lock.
   at System.Threading.ReaderWriterLockSlim.Dispose(Boolean disposing)
   at Microsoft.Azure.Documents.Rntbd.LoadBalancingPartition.Dispose()
   at Microsoft.Azure.Documents.Rntbd.LoadBalancingChannel.System.IDisposable.Dispose()
   at Microsoft.Azure.Documents.Rntbd.LoadBalancingChannel.Close()
   at Microsoft.Azure.Documents.Rntbd.ChannelDictionary.Dispose()
   at Microsoft.Azure.Documents.Rntbd.TransportClient.Dispose()
   at Microsoft.Azure.Documents.StoreClientFactory.Dispose()
   at Microsoft.Azure.Cosmos.DocumentClient.Dispose()
   at Microsoft.Azure.Cosmos.ClientContextCore.Dispose(Boolean disposing)
   at Microsoft.Azure.Cosmos.ClientContextCore.Dispose()
   at Microsoft.Azure.Cosmos.CosmosClient.Dispose(Boolean disposing)
   at Microsoft.Azure.Cosmos.CosmosClient.Dispose()
   at Microsoft.Azure.XXX.Documents.Cosmos.CosmosClientFactory.DisposeAsync() in 
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope.DisposeAsync()
--- End of stack trace from previous location ---
   at Microsoft.Extensions.Hosting.Internal.Host.<DisposeAsync>g__DisposeAsync|16_0(Object o)
   at Microsoft.Extensions.Hosting.Internal.Host.DisposeAsync()
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.Run(IHost host)
   at Microsoft.Azure.XXXProgram.Main(String[] args) 

[ealsur]

@kundadebdatta Can you take a look at this report?

@ohads-MSFT It looks like you are disposing the client while there are requests in-progress, is that expected?

[ealsur]

If the app is shutting down, what is the impact of these exceptions? The biggest impact is that you are leaving requests in-flight, right?

[ohads-MSFT]

It looks like you are disposing the client while there are requests in-progress, is that expected?

@ealsur sounds right, I'm guessing the Cosmos SDK doesn't have any ASP.NET shutdown hooks to drain all pending requests

If the app is shutting down, what is the impact of these exceptions? The biggest impact is that you are leaving requests in-flight, right?

Arguably an even bigger impact is the fact that an exception is thrown from Dispose() generally, and during ASP.NET app shutdown specifically, which crashes the process. Beyond the fact that it creates logging noise and red herrings, it means that other components in the application might not shut down gracefully (e.g. Dispose() is not called for them).

[rinatmini]

Fixed in the Microsoft.Azure.Cosmos.Direct package and will be delivered with the next release Microsoft.Azure.Cosmos.Direct