CosmosClient - ObjectDisposed on background refresh

[ealsur]

Related to #2619 (comment)

When creating and disposing a client instance without performing operations (or not waiting for the operation to complete and disposing quickly), the initialization or background worker task that refreshes/gets the account information still tries to complete one execution and fails finding the HttpClient disposed.

This could also be related to the AsyncCacheNonBlocking retrying when the operation:

AsyncCacheNonBlocking Failed GetAsync with key: {0}, Exception: {1}" Argument0="InitTaskKey" Argument1="System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'System.Net.Http.HttpClient'.
at System.Net.Http.HttpClient.CheckDisposed()
at System.Net.Http.HttpClient.CheckRequestBeforeSend(HttpRequestMessage request)
at Microsoft.Azure.Cosmos.CosmosHttpClientCore.ExecuteHttpHelperAsync(HttpRequestMessage requestMessage, ResourceType resourceType, CancellationToken cancellationToken)
at Microsoft.Azure.Cosmos.CosmosHttpClientCore.SendHttpHelperAsync(Func`1 createRequestMessageAsync, ResourceType resourceType, HttpTimeoutPolicy timeoutPolicy, IClientSideRequestStatistics clientSideRequestStatistics, CancellationToken cancellationToken)
at Microsoft.Azure.Cosmos.GatewayAccountReader.GetDatabaseAccountAsync(Uri serviceEndpoint)
at Microsoft.Azure.Cosmos.Routing.GlobalEndpointManager.GetAccountPropertiesHelper.GetAndUpdateAccountPropertiesAsync(Uri endpoint)
at Microsoft.Azure.Cosmos.Routing.GlobalEndpointManager.GetAccountPropertiesHelper.GetAccountPropertiesAsync()
at Microsoft.Azure.Cosmos.GatewayAccountReader.InitializeReaderAsync()
at Microsoft.Azure.Cosmos.CosmosAccountServiceConfiguration.InitializeAsync()
at Microsoft.Azure.Cosmos.DocumentClient.InitializeGatewayConfigurationReaderAsync()
at Microsoft.Azure.Cosmos.DocumentClient.GetInitializationTaskAsync(IStoreClientFactory storeClientFactory)
at Microsoft.Azure.Documents.BackoffRetryUtility`1.ExecuteRetryAsync(Func`1 callbackMethod, Func`3 callShouldRetry, Func`1 inBackoffAlternateCallbackMethod, TimeSpan minBackoffForInBackoffCallback, CancellationToken cancellationToken, Action`1 preRetryCallback)
at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
at Microsoft.Azure.Documents.BackoffRetryUtility`1.ExecuteRetryAsync(Func`1 callbackMethod, Func`3 callShouldRetry, Func`1 inBackoffAlternateCallbackMethod, TimeSpan minBackoffForInBackoffCallback, CancellationToken cancellationToken, Action`1 preRetryCallback)
at Microsoft.Azure.Cosmos.AsyncCacheNonBlocking`2.GetAsync(TKey key, Func`2 singleValueInitFunc, Func`2 forceRefresh)" TraceSource="DocDBTrace"

The initialization task is defined in:

https://github.com/Azure/azure-cosmos-dotnet-v3/blob/master/Microsoft.Azure.Cosmos/src/DocumentClient.cs#L926-L930

Which leads to:

azure-cosmos-dotnet-v3/Microsoft.Azure.Cosmos/src/DocumentClient.cs

Lines 6593 to 6608 in 01e5b29

	private async Task InitializeGatewayConfigurationReaderAsync()
	{
	GatewayAccountReader accountReader = new GatewayAccountReader(
	serviceEndpoint: this.ServiceEndpoint,
	cosmosAuthorization: this.cosmosAuthorization,
	connectionPolicy: this.ConnectionPolicy,
	httpClient: this.httpClient);
	this.accountServiceConfiguration = new CosmosAccountServiceConfiguration(accountReader.InitializeReaderAsync);
	await this.accountServiceConfiguration.InitializeAsync();
	AccountProperties accountProperties = this.accountServiceConfiguration.AccountProperties;
	this.UseMultipleWriteLocations = this.ConnectionPolicy.UseMultipleWriteLocations && accountProperties.EnableMultipleWriteLocations;
	this.GlobalEndpointManager.InitializeAccountPropertiesAndStartBackgroundRefresh(accountProperties);
	}

If the client is Disposed before or during this step, then the HttpClient would be attempted to be used disposed.

When the internal client is Disposed the related objects are Disposed but potentially not inflight operations,

azure-cosmos-dotnet-v3/Microsoft.Azure.Cosmos/src/DocumentClient.cs

Line 1196 in 01e5b29

public void Dispose()

Ideas

1. Add a CancellationTokenSource to the DocumentClient.cs that Cancels when the client is Disposed
2. Wire that CancellationTokenSource.Token as an input parameter of AsyncCacheNonBlocking

   azure-cosmos-dotnet-v3/Microsoft.Azure.Cosmos/src/Routing/AsyncCacheNonBlocking.cs

   Line 40 in 01e5b29

   public AsyncCacheNonBlocking()

   so the internal CancellationTokenSource is created as Linked (CancellationTokenSource.CreateLinkedTokenSource). This should be an optional parameter that otherwise creates a normal CancellationTokenSource
3. Pass the CancellationTokenSource.Token to

   azure-cosmos-dotnet-v3/Microsoft.Azure.Cosmos/src/DocumentClient.cs

   Line 6595 in 01e5b29

   GatewayAccountReader accountReader = new GatewayAccountReader(

4. Pass it to

   azure-cosmos-dotnet-v3/Microsoft.Azure.Cosmos/src/GatewayAccountReader.cs

   Line 55 in 01e5b29

   cancellationToken: default))

@imanvt feel free to think or propose others