Transient HTTP exceptions: Adds retry logic to all http requests

[j82w]

Pull Request Template

Description

This PR adds an abstraction on top of HttpClient called CosmosHttpClient. All the caches and gateway calls have been refactored to use the new abstraction. This adds retry logic for transient exceptions that should be retried on.

Type of change

Please delete options that are not relevant.

• [] Bug fix (non-breaking change which fixes an issue)
• [] New feature (non-breaking change which adds functionality)
• [] Breaking change (fix or feature that would cause existing functionality to not work as expected)
• [] This change requires a documentation update

Closing issues

closes #1769

[YohanSciubukgian]

As you have recently added HttpClientFactory (https://devblogs.microsoft.com/cosmosdb/httpclientfactory-cosmos-db-net-sdk/), why don't you use Polly which is now added as an extension to the framework (https://docs.microsoft.com/en-us/dotnet/architecture/microservices/implement-resilient-applications/use-httpclientfactory-to-implement-resilient-http-requests) ?

It will allow us to take full advantages of MessageHandlers registration from the HttpClientFactory and have full option control over Polly directly (retry policy, circuit breaker,...).

[j82w]

As you have recently added HttpClientFactory (https://devblogs.microsoft.com/cosmosdb/httpclientfactory-cosmos-db-net-sdk/), why don't you use Polly which is now added as an extension to the framework (https://docs.microsoft.com/en-us/dotnet/architecture/microservices/implement-resilient-applications/use-httpclientfactory-to-implement-resilient-http-requests) ?

It will allow us to take full advantages of MessageHandlers registration from the HttpClientFactory and have full option control over Polly directly (retry policy, circuit breaker,...).

@YohanSciubukgian

1. General guideline is to avoid dependencies because it causes conflicts with other projects. https://azure.github.io/azure-sdk/dotnet_introduction.html#dotnet-dependencies
2. This model guarantees that the retries always occur. If the user specifies a custom http factory then it's not possible for us to add the retry handler. Depending on every one that uses a custom http client factory to implement the correct retry logic doesn't seem realistic. The better model seems to be to wrap the http client and always retry.

[YohanSciubukgian]

I didn't have the compatibility with .Net Standard in mind neither the need to avoid dependencies as much as possible (even with Microsoft.* packages)
Thanks for your explanations 😄

[kirankumarkolli]

Will it impact compute?

[j82w]

No, they only pass in the sending request. I didn't find any references to the properties in Cassandra or Mongo projects. It's also really dangerous because if they set a custom httpclient then this field no longer works because it's not possible to add a http client handler after that the http client is created.

[kirankumarkolli]

[ghost]

Closing due to in-activity, pease feel free to re-open.