Handle the transient errors in efcore when using Cosmos DB

[rmt2021]

It is possible to encounter transient errors when using efcore with Cosmos DB, and it seems that the efcore cannot handle such errors (like 410 Gone) now. Is it better if we can provide some retries for the transient error codes? Otherwise APIs like: CreateDatabaseIfNotExistsAsync, CreateContainerStreamAsync, CreateItemStreamAsync, ReadNextAsync, ReadNextAsync, DeleteItemStreamAsync, ReplaceItemStreamAsync, ReadItemStreamAsync will directly fail due to transient faults.

As this document shows, there are some transient error codes (408, 410, 429, 449, and 503) that we can retry to make efcore more resilient:
https://docs.microsoft.com/en-us/azure/cosmos-db/sql/conceptual-resilient-sdk-applications#should-my-application-retry-on-errors

[roji]

@rmt2021 see the conversation in #8443 (comment). As the docs you linked to specify, the error codes you listed should be retried by the SDK, which the EF provider uses. Are you seeing a different behavior?

[rmt2021]

Thanks for sharing this conversation @roji.

The SDK will only retry on the error codes when there are multiple available regions. For example, if 503 happens, and there is only one region for the Cosmos DB, then the retry will not happen, as the SDK code written:
https://github.com/Azure/azure-cosmos-dotnet-v3/blob/b713ce4cb3e482175a8a6a9b8fc7051d9c7b5e91/Microsoft.Azure.Cosmos/src/ClientRetryPolicy.cs#L373-L378

I notice that efcore actually has this code snippet to retry for transient error codes:

efcore/src/EFCore.Cosmos/Storage/Internal/CosmosExecutionStrategy.cs

Lines 105 to 107 in 0d1d602

	static bool IsTransient(HttpStatusCode statusCode)
	=> statusCode == HttpStatusCode.ServiceUnavailable
	|| statusCode == HttpStatusCode.TooManyRequests;

which is to retry for 503 (ServiceUnavailable) and 429 (TooManyRequests).

I believe we should also retry on 408, 410 and 449.