Availability: Adds cross-region retry mechanism on transient connectivity issues

[ealsur]

Description

Currently, when a client instance has issues connecting to a particular regional endpoint (connectivity timeouts, Azure SNAT issues, transient network blips) the connectivity stack does local (same region) retries, after which, the issue bubbles up as the known 503 - Service Unavailable (which normally has a TransportException inside).

This retry policy is scoped to a particular request, it does not affect other requests in-flight.

Requirements

For the retry mechanism to kick in (in a georeplicated Cosmos DB account in more than 1 region), the requirement is that the CosmosClient is initialized with either:

• CosmosClientOptions.ApplicationRegion defining which is the current region where the application resides on. This builds a preference list of regions where the ApplicationRegion is on top of the list.
• CosmosClientOptions.ApplicationPreferredRegions defining a preference list of 2 or more regions.

When using any of these 2 initialization options, the client builds a list of preferred regions to connect to (which is a subset of the account regions).

Which are the operations that get retried

Only the following subset of operations will be affected by this retry mechanism (if requirements are met):

• Single master accounts - Read requests.
• Multi master accounts - Read requests.
• Multi master accounts - Write requests.

Retry flow

1. One of the retriable operations has a 503 result from the transport stack.
2. The retry policy verifies the client configuration to see which is the list of preferred regions and if the required configuration is met.
3. The retry policy then picks the second region in the preference list.
4. A single retry is done repeating the request to the second region.
5. If the retry also faces a 503 failure, the 503 is bubbled up to the user.

Type of change

• New feature (non-breaking change which adds functionality)

[ghost]

Closing due to in-activity, pease feel free to re-open.