Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Control Plane Hot Path: Adds more aggressive timeout and retry logic #1954

Merged
merged 29 commits into from
Dec 17, 2020

Conversation

j82w
Copy link
Contributor

@j82w j82w commented Oct 22, 2020

Pull Request Template

Description

  1. Adds the timeout and retry logic
    Non-hotpath (Account information)
    Read:
    5 second timeout (No delay on first retry)
    10 second timeout
    20 second timeout

Hotpath (QueryPlan, partition key ranges, address refresh)
Read retry in the same region. Does not impact region failover:
.5 second timeout (No delay on first retry)
5 second timeout
10 second timeout

All other operations use the current default of 65 seconds.

  1. All HttpRequestExceptions for get operations are retried 3 times with a maximum of 30 seconds.

Tests are still in progress.

Type of change

Please delete options that are not relevant.

  • [] Bug fix (non-breaking change which fixes an issue)
  • [] New feature (non-breaking change which adds functionality)
  • [] Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • [] This change requires a documentation update

Closing issues

closes #1872 #1912

Copy link
Contributor

@bchong95 bchong95 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

New timeouts for query plan need to be approved by query team.

bchong95
bchong95 previously approved these changes Oct 26, 2020
@j82w j82w dismissed stale reviews from bchong95 and ealsur via 540e2c5 October 29, 2020 12:03
@j82w j82w changed the title Draft: Gateway exception handling: Adds additional retry logic and makes it more aggressive Gateway exception handling: Adds additional retry logic and makes it more aggressive Oct 30, 2020
@j82w j82w changed the title Gateway exception handling: Adds additional retry logic and makes it more aggressive Control Plane Hot Path: Adds more aggressive timeout and retry logic Oct 30, 2020
Copy link
Member

@FabianMeiswinkel FabianMeiswinkel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM besides a couple of small comments

@j82w j82w self-assigned this Dec 3, 2020
ealsur
ealsur previously approved these changes Dec 15, 2020
Copy link
Member

@ealsur ealsur left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, just left some nits in case it makes sense to apply

@j82w j82w merged commit 33c0e9a into master Dec 17, 2020
@j82w j82w deleted the users/jawilley/http/transientRetries branch December 17, 2020 22:31
j82w added a commit that referenced this pull request Dec 28, 2020
PR #1954 only did retries on get operations. Get query plan is a post causing it to not do the faster retries. This adds explicit test to verify the behavior and to prevent future regression.

The diagnostics was updated to always include the retry policy to help trouble shoot issues in the future.
@ghost
Copy link

ghost commented Dec 15, 2021

Closing due to in-activity, pease feel free to re-open.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

HttpRequestException thrown due to WSAETimeout errors
5 participants