-
Notifications
You must be signed in to change notification settings - Fork 147
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implementing retry for remote connector to mitigate throttling issue #2462
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: zhichao-aws <zhichaog@amazon.com>
Signed-off-by: zhichao-aws <zhichaog@amazon.com>
Signed-off-by: zhichao-aws <zhichaog@amazon.com>
Signed-off-by: zhichao-aws <zhichaog@amazon.com>
Signed-off-by: zhichao-aws <zhichaog@amazon.com>
Signed-off-by: zhichao-aws <zhichaog@amazon.com>
Signed-off-by: zhichao-aws <zhichaog@amazon.com>
Signed-off-by: zhichao-aws <zhichaog@amazon.com>
Signed-off-by: zhichao-aws <zhichaog@amazon.com>
Zhangxunmt
approved these changes
Jun 6, 2024
zane-neo
approved these changes
Jun 6, 2024
opensearch-trigger-bot bot
pushed a commit
that referenced
this pull request
Jun 6, 2024
…2462) * use retryable action; execution context Signed-off-by: zhichao-aws <zhichaog@amazon.com> * change to groupedActionListener Signed-off-by: zhichao-aws <zhichaog@amazon.com> * fix group Signed-off-by: zhichao-aws <zhichaog@amazon.com> * retry policy Signed-off-by: zhichao-aws <zhichaog@amazon.com> * base time Signed-off-by: zhichao-aws <zhichaog@amazon.com> * retry option, cluster settings Signed-off-by: zhichao-aws <zhichaog@amazon.com> * nit Signed-off-by: zhichao-aws <zhichaog@amazon.com> * lint Signed-off-by: zhichao-aws <zhichaog@amazon.com> * change interface to class Signed-off-by: zhichao-aws <zhichaog@amazon.com> * fix ut due to code change Signed-off-by: zhichao-aws <zhichaog@amazon.com> * license header Signed-off-by: zhichao-aws <zhichaog@amazon.com> * add ut Signed-off-by: zhichao-aws <zhichaog@amazon.com> * add test Signed-off-by: zhichao-aws <zhichaog@amazon.com> * fix core interface Signed-off-by: zhichao-aws <zhichaog@amazon.com> * test Signed-off-by: zhichao-aws <zhichaog@amazon.com> * license header Signed-off-by: zhichao-aws <zhichaog@amazon.com> * use exception holder Signed-off-by: zhichao-aws <zhichaog@amazon.com> * add max retry times settings Signed-off-by: zhichao-aws <zhichaog@amazon.com> * fix typo Signed-off-by: zhichao-aws <zhichaog@amazon.com> * nit Signed-off-by: zhichao-aws <zhichaog@amazon.com> * change the order to avoid misleading log Signed-off-by: zhichao-aws <zhichaog@amazon.com> * license header Signed-off-by: zhichao-aws <zhichaog@amazon.com> * move settings to connector Signed-off-by: zhichao-aws <zhichaog@amazon.com> * remove settings Signed-off-by: zhichao-aws <zhichaog@amazon.com> * add test Signed-off-by: zhichao-aws <zhichaog@amazon.com> * add retry_backoff_policy setting Signed-off-by: zhichao-aws <zhichaog@amazon.com> * changes for comments Signed-off-by: zhichao-aws <zhichaog@amazon.com> * fix retry times Signed-off-by: zhichao-aws <zhichaog@amazon.com> * make the error handling more neat in MLSdkAsyncHttpResponseHandler Signed-off-by: zhichao-aws <zhichaog@amazon.com> * change to SageMakerThrottlingException Signed-off-by: zhichao-aws <zhichaog@amazon.com> * use enum for retry backoff policy Signed-off-by: zhichao-aws <zhichaog@amazon.com> * fix seconds to milliseconds in equal jitter policy Signed-off-by: zhichao-aws <zhichaog@amazon.com> * disable retry by default Signed-off-by: zhichao-aws <zhichaog@amazon.com> --------- Signed-off-by: zhichao-aws <zhichaog@amazon.com> (cherry picked from commit 399825f)
ylwu-amzn
pushed a commit
that referenced
this pull request
Jun 6, 2024
…2462) (#2509) * use retryable action; execution context Signed-off-by: zhichao-aws <zhichaog@amazon.com> * change to groupedActionListener Signed-off-by: zhichao-aws <zhichaog@amazon.com> * fix group Signed-off-by: zhichao-aws <zhichaog@amazon.com> * retry policy Signed-off-by: zhichao-aws <zhichaog@amazon.com> * base time Signed-off-by: zhichao-aws <zhichaog@amazon.com> * retry option, cluster settings Signed-off-by: zhichao-aws <zhichaog@amazon.com> * nit Signed-off-by: zhichao-aws <zhichaog@amazon.com> * lint Signed-off-by: zhichao-aws <zhichaog@amazon.com> * change interface to class Signed-off-by: zhichao-aws <zhichaog@amazon.com> * fix ut due to code change Signed-off-by: zhichao-aws <zhichaog@amazon.com> * license header Signed-off-by: zhichao-aws <zhichaog@amazon.com> * add ut Signed-off-by: zhichao-aws <zhichaog@amazon.com> * add test Signed-off-by: zhichao-aws <zhichaog@amazon.com> * fix core interface Signed-off-by: zhichao-aws <zhichaog@amazon.com> * test Signed-off-by: zhichao-aws <zhichaog@amazon.com> * license header Signed-off-by: zhichao-aws <zhichaog@amazon.com> * use exception holder Signed-off-by: zhichao-aws <zhichaog@amazon.com> * add max retry times settings Signed-off-by: zhichao-aws <zhichaog@amazon.com> * fix typo Signed-off-by: zhichao-aws <zhichaog@amazon.com> * nit Signed-off-by: zhichao-aws <zhichaog@amazon.com> * change the order to avoid misleading log Signed-off-by: zhichao-aws <zhichaog@amazon.com> * license header Signed-off-by: zhichao-aws <zhichaog@amazon.com> * move settings to connector Signed-off-by: zhichao-aws <zhichaog@amazon.com> * remove settings Signed-off-by: zhichao-aws <zhichaog@amazon.com> * add test Signed-off-by: zhichao-aws <zhichaog@amazon.com> * add retry_backoff_policy setting Signed-off-by: zhichao-aws <zhichaog@amazon.com> * changes for comments Signed-off-by: zhichao-aws <zhichaog@amazon.com> * fix retry times Signed-off-by: zhichao-aws <zhichaog@amazon.com> * make the error handling more neat in MLSdkAsyncHttpResponseHandler Signed-off-by: zhichao-aws <zhichaog@amazon.com> * change to SageMakerThrottlingException Signed-off-by: zhichao-aws <zhichaog@amazon.com> * use enum for retry backoff policy Signed-off-by: zhichao-aws <zhichaog@amazon.com> * fix seconds to milliseconds in equal jitter policy Signed-off-by: zhichao-aws <zhichaog@amazon.com> * disable retry by default Signed-off-by: zhichao-aws <zhichaog@amazon.com> --------- Signed-off-by: zhichao-aws <zhichaog@amazon.com> (cherry picked from commit 399825f) Co-authored-by: zhichao-aws <zhichaog@amazon.com>
opensearch-trigger-bot bot
pushed a commit
that referenced
this pull request
Sep 30, 2024
…2462) * use retryable action; execution context Signed-off-by: zhichao-aws <zhichaog@amazon.com> * change to groupedActionListener Signed-off-by: zhichao-aws <zhichaog@amazon.com> * fix group Signed-off-by: zhichao-aws <zhichaog@amazon.com> * retry policy Signed-off-by: zhichao-aws <zhichaog@amazon.com> * base time Signed-off-by: zhichao-aws <zhichaog@amazon.com> * retry option, cluster settings Signed-off-by: zhichao-aws <zhichaog@amazon.com> * nit Signed-off-by: zhichao-aws <zhichaog@amazon.com> * lint Signed-off-by: zhichao-aws <zhichaog@amazon.com> * change interface to class Signed-off-by: zhichao-aws <zhichaog@amazon.com> * fix ut due to code change Signed-off-by: zhichao-aws <zhichaog@amazon.com> * license header Signed-off-by: zhichao-aws <zhichaog@amazon.com> * add ut Signed-off-by: zhichao-aws <zhichaog@amazon.com> * add test Signed-off-by: zhichao-aws <zhichaog@amazon.com> * fix core interface Signed-off-by: zhichao-aws <zhichaog@amazon.com> * test Signed-off-by: zhichao-aws <zhichaog@amazon.com> * license header Signed-off-by: zhichao-aws <zhichaog@amazon.com> * use exception holder Signed-off-by: zhichao-aws <zhichaog@amazon.com> * add max retry times settings Signed-off-by: zhichao-aws <zhichaog@amazon.com> * fix typo Signed-off-by: zhichao-aws <zhichaog@amazon.com> * nit Signed-off-by: zhichao-aws <zhichaog@amazon.com> * change the order to avoid misleading log Signed-off-by: zhichao-aws <zhichaog@amazon.com> * license header Signed-off-by: zhichao-aws <zhichaog@amazon.com> * move settings to connector Signed-off-by: zhichao-aws <zhichaog@amazon.com> * remove settings Signed-off-by: zhichao-aws <zhichaog@amazon.com> * add test Signed-off-by: zhichao-aws <zhichaog@amazon.com> * add retry_backoff_policy setting Signed-off-by: zhichao-aws <zhichaog@amazon.com> * changes for comments Signed-off-by: zhichao-aws <zhichaog@amazon.com> * fix retry times Signed-off-by: zhichao-aws <zhichaog@amazon.com> * make the error handling more neat in MLSdkAsyncHttpResponseHandler Signed-off-by: zhichao-aws <zhichaog@amazon.com> * change to SageMakerThrottlingException Signed-off-by: zhichao-aws <zhichaog@amazon.com> * use enum for retry backoff policy Signed-off-by: zhichao-aws <zhichaog@amazon.com> * fix seconds to milliseconds in equal jitter policy Signed-off-by: zhichao-aws <zhichaog@amazon.com> * disable retry by default Signed-off-by: zhichao-aws <zhichaog@amazon.com> --------- Signed-off-by: zhichao-aws <zhichaog@amazon.com> (cherry picked from commit 399825f)
dhrubo-os
pushed a commit
that referenced
this pull request
Sep 30, 2024
…2462) (#3013) * use retryable action; execution context Signed-off-by: zhichao-aws <zhichaog@amazon.com> * change to groupedActionListener Signed-off-by: zhichao-aws <zhichaog@amazon.com> * fix group Signed-off-by: zhichao-aws <zhichaog@amazon.com> * retry policy Signed-off-by: zhichao-aws <zhichaog@amazon.com> * base time Signed-off-by: zhichao-aws <zhichaog@amazon.com> * retry option, cluster settings Signed-off-by: zhichao-aws <zhichaog@amazon.com> * nit Signed-off-by: zhichao-aws <zhichaog@amazon.com> * lint Signed-off-by: zhichao-aws <zhichaog@amazon.com> * change interface to class Signed-off-by: zhichao-aws <zhichaog@amazon.com> * fix ut due to code change Signed-off-by: zhichao-aws <zhichaog@amazon.com> * license header Signed-off-by: zhichao-aws <zhichaog@amazon.com> * add ut Signed-off-by: zhichao-aws <zhichaog@amazon.com> * add test Signed-off-by: zhichao-aws <zhichaog@amazon.com> * fix core interface Signed-off-by: zhichao-aws <zhichaog@amazon.com> * test Signed-off-by: zhichao-aws <zhichaog@amazon.com> * license header Signed-off-by: zhichao-aws <zhichaog@amazon.com> * use exception holder Signed-off-by: zhichao-aws <zhichaog@amazon.com> * add max retry times settings Signed-off-by: zhichao-aws <zhichaog@amazon.com> * fix typo Signed-off-by: zhichao-aws <zhichaog@amazon.com> * nit Signed-off-by: zhichao-aws <zhichaog@amazon.com> * change the order to avoid misleading log Signed-off-by: zhichao-aws <zhichaog@amazon.com> * license header Signed-off-by: zhichao-aws <zhichaog@amazon.com> * move settings to connector Signed-off-by: zhichao-aws <zhichaog@amazon.com> * remove settings Signed-off-by: zhichao-aws <zhichaog@amazon.com> * add test Signed-off-by: zhichao-aws <zhichaog@amazon.com> * add retry_backoff_policy setting Signed-off-by: zhichao-aws <zhichaog@amazon.com> * changes for comments Signed-off-by: zhichao-aws <zhichaog@amazon.com> * fix retry times Signed-off-by: zhichao-aws <zhichaog@amazon.com> * make the error handling more neat in MLSdkAsyncHttpResponseHandler Signed-off-by: zhichao-aws <zhichaog@amazon.com> * change to SageMakerThrottlingException Signed-off-by: zhichao-aws <zhichaog@amazon.com> * use enum for retry backoff policy Signed-off-by: zhichao-aws <zhichaog@amazon.com> * fix seconds to milliseconds in equal jitter policy Signed-off-by: zhichao-aws <zhichaog@amazon.com> * disable retry by default Signed-off-by: zhichao-aws <zhichaog@amazon.com> --------- Signed-off-by: zhichao-aws <zhichaog@amazon.com> (cherry picked from commit 399825f) Co-authored-by: zhichao-aws <zhichaog@amazon.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Please check #2438 for more details and context
What does this PR change:
a. New class RetryableException is added. And we can extend it once we found other retryable cases.
With previous code we call onResponse/onFailure only after all sub-requests have response. With the refactor, we call onResponse/onFailure for all sub-requests, and the onFailure will further trigger the retry logic for sub-request in RetryableAction.
Note: the retry is disabled by default to make the behavior consistent with previous version.
Issues Resolved
#2438
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.